Lazy Load techniques .. another one hack ..

We have already seen few lazy load techniques here and here.  If we have large set of HTML , it takes so much time to load page.


Here is a trick that allows you to bundle all of your modules into a single resource without having to parse any of the JavaScript. Of course, with this strategy, there is greater latency with the initial download of the single resource (since it has all your JavaScript modules), but once the resource is stored in the browser's application cache, this issue becomes much less of a factor.

To combine all modules into a single resource, we wrote each module into a separate script tag and hide the code inside a comment block (/* */). When the resource first loads, none of the code is parsed since it is commented out. To load a module, find the DOM element for the corresponding script tag, strip out the comment block, and eval() the code. If the web app supports XHTML, this trick is even more elegant as the modules can be hidden inside a CDATA tag instead of a script tag. An added bonus is the ability to lazy load your modules synchronously since there's no longer a need to fetch the modules asynchronously over the network.

200k of JavaScript held within a block comment adds 240ms during page load, whereas 200k of JavaScript that is parsed during page load added 2600 ms. That's more than a 10x reduction in startup latency by eliminating 200k of unneeded JavaScript during page load! Take a look at the code sample below to see how this is done.



<html>
...
<script id="lazy">
// Make sure you strip out (or replace) comment blocks in your JavaScript first.
/*
JavaScript of lazy module
*/
</script>

<script>
  function lazyLoad() {
    var lazyElement = document.getElementById('lazy');
    var lazyElementBody = lazyElement.innerHTML;
    var jsCode = stripOutCommentBlock(lazyElementBody);
    eval(jsCode);
  }
</script>

<div onclick=lazyLoad()> Lazy Load </div>
</html>

File based caching on high traffic website

If you have a lot of traffic, and you’re caching a page for an hour… Over the course of one hour, you may read the file possibly 200 times, and write just once… The time taken to write is minimal, around 10ms? - So, out of one hour, you have 10ms ‘downtime’ for your cache reads.

Which is 0.00027% downtime… Extremely minimal, and not really worth worrying about.
Locks will only cause this fractional locking time, and, if it only effects 2 people trying to write at the same time.

The correct way to work around it is explained here:-

* 1 page is cached for an hour.
* When that cache expires, 2 people visit the site with 10ms of each other (the only time this will cause an ‘issue’, because otherwise, the first will have written the cache before the second requests the page).
* The first user ‘reads’ the cache file, sees it’s expired so begins rendering the page normally with PHP/MySQL
* The second user has the same thing going on… the cache hasn’t been updated by the second user.
* The first user finishes building the page, and begins writing the file, locking it.
* The second user also finishes building the page, and attempts to write it.
* Because the first user has locked the file, the second user can’t write it…
* This is a problem for the second user, so you simply build a little fix to make it all work like magic.

If the second user can’t write the the cache (because it is locked), simply echo the contents of the rendered page, not saving it to the cache.
The first user is dealing with the cache, so just render the output.

With Codegniter, this is how it works…

Line 299 of Output.php:-


if ( ! $fp = @fopen($cache_path, 'wb'))
        {
            log_message('error', "Unable to write cache file: ".$cache_path);
            return;
        }

The cache file is only written if it’s ‘really_writable’ (not locked)

Line 59 of Output.php:


elseif (($fp = @fopen($file, 'ab')) === FALSE)
    {
        return FALSE;


So… if we cant write it, just render the second users’ request.

Simple, and it works, beautifully... :-)

developer or team lead : pure development role to team leadership role

Few months ago, my role was evolving in pure development.My day used to be pure coding.I had generally enjoying it. Since last few months my job responsibilities are changed from developer to team lead.Now, I still have a largely full plate of coding duties, but few others are added in my duties like I'm expected to mentor other developers,work on requirements, make design decisions for other developers, evaluate bug reports from users, assign them to developers,team communication etc.

I find that my day has become on interruption after another and the prolonged periods of sustained concentration needed to get any actual quality coding done are becoming rarer and rarer.

As an individually-contributing developer, my job was to turn my own time in to software that the business could sell for a profit.

As a team lead, my job is to see that the team effectively turns their time in to software that the business could sell for a profit.

Some things fundamentally change when your perspective changes like that. These things have become much more important:-

• Keeping other members of the team in a state that they can be productive
• Delegating tasks to the least loaded team member
• Strategically choosing which developer needs to learn which new skill to better load-balance the team, and investing some degree of my time in helping them learn that skill
• Effectively communicating requirements (somebody told "Think twice before you start programming or you will program twice before you start thinking")

Notice that "writing good code myself" is no longer on my list of top concerns. If the task of "develop this major new thing" falls on me, it's almost always for one of a few reasons:-

• The new thing is a framework item that will enable the rest of the team to be more productive (thus keeping them in a productive state)
• The thing I'm working on is super-critical for customer satisfaction (usually that means it has to be done quickly and with little risk of failure)
• The thing I'm working on has poorly-understood requirements, requiring someone with a high degree of domain knowledge to make quality requirement decisions while simultaneously doing development (one could argue that in this case, my inability to adequately form the requirements is the real shortcoming.)
• helping the team most efficiently and effectively turn collective time in to software that the business can sell for a profit.

However it's debatable but i believe it will works well and rocking. willing to know what others thinking ?

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning."

– Rich Cook

challenging SQL queries

Recently got chance to work on a typical query. Sql query was find out second highest salary of employee without using max ? I think you guys are aware oracle default employee and department tables. so after some brainstorming finally got cool solution.
Here i am mentioning few more queries for fun. Hope you will enjoy.

1. display the all employees whom salary is greater than average salary of their own department and average salary if their department should be greater than the highest salary of department that has lowest no of employee?
2. display department wise first three highest salary ?
3. display 3rd lowest paid salary got employees detail ?
4. display the details of lowest paid manager ?
5. display the details of employees who having highest salary in diff dept ?
6. display the details of employee to whome max numbers of employees reporting ?
7. display the details of having highest salary details belonging department having highest no of employees ?
8. display the details employees if department have highest no of employees ?
9. select highest salary without using max ?

If you have any devastating query , you can post here.

List of goals 2011

Another year has passed and this has been the not best yet.Need to understand what I have achieved or where I need to improve to better myself, so this year I decided to set myself a list of goals to help shape my life. My list was split into for sections: Personal, Fitness, Programming and Office Job/Business so I will follow that theme with this review.
Personal
This part was about bettering myself as a person. Gaining new experiences, seeing new things and opening my mind,specially would like say thanks to my current employer where i learnt so many things.
Fitness
Total letdown... increased so huge extra pounds .. now feels that it's create big problem for me. so need to make separate plan for it .. anybody who can help please send me mail.
Programming
The coding,THE CODING!!! THE CODING !!! IT HASN'T STOPPED! :-)
This year is about coding new things and breaking out of my stagnant slump of 100% PHP and working on CodeIgniter. As for learning new languages, I left this a little late. 
So many things are in my Todo list like RoR,Python and Objective C.Sadly till i goes as far as Hello World! only and still i have not touch surface of any of these but i am sure and have confident that i will be a good iPhone App developer in the this Year [still 12 months left ;-) ]
Actually i am planing to write my own framework and for this i have did some basic ground level study like what developer want and what are the current issues like performance, latency issue, page rendering and js and css issues and inline js css issues as well as heavy page size. i also seen good features in Kohana,CodeIgniter, Rails,Yii and a few others. main features i would like introduce are object caching and provide ability to end user for design it's own modeling .. like whatever design pattern user want to use , he/she will be easily integrate it.Planing for two different MVC approaches one for forntend and another one is for backend. avoiding unnecessary loc code like input filters etc when php provide better extensions for it. just install extension and configure ini file and it works same as PDO etc .. so overall idea is provide fat free, robust and fast. Let's hope by end of this year, i will launch the BETA.
But for now i am looking for 1-2 ninja developers who can work with me and help out. if you think you have ability to work for it please send me mail at tussion at ymail dot com.Where does that I leave CodeIgniter? Well I am still loving it "long-time" since last 3 years and still i am working on it .. at least in my office.
Business/Job
Now for all this open-source coding,blogging,consultancy  i still need to make good number of money.'money hai to honey hai' ... :-).In all, Hopefully business is looking good for 2011.
Goals For 2011:-
1. Reduce upto 15 Kgs over in year.
2. Plan to kick-ass structural unemployment.. looking for rock solid organization...specially wants to work on android/Iphone based apps,cloud computing,nosql such untouch areas.I know it's not easy .. but let's Try.Plan for double blog traffic over the next year. If I start to struggle i can just post photos of top indian heroines ;-)
3. Kayak to hills specially wants to see around from pinnacle of mountain.       http://student.himadventures.net/climbing_courses_students.htm
4. planning to purchase Android,I phone .. 
5. Leave tobacco
6. Need to improve better polymorphous behaviour in real life.
7. Courage. Fear the unknown less. (Trust God more)
8. Design matters. (Self Branding)
9. Better budgeting, better planning (2010 was worst for me so will take care about it)
10. Using my time better in down time (ie. particularly Saturday and Sunday, but all of it)
11. Less time watching discovery

Know more about your first enemy Latency

Today morning i posted an status msg in Facebook about network delay .. 


The speed of light has a fixed upper limit. In fibre, this is about 200,000 Km/s, and it’s about the same for electricity through copper. This means that a signal sent over a cable that runs 2754 Km from Jammu to Kanyakumari would take about 14ms to get through...




So we can calculate a round trip time would be ~ 28ms .. and this is for one cycle only.. suppose it will be happened 250 times then round trip time will be 3 sec. i think .. so you got clear idea .. how latency delay effect HTTP request and response drama :-)

French ISPs have been advertising for years about being able to reach 24Mb/s with DSL lines. Why even bother about website performance if they think everyone has 24Mb/s available?


Bandwidth is actually how much data we can transfer at once. Latency is how fast a byte of data will travel end to end (length of the road and speed of the cars).Here we are discussing about the round trip time:- the latency to travel back and forth.


Latency (round trip time) depends mainly on the distance between you and your peers.
Take that distance, divide by the speed of light, divide again by 66% (slowness of a light fiber), and multiply by two to have back and forth. Then add about 10 to 20ms for your hardware, your ISP infrastructure, and the web server hardware and network. You will have the minimal latency you may hope for (but never reach):-


Latency (round trip) = 2 x (distance) / (0.66 x speed of light) + 20ms


For example In France(France has better figures than most other countries, so please expect worse than this figures) the latency of usual DSL lines goes from 30ms (French websites and CDNs) to 60-70ms (big players in Europe).We can expect 100 to 200ms for an US website with no relay in Europe.


3G phone networks usually have an added tax of 100ms, sometimes more. VPN, bad proxies, antivirus software, badly written portals and badly set up internal networks may also noticeably
increase latency.


Big companies have their own private "serious" direct connection to Internet. They also often have (at least in France) networks with filtering firewalls, complex architecture between the head office and branch offices, and sometimes overloaded switches and routers. You can expect an added tax of 50ms to 250ms compared to a simple DSL line.


So 50ms is really small, so latency isn’t so important, is it??


Round trip time is a primary concern.Mostly, your browser waits; it waits because of the latency.
When you do a request, you have to wait a few milliseconds to let the server generate the response, but also a few to have your request and its response travel back and forth. Each time you perform a request, you will have to wait one round trip time.


Suppose an website home page required 250 requests and Microsoft Internet Explorer 7 has two parallel download queues, so that’s 125 requests each. With a standard round trip time of 60ms we will be assured to wait at least 7.5 seconds before the page fully loads.Then we have to add the time needed to download and process the files themselves.


It's All about TCP Game ??


TCP is the protocol we use to connect to a web server and then send it our request. It’s like when you’re chatting on the phone: you never directly tell what you plan to tell, you first say "hello", wait for your peer to say "hello", then ask an academic "what’s up?" and wait for an answer (that you probably won’t even listen to but you will wait for it anyway). In Internet’s life, this courtesy is named TCP.TCP sends a "SYN" in place of "hello", and gets a "SYN-ACK" back, as an answer. The more latency you have, the more this initialization will take time.


DNS before TCP ??


That’s not all. Beforesaying “hello” to your friend on the phone, you have to dial his phone number. For internet it’s the IP address. Either your browser performed a request to the same domain a few seconds before and it may reuse the same result, or it has to perform a DNS request. This request may be in your ISP cache (cheap) or needs to be sent to a distant server (expensive if the domain name server is far away).


For each TCP connection, you will have to wait again, a time depending on the latency: latency to your ISP if you find a result on your ISP cache, latency to the DNS if not.


No !! IT's not end .. meet with UDP protocol :-)


DNS often uses the UDP protocol. UDP is a simple “quick and cheap” request/response protocol, with no need to establish a connection before like TCP has. However, when the response that weight over 512 bytes, it may either send a larger response (EDNS specification) or ask for TCP. Large DNS responses used to be rare in the past, but now DNS uses a security extension (DNSSEC) that requires larger responses.


The problem is that many badly configured firewalls still block DNS responses of more than 512 bytes. A few others will block the UDP fragmentation needed for response of more than 1.5KB (UDP fragmentation is a way to send the response with multiple UDP packets, as each one is limited in size). For short: You may well have UDP DNS requests first then a fallback to TCP.


If that happens, the client first requests in UDP, the server answers “please go on TCP”, client opens a TCP connection (SYN + SYN-ACK) and then asks again. In place of one round trip time, we now have three.


Finally ??


A simple 10KB image will need 3 round trips. jQuery (77KB) will need 7 round trips. At 60 to 100ms the round trip time, it is easy to understand that latency is far more important than anything else.
So we have seen that how how slow start and congestion control affect the throughput of a network connection. Each network roundtrip is limited by how long it takes photons or electrons to get through, and anything we can do to reduce the number of roundtrips should reduce total page download time, right? Well, it may not be that simple. We only really care about roundtrips that run end-to-end.Latency has been a problem whenever signals have had to be transmitted over a distance. Whether it is a rider on a horse, or electrons running through metal, each has had its own problems with it.

Third party tools that helps in optimization

Of course First name is YUI Compressor. it minifying both Java script and CSS files. The YUI Compressor needs Java to work, so you will need to be sure to have a Java runtime installed.


Second name is OptiPng. it's PNG optimization tools which you can run from the command line exist. checkout http://www.phpied.com/png-optimization-tools/ & http://optipng.sourceforge.net/


Third name is  CSSEmbed. It's tool to automatically embed images into CSS files as data URIs. This is a very small, simple tool that reads in a CSS file, identifies the images referenced within, converts them to data URIs, and outputs the resulting style sheet. The newly-created stylesheet is an exact duplicate of the original, complete with comments and indentation intact; the only difference is that all references to image files have been replaced with data URIs. download it from http://github.com/nzakas/cssembed/


Fourth one is used for optimization jpeg files using a tool like JPEGtran. it covers following tasks.

1. tripping meta data (meta is sometimes bulky and useless for web display)
2. Optimizing Huffman tables or
3. Convert a JPEG to progressive encoding

read more how use JPEGtran http://www.phpied.com/installing-jpegtran-mac-unix-linux/

Another hack to render heavy HTML pages

when HTML page is loaded, browser needs to do a lot work. It has to parse HTML, build elements collections (so things likegetElementsByTagName() can work faster), match CSS rules, etc., etc. And then, finally, render all those elements—you may know this process as repaint. Repainting is one of the slowest process in browsers.


One quick solution is that we don’t need to show all 500KB text at once, we can pick a first few sentences and push them to the screen so the user can start reading while browser parses the rest of the page.


How we do same ??


To make all this large text invisible to the browser, all we have to do is to comment it:-



<body>
    <!--
    <p>Well, LARGE HTML HERE ...</p>
    -->


</body>

When the text content was commented out, the page parsing took quickly.

So, we have a commented text, what now? Actually, HTML comment is not just a hidden code chunk, it’s a DOM node which can be easily accessed. Now we need to find this node and parse its content into a DOM tree:-

var elems = document.body.childNodes;

for (var i = 0, il = elems.length; i < il; i++) {

  var el = elems[i];

  if (el.nodeType == 8) { //it’s a comment

    var div = document.createElement('div');

    div.innerHTML = el.nodeValue;

    // now DIV element contains parsed DOM elements so we can work with them

    break;

  }

}

Since such plain text parsing doesn’t require browser to do CSS matching, repainting and other stuff that it normally does on page parsing, it also performs very fast.

Few more thoughts on script loaders in websites

Last week JS GURU Steve Souders (Google) released his ControlJS project. The goal of the project is that to provide freedom to developer to load js files and execute them later on a page as per user action.
In our shiksha.com, we already applied same technique. we load heavy dynamic pages in overlay (modal box) through AJAX but initially we encountered with one problem .. if we load a page with AJAX and suppose that page contain inline JS code .. then that JS code will not be executed. so finally we used some technique/ hack and solved issue.

Actually we parse whole inline JS and CSS that comes in script and css html tag and evaled it later once we get ajax success callback. here is code for same.

function ajax_parseJs(obj)
{
    var scriptTags = obj.getElementsByTagName('SCRIPT');
    var string = '';
    var jsCode = '';
    for(var no=0;no<scriptTags.length;no++){
        if(scriptTags[no].src){
            var head = document.getElementsByTagName("head")[0];
            var scriptObj = document.createElement("script");

            scriptObj.setAttribute("type", "text/javascript");
            scriptObj.setAttribute("src", scriptTags[no].src);
        }else{
            if(navigator.userAgent.indexOf('Opera')>=0){
                jsCode = jsCode + scriptTags[no].text + 'n';
            }
            else
                jsCode = jsCode + scriptTags[no].innerHTML;
        }

    }

    if(jsCode)ajax_installScript(jsCode);
}
function evaluateCss(obj)
{
   var cssTags = obj.getElementsByTagName('STYLE');
   var head = document.getElementsByTagName('HEAD')[0];
   for(var no=0;no<cssTags.length;no++){
      head.appendChild(cssTags[no]);
   }
}
function ajax_installScript(script)
{
    if (!script)
        return;
    if (window.execScript){
        window.execScript(script)
    }else if(window.jQuery && jQuery.browser.safari){ // safari detection in jQuery
        window.setTimeout(script,0);
    }else{
        window.setTimeout( script, 0 );
    }
}

So i thought that can we do same for script loading ? i think it's not a big deal to load script and execute it
when developer want. here is code for same.


function loadScript(url, callback){
    var script = document.createElement("script")
    script.type = "text/javascript";
    if (script.readyState){  //IE
        script.onreadystatechange = function(){
            if (script.readyState == "loaded" ||
                    script.readyState == "complete"){
                script.onreadystatechange = null;
                callback();
            }
        };
    } else {  //Others
        script.onload = function(){
            callback();
        };
    }
    script.src = url;
    document.getElementsByTagName("head")[0].appendChild(script);
}


var script = document.createElement("script");
script.type = "text/cache";
script.src = "foo.js";
script.onload = function(){
    //script has been loaded but not executed
};
document.body.insertBefore(script, document.body.firstChild);

//at some point later
script.execute();

Hope above techniques are clear and you don't have any doubts .. if still you have any query then write me on mail @ tussion @ ymail dot com

Happy coding ... Enjoy XMAS holidays ...

W3C DOM vs. innerHTML which is slower ?

We can check by running test script.


<div id="writeroot" style="width:1px; height:1px; overflow:hidden;"></div>
<script>
 function removeTable() {
  document.getElementById('writeroot').innerHTML = '';
 }
</script>


W3CDOM 1: Create all elements as needed:-



removeTable();
var x = document.createElement('table');
var y = x.appendChild(document.createElement('tbody'));
for (var i = 0; i < 20; i++) {
 var z = y.appendChild(document.createElement('tr'));
 for (var j = 0; j < 20; j++) {
  var a = z.appendChild(document.createElement('td'));
  a.appendChild(document.createTextNode('*'));
 }
}
document.getElementById('writeroot').appendChild(x);

Result is 55 % slower as compare others.

W3CDOM 2: Create elements once, then clone:-


removeTable();
var x = document.createElement('table');
var y = x.appendChild(document.createElement('tbody'));
var tr = document.createElement('tr');
var td = document.createElement('td');
var ast = document.createTextNode('*');
for (var i = 0; i < 20; i++) {
 var z = y.appendChild(tr.cloneNode(true));
 for (var j = 0; j < 20; j++) {
  var a = z.appendChild(td.cloneNode(true));
  a.appendChild(ast.cloneNode(true));
 }
}
document.getElementById('writeroot').appendChild(x);

Result is 36 % slower as compare others.

tableMethods:-


removeTable();
var x = document.createElement('table');
var y = x.appendChild(document.createElement('tbody'));
for (var i = 0; i < 20; i++) {
 var z = y.insertRow(0);
 for (var j = 0; j < 20; j++) {
  var a = z.insertCell(0).appendChild(document.createTextNode('*'));
 }
}
document.getElementById('writeroot').appendChild(x);

Result is 50 % slower as compare others.

INNERHTML 1: concatenate one string:-


removeTable();
var string = '<table><tbody>';
for (var i = 0; i < 20; i++) {
 string += '<tr>';
 for (var j = 0; j < 20; j++) {
  string += '<td>*</td>';
 }
 string += '</tr>';
}
string += '</tbody></table>';
document.getElementById('writeroot').innerHTML = string;

Result is 5 % fastest as compare others.

INNERHTML 2: push and join:-


removeTable();
var string = new Array();
string.push('<table><tbody>');
for (var i = 0; i < 20; i++) {
 string.push('<tr>');
 for (var j = 0; j < 20; j++) {
  string.push('<td>*</td>');
 }
 string.push('</tr>');
}
string.push('</tbody></table>');
var writestring = string.join('');
document.getElementById('writeroot').innerHTML = writestring;

Result is 2% slower than others tests.

Actual results are as follows.
Columns as as follows. innerHTML1,innerHTML2,W3CDOM 1,W3CDOM 2,tableMethods and No of Tests


Chrome 8.0.552 197 194 617 647 634 10
Chrome 9.0.597 175 180 349 362 398 5
Chrome 10.0.612 202 207 743 718 684 3
Firefox 3.6.11         93 90 81 71 79 1
Firefox 3.6.12         208 204 177 150 172 4
Firefox 3.6.13         115 112 105 86 106 3
Firefox Beta
 4.0b7                786 696 508 409 378 7
IE 6.0                20 84 18 19 10 8
IE 8.0                240 234 43 47 47 9
iPhone 4.2.1       18 19 47 49 49 1
Opera 11.00       772 752 347 491 383 1
Safari 5.0.2       190 196 616 607 589 1
Safari 5.0.3       209 219 623 595 584 11

So Finally inner HTML won and it's fastest among all methods.