Number of bugs being filed is very high

We have discussed last week about total number of bug count and how can we reduce bugs. It was an open discussion, and I really enjoyed people's comments.

Few major reasons described by people are:

• UI Issues
• Legacy code issue/Existing functionality issues
• KLOC rule/BUGS per 1000 LOC
• Bug logging categorization ( one bug but reported many times )
• Strict timeline
• Lack of proper unit testing/integration testing cycle
• Lack of impact area analysis

Before discussing on these points in detail let's have a look on what is actual meaning of "so many bugs". 

One hair in soup is too many and in head too few. It all depends. There is no context to judge it. Firefox's each release contains about 1000 open bugs, but it is considered to be quite usable.

Perception of the quality of a product can vary greatly between the programmer and other user. Developers and users/business people have very different views of what bugs mean. For a developer, every bug found and fixed increases the developer's confidence in the code, since it is getting more robust. For a user, every bug found (even if fixed) decreases their confidence.

If you are judging only by the bug count found in testing, then that is unfortunate.

I have seen may products which are judged  on the basis of user's experience and it plays a big role however there were no bugs are reported from QA end.

Few products were reported as buggy (from product due to complexity of product) but they are generating good percentage amount of revenue and running smoothly however no bugs are reported from QA end.

Sometimes End user reported few issues which are basically marked as missing features most of times and in few cases it was closed due to wrong manipulation of data analysis.
In few cases  such bugs were closed by providing data in Excel to client or closed by discussion with sales person and clients.

UI Issues: I believe that HTML developer provide W3C validated HTML code and HTML also verified via QA Team. Developer when integrate HTML,  code will become buggy.
There are many reasons for UI issues.

Developer avoid cross browser testing with all versions of browses.(see browser plus OS matrix) 

Sometimes developers write own front-end logic which is already written and cross checked but newly written code fails as per standard that follow through out website.

KLOC rule/BUGS per 1000 LOC reason is not fit in our case(website development) because if we look into codebase and nature of bugs 60-80 % of code lines were captured by HTML of code associated with front-end page rendering logic.

Another point is in house testing  is not also so much focusing on security testing, environmental issues or load and performance testing.

Some times we have contributed more by removing code than adding ;-)

In our last release we spent good amount of time to discover that code required pdo_mysql extension and few code APIs are behaving unexpected results in 64 bit machine.

Here bootstrap plays a powerful role.  https://github.com/twitter/bootstrap  (Bootstrap is a sleek, intuitive, and powerful front-end framework for faster and easier web development.)

Lets start look seriously at front-end part, try to avoiding  unmoduler JS and HTML codebase.

Legacy code issue/Existing functionality issues : 

Always Fix bugs before write new code and mark such bugs with a new tag in bugzilla.

see here http://www.joelonsoftware.com/articles/fog0000000043.html

Lack of impact area analysis:

It might be that the requirements weren't clear or correct, and that was reflected in the code and picked up by the testers.
Failure mode and effects analysis should be a part of designing.
Some how we can say we should follow more  tighter software development processes. During writing code, always think in mind that we are building software of  medical or military environment. 

Other important areas are Static code checking, peer code reviews, unit testing, component testing, system tests. It's easy to get a couple of hundred lines of code per day but try to get a couple of hundred quality lines of code per day and it's not so easy.

Is microtime enough to get consumed time in software application

I heard that few engineers are debating on to find out actual time consumption in profile page and specially accuracy in php micortime. A sort of benchmarking that tells how much CPU utilization were involved in new peaces of code.

We live in a time of multi-tasking,multi processors and background processing. So debating on microtime or cpu usages is worthless if you are working for web application. Each and every measurement and benchmarking is required to make robust and reliable application.

Lets back to system time,wall time debate.

Open a browser for one hour doesn’t mean that computer has spent an hour of dedicating it’s resources to the browser.(unless it’s IE of course…:P)

Also its depend on application. APC,memcached,varnish,gearman,cassandra etc can divert the result to benchmark application as well as estimated results.

There are three different types of times which was taken by application.

1. System time(processor spends on behalf of the current process)
2. User time (time spend in executing code in user mode)
3. Wall time (Amount of time that passes on our wall clock while process runs)

The advantage of system time is it will give the amount of time that any subproccesses right down to the low lever system processes (ie. file i/o, socket operations) take. This can also be a
disadvantage though because it may make it harder to determine if a bottleneck is happening in your code or in some sub-system your code uses.

getrusage() information is CPU time used and microtime() is wall clock time. The program may run for 10 minutes according to the clock on the wall, but internally may only use a few seconds of CPU
time. Then there's contending for CPU time with all the background programs running on the system, resource contention, plus regular housekeeping.

There's far too many factors involved to be able to get an accurate timing for such short periods. Doing three runs of the while(microtime()) version of loop, I got the following timings:

user: 0.98, 0.09, 0.90 sys: 0.12, 0.05, 0.94

Obviously quite the variance. Even just a simple has utime/stimes ranging from 0 to 0.03.

Try running application longer periods, and do something within them to increase cpu usage.

API in PHP to check server Load
// make sure safe mode shud be off

function ServerLoad()
{
    $stats = exec('uptime');
    preg_match('/averages?: ([0-9.]+),[s]+([0-9.]+),[s]+([0-9.]+)/', $stats, $regs);
    return ($regs[1].', '.$regs[2].', '.$regs[3]);
}

Happy Coding !!! Enjoy !

design components of distributed application architecture

Its requires time to discuss in details, however this post is in draft state but i figured out 6 important layers in architecture. Please go thrugh and feel free suggest if you have any better suggestions.

1. Create a web service with something like Apache Axis
2. Use an ESB - something like Mule or JBoss
3. Use a simple web Servlet on the server, and submit data using HTTP POST. You could use a simple embeddable Java web server like Jetty to do this.
4. Use a messaging protocol like Kryonet or Google's protocol buffers
5. Use a more general network application framework such as Netty

web based business development and node js

I really shocked when i install node js and run few test scripts. it's tremendous and powerful and gives wings to do anything. we can build chat server in few minutes ... can build web server,TCP/IP service,Message Queues,DB connections etc. 


Main advantages are:

1. Web development in a dynamic language (JavaScript) on a VM that is incredibly fast (V8). It is much faster than Ruby, Python,PHP or Perl.
2. Ability to handle thousands of concurrent connections with minimal overhead on a single process.
3. JavaScript is perfect for event loops with first class function objects and closures. People already know how to use it this way having used it in the browser to respond to user initiated events.
4. A lot of people already know JavaScript, even people who do not claim to be programmers. It is arguably the most popular programming language.
5. Using JavaScript on a web server as well as the browser reduces the impedance mismatch between the two programming environments which can communicate data structures via JSON that work the same on both sides of the equation. Duplicate form validation code can be shared between server and client, etc.

 While i am elaborating things , will share few more things with screen shots. which might be intrested to you people to understand magic of NODE JS.

File based caching on high traffic website

If you have a lot of traffic, and you’re caching a page for an hour… Over the course of one hour, you may read the file possibly 200 times, and write just once… The time taken to write is minimal, around 10ms? - So, out of one hour, you have 10ms ‘downtime’ for your cache reads.

Which is 0.00027% downtime… Extremely minimal, and not really worth worrying about.
Locks will only cause this fractional locking time, and, if it only effects 2 people trying to write at the same time.

The correct way to work around it is explained here:-

* 1 page is cached for an hour.
* When that cache expires, 2 people visit the site with 10ms of each other (the only time this will cause an ‘issue’, because otherwise, the first will have written the cache before the second requests the page).
* The first user ‘reads’ the cache file, sees it’s expired so begins rendering the page normally with PHP/MySQL
* The second user has the same thing going on… the cache hasn’t been updated by the second user.
* The first user finishes building the page, and begins writing the file, locking it.
* The second user also finishes building the page, and attempts to write it.
* Because the first user has locked the file, the second user can’t write it…
* This is a problem for the second user, so you simply build a little fix to make it all work like magic.

If the second user can’t write the the cache (because it is locked), simply echo the contents of the rendered page, not saving it to the cache.
The first user is dealing with the cache, so just render the output.

With Codegniter, this is how it works…

Line 299 of Output.php:-


if ( ! $fp = @fopen($cache_path, 'wb'))
        {
            log_message('error', "Unable to write cache file: ".$cache_path);
            return;
        }

The cache file is only written if it’s ‘really_writable’ (not locked)

Line 59 of Output.php:


elseif (($fp = @fopen($file, 'ab')) === FALSE)
    {
        return FALSE;


So… if we cant write it, just render the second users’ request.

Simple, and it works, beautifully... :-)

Know more about your first enemy Latency

Today morning i posted an status msg in Facebook about network delay .. 


The speed of light has a fixed upper limit. In fibre, this is about 200,000 Km/s, and it’s about the same for electricity through copper. This means that a signal sent over a cable that runs 2754 Km from Jammu to Kanyakumari would take about 14ms to get through...




So we can calculate a round trip time would be ~ 28ms .. and this is for one cycle only.. suppose it will be happened 250 times then round trip time will be 3 sec. i think .. so you got clear idea .. how latency delay effect HTTP request and response drama :-)

French ISPs have been advertising for years about being able to reach 24Mb/s with DSL lines. Why even bother about website performance if they think everyone has 24Mb/s available?


Bandwidth is actually how much data we can transfer at once. Latency is how fast a byte of data will travel end to end (length of the road and speed of the cars).Here we are discussing about the round trip time:- the latency to travel back and forth.


Latency (round trip time) depends mainly on the distance between you and your peers.
Take that distance, divide by the speed of light, divide again by 66% (slowness of a light fiber), and multiply by two to have back and forth. Then add about 10 to 20ms for your hardware, your ISP infrastructure, and the web server hardware and network. You will have the minimal latency you may hope for (but never reach):-


Latency (round trip) = 2 x (distance) / (0.66 x speed of light) + 20ms


For example In France(France has better figures than most other countries, so please expect worse than this figures) the latency of usual DSL lines goes from 30ms (French websites and CDNs) to 60-70ms (big players in Europe).We can expect 100 to 200ms for an US website with no relay in Europe.


3G phone networks usually have an added tax of 100ms, sometimes more. VPN, bad proxies, antivirus software, badly written portals and badly set up internal networks may also noticeably
increase latency.


Big companies have their own private "serious" direct connection to Internet. They also often have (at least in France) networks with filtering firewalls, complex architecture between the head office and branch offices, and sometimes overloaded switches and routers. You can expect an added tax of 50ms to 250ms compared to a simple DSL line.


So 50ms is really small, so latency isn’t so important, is it??


Round trip time is a primary concern.Mostly, your browser waits; it waits because of the latency.
When you do a request, you have to wait a few milliseconds to let the server generate the response, but also a few to have your request and its response travel back and forth. Each time you perform a request, you will have to wait one round trip time.


Suppose an website home page required 250 requests and Microsoft Internet Explorer 7 has two parallel download queues, so that’s 125 requests each. With a standard round trip time of 60ms we will be assured to wait at least 7.5 seconds before the page fully loads.Then we have to add the time needed to download and process the files themselves.


It's All about TCP Game ??


TCP is the protocol we use to connect to a web server and then send it our request. It’s like when you’re chatting on the phone: you never directly tell what you plan to tell, you first say "hello", wait for your peer to say "hello", then ask an academic "what’s up?" and wait for an answer (that you probably won’t even listen to but you will wait for it anyway). In Internet’s life, this courtesy is named TCP.TCP sends a "SYN" in place of "hello", and gets a "SYN-ACK" back, as an answer. The more latency you have, the more this initialization will take time.


DNS before TCP ??


That’s not all. Beforesaying “hello” to your friend on the phone, you have to dial his phone number. For internet it’s the IP address. Either your browser performed a request to the same domain a few seconds before and it may reuse the same result, or it has to perform a DNS request. This request may be in your ISP cache (cheap) or needs to be sent to a distant server (expensive if the domain name server is far away).


For each TCP connection, you will have to wait again, a time depending on the latency: latency to your ISP if you find a result on your ISP cache, latency to the DNS if not.


No !! IT's not end .. meet with UDP protocol :-)


DNS often uses the UDP protocol. UDP is a simple “quick and cheap” request/response protocol, with no need to establish a connection before like TCP has. However, when the response that weight over 512 bytes, it may either send a larger response (EDNS specification) or ask for TCP. Large DNS responses used to be rare in the past, but now DNS uses a security extension (DNSSEC) that requires larger responses.


The problem is that many badly configured firewalls still block DNS responses of more than 512 bytes. A few others will block the UDP fragmentation needed for response of more than 1.5KB (UDP fragmentation is a way to send the response with multiple UDP packets, as each one is limited in size). For short: You may well have UDP DNS requests first then a fallback to TCP.


If that happens, the client first requests in UDP, the server answers “please go on TCP”, client opens a TCP connection (SYN + SYN-ACK) and then asks again. In place of one round trip time, we now have three.


Finally ??


A simple 10KB image will need 3 round trips. jQuery (77KB) will need 7 round trips. At 60 to 100ms the round trip time, it is easy to understand that latency is far more important than anything else.
So we have seen that how how slow start and congestion control affect the throughput of a network connection. Each network roundtrip is limited by how long it takes photons or electrons to get through, and anything we can do to reduce the number of roundtrips should reduce total page download time, right? Well, it may not be that simple. We only really care about roundtrips that run end-to-end.Latency has been a problem whenever signals have had to be transmitted over a distance. Whether it is a rider on a horse, or electrons running through metal, each has had its own problems with it.

Third party tools that helps in optimization

Of course First name is YUI Compressor. it minifying both Java script and CSS files. The YUI Compressor needs Java to work, so you will need to be sure to have a Java runtime installed.


Second name is OptiPng. it's PNG optimization tools which you can run from the command line exist. checkout http://www.phpied.com/png-optimization-tools/ & http://optipng.sourceforge.net/


Third name is  CSSEmbed. It's tool to automatically embed images into CSS files as data URIs. This is a very small, simple tool that reads in a CSS file, identifies the images referenced within, converts them to data URIs, and outputs the resulting style sheet. The newly-created stylesheet is an exact duplicate of the original, complete with comments and indentation intact; the only difference is that all references to image files have been replaced with data URIs. download it from http://github.com/nzakas/cssembed/


Fourth one is used for optimization jpeg files using a tool like JPEGtran. it covers following tasks.

1. tripping meta data (meta is sometimes bulky and useless for web display)
2. Optimizing Huffman tables or
3. Convert a JPEG to progressive encoding

read more how use JPEGtran http://www.phpied.com/installing-jpegtran-mac-unix-linux/

Another hack to render heavy HTML pages

when HTML page is loaded, browser needs to do a lot work. It has to parse HTML, build elements collections (so things likegetElementsByTagName() can work faster), match CSS rules, etc., etc. And then, finally, render all those elements—you may know this process as repaint. Repainting is one of the slowest process in browsers.


One quick solution is that we don’t need to show all 500KB text at once, we can pick a first few sentences and push them to the screen so the user can start reading while browser parses the rest of the page.


How we do same ??


To make all this large text invisible to the browser, all we have to do is to comment it:-



<body>
    <!--
    <p>Well, LARGE HTML HERE ...</p>
    -->


</body>

When the text content was commented out, the page parsing took quickly.

So, we have a commented text, what now? Actually, HTML comment is not just a hidden code chunk, it’s a DOM node which can be easily accessed. Now we need to find this node and parse its content into a DOM tree:-

var elems = document.body.childNodes;

for (var i = 0, il = elems.length; i < il; i++) {

  var el = elems[i];

  if (el.nodeType == 8) { //it’s a comment

    var div = document.createElement('div');

    div.innerHTML = el.nodeValue;

    // now DIV element contains parsed DOM elements so we can work with them

    break;

  }

}

Since such plain text parsing doesn’t require browser to do CSS matching, repainting and other stuff that it normally does on page parsing, it also performs very fast.

Few more thoughts on script loaders in websites

Last week JS GURU Steve Souders (Google) released his ControlJS project. The goal of the project is that to provide freedom to developer to load js files and execute them later on a page as per user action.
In our shiksha.com, we already applied same technique. we load heavy dynamic pages in overlay (modal box) through AJAX but initially we encountered with one problem .. if we load a page with AJAX and suppose that page contain inline JS code .. then that JS code will not be executed. so finally we used some technique/ hack and solved issue.

Actually we parse whole inline JS and CSS that comes in script and css html tag and evaled it later once we get ajax success callback. here is code for same.

function ajax_parseJs(obj)
{
    var scriptTags = obj.getElementsByTagName('SCRIPT');
    var string = '';
    var jsCode = '';
    for(var no=0;no<scriptTags.length;no++){
        if(scriptTags[no].src){
            var head = document.getElementsByTagName("head")[0];
            var scriptObj = document.createElement("script");

            scriptObj.setAttribute("type", "text/javascript");
            scriptObj.setAttribute("src", scriptTags[no].src);
        }else{
            if(navigator.userAgent.indexOf('Opera')>=0){
                jsCode = jsCode + scriptTags[no].text + 'n';
            }
            else
                jsCode = jsCode + scriptTags[no].innerHTML;
        }

    }

    if(jsCode)ajax_installScript(jsCode);
}
function evaluateCss(obj)
{
   var cssTags = obj.getElementsByTagName('STYLE');
   var head = document.getElementsByTagName('HEAD')[0];
   for(var no=0;no<cssTags.length;no++){
      head.appendChild(cssTags[no]);
   }
}
function ajax_installScript(script)
{
    if (!script)
        return;
    if (window.execScript){
        window.execScript(script)
    }else if(window.jQuery && jQuery.browser.safari){ // safari detection in jQuery
        window.setTimeout(script,0);
    }else{
        window.setTimeout( script, 0 );
    }
}

So i thought that can we do same for script loading ? i think it's not a big deal to load script and execute it
when developer want. here is code for same.


function loadScript(url, callback){
    var script = document.createElement("script")
    script.type = "text/javascript";
    if (script.readyState){  //IE
        script.onreadystatechange = function(){
            if (script.readyState == "loaded" ||
                    script.readyState == "complete"){
                script.onreadystatechange = null;
                callback();
            }
        };
    } else {  //Others
        script.onload = function(){
            callback();
        };
    }
    script.src = url;
    document.getElementsByTagName("head")[0].appendChild(script);
}


var script = document.createElement("script");
script.type = "text/cache";
script.src = "foo.js";
script.onload = function(){
    //script has been loaded but not executed
};
document.body.insertBefore(script, document.body.firstChild);

//at some point later
script.execute();

Hope above techniques are clear and you don't have any doubts .. if still you have any query then write me on mail @ tussion @ ymail dot com

Happy coding ... Enjoy XMAS holidays ...

Difference among all these Load/Utilization/Scalability/Throughput/Concurrency/Capacity?

X = Time/Task, R = Time/Task

Load:- how much work is incoming? or, how big is the back log?

Utilization:- how much of a system's resources are used?

Scalability:- what is the relationship between utilization and R?

Throughput:- X - how many tasks can be done per unit of time?

Concurrency:- how many tasks can we do at once?

Capacity:- how big can X go without making other things unacceptable?

More about frontend optimization

Generally people thought that they know Yahoo 14 Rules so it's easy to optimize page rendering and one and only one solution is Ajaxify that page. But it's doesn't mean all websites are rendered fast in web. as i seen and now strongly believe that  every website has it's own unique solution to fix their speed related issue and mainly people are unable to identify where is bottleneck. recently i have seen what  FB did ? 


In Facebook case, it's not easy to handle 500M Users.When Average hours per month per user is more than 5 hours.. google & yahoo has less than 2 hours only. FB has complex Frontend Infrastructure. FB runs 2 JS demons to handle Real time updates and Cache consistency.Main tasks are Incremental updates,In-page writes,Cross-page writes.Every state-changing operations are recorded and send to backend. Backend check when a write operation is detected, send a signal to the client to invalidate the cache.So usually user browses FB with three version. 1.cached version 2.state-changing version 3.Restored version.


This is one frontend solution that FB uses for caching other big ones are "BIG PIPE" and "Quickling" these are very advance techniques that are still need for me to understand
 but what i know till now is :-


1.  Use Network Latency and Page rendering delta time  for other work
2.  Try to reduce domcontentloaded and window load time gap
3.  Use AJAX but in smart way .. mean Time-to-Interact should be very less i.e. page would fully render, but be frozen,User can't interact while JavaScript is being fetched/parsed/executed


Performance is hard so please think twise before move ahead .. now come  to AJAX .. everybody know how use AJAX but very less people know AJAX design pattern :P


When AJAX call occurred it goes into following steps:-


1. Round trip Time: The first step is the amount of time between when the browser sends
the request and the time it receives the response
2. Parse Time: Next, the response returned from the server has to be parsed
3. JavaScript/CSS Download Time: [I believe you are smarter so you will download Widget's JS and CSS file lazy ;) ] Each response can indicate it needs more JavaScript/CSS before the
content can be used
4. Render Time : The amount of time it takes to actually change the display via innerHTML


So i would like know what solutions you have to fix above four issues ? how you optimize these areas ?


Best of Luck !!! Happy Coding !!!

Serve Pre-Generated Static Files Instead Of Dynamic Pages

Well .. still i am not sure but i read few articles about how scale site performance without putting some extra efforts.i think it's cheapest terminology i have seen ever. We know well that static files have the advantage of being very fast to serve. Read from disk and display. Simple and fast. Especially when caching proxies are used. The issue is how do you bulk generate the initial files, how do you serve the files, and how do you keep the changed files up to date? specially regenerate static pages when changes occur... When a new entity  is added to system hundreds of pages could be impacted, which would require the effected static pages to be regenerated.It's a very pragmatic solution and rock solid in operation. see more detail http://eventseer.net/p/thomas_brox_roest/whiteboardentry/13/ 

Web site Front-end optimization headaches

I have seen that typically 80-90% of page response time comes from other things than fetching HTML of the page :- fetching CSS, JavaScript, Images,banners which makes it number one thing to focus your optimization. Sharing few points, specially for front-end optimization. Hope this will be helpful.

1. Try to split initial payload .. first load all required js to load page then other scripts/things.

Lazyload,ajaxload,load scripts in parrller,load script after on windowload, these are different things and you must know about it.When the browser starts downloading an external script, it won’t start any additional downloads until the script has been completely downloaded, parsed, and executed. So load external scripts asynchronously only. we also need to care about inline JS, here we can use callback functions,timers,on window load etc see following chart , here few techniques are listed to load JS scripts.

2. Flushing the Document Early:-

 As the server parses the PHP page, all output is written to STDOUT. Rather than being sent immediately, one character, word, or line at a time, the output is queued up and sent to the browser in larger chunks. This is more efficient because it results in fewer packets being sent from the server to the browser. Each packet sent incurs some network latency, so it’s usually better to send a small number of large packets, rather than a large number of small packets. Calling flush() causes anything queued up in STDOUT to be sent immediately. How- ever, simply flushing STDOUT isn’t enough to achieve the type of speedup experienced in the preceding example. The call to flush has to be made in the right place. idea is , flush before a long time taking process, it can be divide as module wise also like header,footer,sidebar,widgets etc.

3. Repaint and Reflow is costly:-

A repaint occurs when a visual change doesn't require recalculation of layout Changes to visibility, colors (text/background), background images, etc.

  

  

A reflow occurs when a visual change requires a change in layout Initial page load

 browser resize,DOM structure change,layout style change layout information retrieved

 

 

4. Eliminate unnecessary cookies:- 

Keep cookie sizes as low as possible to minimize the impact on the user response time,Be mindful of setting cookies at the appropriate domain level so other sub-domains are not affected

5. DOM manipulations are the slowest,Never update the DOM, if that's not possible, at least do it as infrequently as possible. Bunch up your updates to the DOM and save them for a later time.

6. Clone the DOM node you want to work with. Now you will be working with a clone of the real node, and the cloned node doesn't exist in the DOM. Updating the cloned node doesn't affect the DOM. When you are done with your manipulations, replace the original node with the cloned node.

7. However, note that the performance problems in point 4, because of the content and rendering reflow that the browser has to do. You might get similar benefits by simply hiding the element first, making the changes, and then showing it.

Which Javascript frameworks are the most common and why ?

The frameworks i looked for this article were Prototype, JQuery, MooTools, Yahoo! UI Library, Dojo, ExtJS and MochiKit.

Prototype

Prototype is one of the earlier Javascript frameworks.Of the websites in this test, a total of 13 used the Prototype framework.

• CNN
• The New York Times
• Digg
• Apple
• Veoh.com
• TypePad
• Fox News Channel
• Finetune
• iLike
• Last.fm
• Twitter
• Hakia
• YouSendIt

JQuery

JQuery is a framework that has received a lot of attention due to its speed, size and smart modular approach which has led to a big library of plugins. Of the websites in this test, 11 used the JQuery framework.

• Digg
• BBC
• Major League Baseball
• Dell
• IsoHunt
• Break.com
• TinyPic
• FixMyMovie
• eMusic
• Kayak
• Box.net
• code.google.com

MooTools

Just like other Javascript frameworks, MooTools contains several functions to help development. One of the more known ones is its advanced effects component. Of the websites in this test, four used the MooTools frameworks.

• Mint
• Worth1000
• Bebo
• Vimeo
• Main
• Pingdom
• Outages
• Guest posts

Yahoo! UI Library (YUI)

Yahoo has developed its own Javascript framework. They use it for their own websites, but have also made it freely available to others. Of the websites in this test, seven used the Yahoo! UI Library.

• Yahoo! Shopping
• Zillow
• Bebo
• YouSendIt
• ImageShack
• LinkedIn
• Walmart.com

The ones using more than one framework were Digg (Prototype and JQuery), Bebo (MooTools and YUI) and YouSendIt (Prototype and YUI).

Prototype turned out to be the most-used framework in this survey, and of course it takes cake but JQuery not far behind. Please see following links regarding who is best.

http://blog.solnic.eu/2007/11/11/jquery-vs-prototype-part-i

http://blog.solnic.eu/2008/2/3/jquery-vs-prototype-part-ii

finally i must say that Jquery and prototype are both great libraries. I think now jQuery's philosophy (type less, do more, keeping things intuitive and unobtrusive) will make a big difference.

size is always a concern. Most important is speed :) and again prototype got success here.
we can get it as a 14.4kb gzipped version file.

http://groups.google.com/group/prototype-core/browse_thread/thread/ef05ede819727d52
http://ajax.googleapis.com/ajax/libs/prototype/1.6.0.3/prototype.js
http://groups.google.com/group/prototype-core

but at last we can not close eyes. i think now Jquery is really starting to overtake prototype and prototype is losing ground. No matter how great of a library is, with the current strategy , this fantastic lib will not survive.prototype should have components not a whole lot but atleast a few basic ones like jquery’s tabs, accordion ect.. i request to prototype dev team to do something about it because people are starting to get woried !!!

APD - Advanced PHP Debugger

APD is a debugger written in C . It loads as an extension to the Zend Engine. It works by hooking into the Zend internals and intercepting PHP function calls, allowing it to do things like measure function execution time, count function calls, perform stack backtraces and other things.

Installing APD

Currently, three main ways exist to install APD on a Linux system: grab the source and compile it yourself, use PEAR or use the Debian package.

1. pecl install apd
2. Edit php.ini file
zend_extension = /usr/lib/php/modules/apd.so
apd.dumpdir = /var/log/pprof
3. Don't forget restart Apache :)

Test & debugging

First make sure APD is is installed properly ... for this check phpinfo() API. Now add apd_set_pprof_trace() API where you would like debug.

Gathering Some Data

Look at apd.dumpdir folder . pprof tracefile has been written out here. A pprof tracefile is a text file that contains a machine-parsable summary of how your PHP was processed. suppose output file is pprof.08871.0 ,The number is the process ID of the Web server process that handled the request.

Interpreting The pprof Tracefile

APD comes with a little shell script written in PHP, called pprofp, that can be run from the command line to parse the pprof tracefile and give you a human-readable report. To use it, run pprofp and pass it an option and the path of the tracefile, like this:-

pprofp -u /var/log/pprof/pprof.25802

-a: sort by alphabetic names of subroutines.
-l: sort by number of calls to subroutines.
-m: sort by memory used in a function call.
-r: sort by real time spent in subroutines.
-R: sort by real time spent in subroutines (inclusive of child calls).
-s: sort by system time spent in subroutines.
-S: sort by system time spent in subroutines (inclusive of child calls).
-u: sort by user time spent in subroutines.
-U: sort by user time spent in subroutines (inclusive of child calls).
-v: sort by average amount of time spent in subroutines.
-z: sort by user+system time spent in subroutines. (default)

if you found any difficulty , then use kcachegrind tool ... it's free tool and probably installed on each machine.

1. pprof2calltree -f pprof.08871.0 ( it will return a file that is readable by kcachegrind ).
2. kcachegrind cachegrind.out.pprof.08871.0 report will shown like this ...




















Links:-
http://in.php.net/apd

Optimizing MySQL database

well , we have seen so many articles on Mysql performance and optimization. In this article i am trying to summarized all key points. hope this will help to developers and make life easier. :)

How MySQL stores data ?

• Databases are stored as directories.

• Tables are stored as files.

• Columns are stored in the files in dynamic length or fixed size format. In BDB tables the data is stored in pages.

• Memory-based tables are supported.

• Databases and tables can be symbolically linked from different disks.

• On Windows MySQL supports internal symbolic links to databases with .sym files.

MySQL table types

• HEAP tables; Fixed row size tables that are only stored in memory and indexed with a HASH index.

• ISAM tables; The old B-tree table format in MySQL 3.22.

• MyISAM tables; New version of the ISAM tables with a lot of extensions:

  • Binary portability.

  • Index on NULL columns.

  • Less fragmentation for dynamic-size rows than ISAM tables.

  • Support for big files.

  • Better index compression.

  • Better key statistics.

  • Better and faster auto_increment handling.

• Berkeley DB (BDB) tables from Sleepycat: Transaction-safe (with BEGIN WORK / COMMIT | ROLLBACK).

MySQL row types (only relevant for ISAM/MyISAM tables)

• MySQL will create the table in fixed size table format if all columns are of fixed length format (no VARCHAR, BLOB or TEXT columns). If not, the table is created in dynamic-size format.

• Fixed-size format is much faster and more secure than the dynamic format.

• The dynamic-size row format normally takes up less space but may be fragmented over time if the table is updated a lot.

• In some cases it's worth it to move all VARCHAR, BLOB and TEXT columns to another table just to get more speed on the main table.

• With myisampack (pack_isam for ISAM) one can create a read-only, packed table. This minimizes disk usage which is very nice when using slow disks. The packed tables are perfect to use on log tables which one will not update anymore.

MySQL caches (shared between all threads, allocated once)

• Key cache ; key_buffer_size, default 8M

• Table cache ; table_cache, default 64

• Thread cache ; thread_cache_size, default 0.

• Hostname cache ; Changeable at compile time, default 128.

• Memory mapped tables ; Currently only used for compressed tables.

How the MySQL table cache works

• Each open instance of a MyISAM table uses an index file and a data file. If a table is used by two threads or used twice in the same query, MyISAM will share the index file but will open another instance of the data file.

• The cache will temporarily grow larger than the table cache size if all tables in the cache are in use. If this happens, the next table that is released will be closed.

• You can check if your table cache is too small by checking the mysqld variable Opened_tables. If this value is high you should increase your table cache!

Optimizing hardware for MySQL

• If you need big tables ( > 2G), you should consider using 64 bit hardware like Alpha, Sparc or the upcoming IA64. As MySQL uses a lot of 64 bit integers internally, 64 bit CPUs will give much better performance.
• For large databases, the optimization order is normally RAM, Fast disks, CPU power.
  
  More RAM can speed up key updates by keeping most of the used key pages in RAM.
• If you are not using transaction-safe tables or have big disks and want to avoid long file checks, a UPS is good idea to be able to take the system down nicely in case of a power failure.
  
  
• For systems where the database is on a dedicated server, one should look at 1G Ethernet. Latency is as important as throughput.

Optimizing disks

• Have one dedicated disk for the system, programs and for temporary files. If you do very many changes, put the update logs and transactions logs on dedicated disks.

• Low seek time is important for the database disk; For big tables you can estimate that you will need: log(row_count) / log(index_block_length/3*2/(key_length + data_ptr_length))+1 seeks to find a row. For a table with 500,000 rows indexing a medium int: log(500,000)/log(1024/3*2/(3+4)) +1 = 4 seeks The above index would require: 500,000 * 7 * 3/2 = 5.2M. In real life, most of the blocks will be buffered, so probably only 1-2 seeks are needed.

• For writes you will need (as above) 4 seek requests, however, to find where to place the new key, and normally 2 seeks to update the index and write the row.

• For REALLY big databases, your application will be bound by the speed of your disk seeks, which increase by N log N as you get more data.

• Split databases and tables over different disks. In MySQL you can use symbolic links for this.

• Striping disks (RAID 0) will increase both read and write throughput.

• Striping with mirroring (RAID 0+1) will give you safety and increase the read speed. Write speed will be slightly lower.

• Don't use mirroring or RAID (except RAID 0) on the disk for temporary files or for data that can be easily re-generated..

• On Linux use hdparm -m16 -d1 on the disks on boot to enable reading/writing of multiple sectors at a time, and DMA. This may increase the response time by 5-50 %.

• On Linux, mount the disks with async (default) and noatime.

• For some specific application, one may want to have a ram disk for some very specific tables, but normally this is not needed.

Optimizing OS

• If you have memory problems, add more RAM instead or configure your system to use less memory.

• Don't use NFS disks for data (you will have problems with NFS locking).

• Increase number of open files for system and for the SQL server. (add ulimit -n # in the safe_mysqld script).

• Increase the number of processes and threads for the system.

• If you have relatively few big tables, tell your file system to not break up the file on different cylinders (Solaris).

• Use file systems that support big files (Solaris).

• Choose which file system to use; Reiserfs on Linux is very fast for open, read and write. File checks take just a couple of seconds.

• If possible, run OPTIMIZE table once in a while. This is especially important on variable size rows that are updated a lot.

• Update the key distribution statistics in your tables once in a while with myisamchk -a; Remember to take down MySQL before doing this!

• If you get fragmented files, it may be worth it to copy all files to another disk, clear the old disk and copy the files back.

• If you have problems, check your tables with myisamchk or CHECK table.

• Monitor MySQL status with: mysqladmin -i10 processlist extended-status

• With the MySQL GUI client you can monitor the process list and the status in different windows.

• Use mysqladmin debug to get information about locks and performance.

Optimizing SQL

Use SQL for the things it's good at, and do other things in your application. Use the SQL server to:

• Find rows based on WHERE clause.

• JOIN tables

• GROUP BY

• ORDER BY

• DISTINCT

Don't use an SQL server:

• To validate data (like date)

• As a calculator

Tips:

• Use keys wisely.

• Keys are good for searches, but bad for inserts / updates of key columns.

• Keep by data in the 3rd normal database form, but don't be afraid of duplicating information or creating summary tables if you need more speed.

• Instead of doing a lot of GROUP BYs on a big table, create summary tables of the big table and query this instead.

• UPDATE table set count=count+1 where key_column=constant is very fast!

• For log tables, it's probably better to generate summary tables from them once in a while than try to keep the summary tables live.

• Take advantage of default values on INSERT.

Optimizing tables

• MySQL has a rich set of different types. You should try to use the most efficient type for each column.

• The ANALYSE procedure can help you find the optimal types for a table: SELECT * FROM table_name PROCEDURE ANALYSE()

• Use NOT NULL for columns which will not store null values. This is particularly important for columns which you index.

• Change your ISAM tables to MyISAM.

• If possible, create your tables with a fixed table format.

• Don't create indexes you are not going to use.

• Use the fact that MySQL can search on a prefix of an index; If you have and INDEX (a,b), you don't need an index on (a).

• Instead of creating an index on long CHAR/VARCHAR column, index just a prefix of the column to save space. CREATE TABLE table_name (hostname CHAR(255) not null, index(hostname(10)))

• Use the most efficient table type for each table.

• Columns with identical information in different tables should be declared identically and have identical names.

MySQL extensions / optimization that gives you speed

• Use the optimal table type (HEAP, MyISAM, or BDB tables).

• Use optimal columns for your data.

• Use fixed row size if possible.

• Use the different lock types (SELECT HIGH_PRIORITY, INSERT LOW_PRIORITY)

• Auto_increment

• REPLACE (REPLACE INTO table_name VALUES (...))

• INSERT DELAYED

• LOAD DATA INFILE / LOAD_FILE()

• Use multi-row INSERT to insert many rows at a time.

• SELECT INTO OUTFILE

• LEFT JOIN, STRAIGHT JOIN

• LEFT JOIN combined with IS NULL

• ORDER BY can use keys in some cases.

• If you only query columns that are in one index, only the index tree will be used to resolve the query.

• Joins are normally faster than subselects (this is true for most SQL servers).

• LIMIT

  • SELECT * from table1 WHERE a > 10 LIMIT 10,20

  • DELETE * from table1 WHERE a > 10 LIMIT 10

• foo IN (list of constants) is very optimized.

• GET_LOCK()/RELEASE_LOCK()

• LOCK TABLES

• INSERT and SELECT can run concurrently.

• UDF functions that can be loaded into a running server.

• Compressed read-only tables.

• CREATE TEMPORARY TABLE

• CREATE TABLE .. SELECT

• MyISAM tables with RAID option to split a file over many files to get over the 2G limit on some file system.

• Delayed_keys

• Replication

When MySQL doesn't use an index

• Indexes are NOT used if MySQL can calculate that it will probably be faster to scan the whole table. For example if key_part1 is evenly distributed between 1 and 100, it's not good to use an index in the following query:

  • SELECT * FROM table_name where key_part1 > 1 and key_part1 <>

• If you are using HEAP tables and you don't search on all key parts with =

• When you use ORDER BY on a HEAP table

• If you are not using the first key part

  • SELECT * FROM table_name WHERE key_part2=1

• If you are using LIKE that starts with a wildcard

  • SELECT * FROM table_name WHERE key_part1 LIKE '%jani%'

• When you search on one index and do an ORDER BY on another

  • SELECT * from table_name WHERE key_part1 = # ORDER BY key2

Things to avoid with MySQL

• Updates to a table or INSERT on a table with deleted rows, combined with SELECTS that take a long time.

• HAVING on things you can have in a WHERE clause.

• JOINS without using keys or keys which are not unique enough.

• JOINS on columns that have different column types.

• Using HEAP tables when not using a full key match with =

• Forgetting a WHERE clause with UPDATE or DELETE in the MySQL monitor. If you tend to do this, use the --i-am-a-dummy option to the mysq client.

Tricks to give MySQL more information to solve things better

• SELECT SQL_BUFFER_RESULTS ...
  
  Will force MySQL to make a temporary result set. As soon as the temporary set is done, all locks on the tables are released. This can help when you get a problem with table locks or when it takes a long time to transfer the result to the client.

• SELECT SQL_SMALL_RESULT ... GROUP BY ...
  
  To tell the optimizer that the result set will only contain a few rows.

• SELECT SQL_BIG_RESULT ... GROUP BY ...
  
  To tell the optimizer that the result set will contain many rows.

• SELECT STRAIGHT_JOIN ...
  
  Forces the optimizer to join the tables in the order in which they are listed in the FROM clause.

• SELECT ... FROM table_name [USE INDEX (index_list) | IGNORE INDEX (index_list)] table_name2
  
  Forces MySQL to use/ignore the listed indexes.

General tips:

• Use short primary keys. Use numbers, not strings, when joining tables.

• When using multi-part keys, the first part should be the most-used key.

• When in doubt, use columns with more duplicates first to get better key compression.

• If you run the client and MySQL server on the same machine, use sockets instead of TCP/IP when connecting to MySQL (this can give you up to a 7.5 % improvement). You can do this by specifying no hostname or localhost when connecting to the MySQL server.

• Use --skip-locking (default on some OSes) if possible. This will turn off external locking and will give better performance.

• Use application-level hashed values instead of using long keys:

• SELECT * FROM table_name WHERE hash=MD5(concat(col1,col2)) AND

• col_1='constant' AND col_2='constant'

• Store BLOB's that you need to access as files in files. Store only the file name in the database.

• It is faster to remove all rows than to remove a large part of the rows.

• If SQL is not fast enough, take a look at the lower level interfaces to access the data.