Generally people thought that they know Yahoo 14 Rules so it's easy to optimize page rendering and one and only one solution is Ajaxify that page. But it's doesn't mean all websites are rendered fast in web. as i seen and now strongly believe that every website has it's own unique solution to fix their speed related issue and mainly people are unable to identify where is bottleneck. recently i have seen what FB did ?
In Facebook case, it's not easy to handle 500M Users.When Average hours per month per user is more than 5 hours.. google & yahoo has less than 2 hours only. FB has complex Frontend Infrastructure. FB runs 2 JS demons to handle Real time updates and Cache consistency.Main tasks are Incremental updates,In-page writes,Cross-page writes.Every state-changing operations are recorded and send to backend. Backend check when a write operation is detected, send a signal to the client to invalidate the cache.So usually user browses FB with three version. 1.cached version 2.state-changing version 3.Restored version.
This is one frontend solution that FB uses for caching other big ones are "BIG PIPE" and "Quickling" these are very advance techniques that are still need for me to understand
but what i know till now is :-
1. Use Network Latency and Page rendering delta time for other work
2. Try to reduce domcontentloaded and window load time gap
3. Use AJAX but in smart way .. mean Time-to-Interact should be very less i.e. page would fully render, but be frozen,User can't interact while JavaScript is being fetched/parsed/executed
Performance is hard so please think twise before move ahead .. now come to AJAX .. everybody know how use AJAX but very less people know AJAX design pattern :P
When AJAX call occurred it goes into following steps:-
1. Round trip Time: The first step is the amount of time between when the browser sends
the request and the time it receives the response
2. Parse Time: Next, the response returned from the server has to be parsed
3. JavaScript/CSS Download Time: [I believe you are smarter so you will download Widget's JS and CSS file lazy ;) ] Each response can indicate it needs more JavaScript/CSS before the
content can be used
4. Render Time : The amount of time it takes to actually change the display via innerHTML
So i would like know what solutions you have to fix above four issues ? how you optimize these areas ?
Best of Luck !!! Happy Coding !!!
Sunday, December 12, 2010
Friday, December 3, 2010
Serve Pre-Generated Static Files Instead Of Dynamic Pages
Well .. still i am not sure but i read few articles about how scale site performance without putting some extra efforts.i think it's cheapest terminology i have seen ever. We know well that static files have the advantage of being very fast to serve. Read from disk and display. Simple and fast. Especially when caching proxies are used. The issue is how do you bulk generate the initial files, how do you serve the files, and how do you keep the changed files up to date? specially regenerate static pages when changes occur... When a new entity is added to system hundreds of pages could be impacted, which would require the effected static pages to be regenerated.It's a very pragmatic solution and rock solid in operation. see more detail http://eventseer.net/p/thomas_brox_roest/whiteboardentry/13/
Tuesday, November 23, 2010
abc of HTTP cookie .. detailed look
we know well about cookie, how we create and how it works but still i seen some gap to understand it well .. might be i am not so much confidence about cookie :P
A web server specifies a cookie to be stored by sending an HTTP header called Set-Cookie. The format of the Set-Cookie header is a string as follows (parts in square brackets are optional):
Set-Cookie: value[; expires=date][; domain=domain][; path=path][; secure]
The first part of the header, the value, is typically a string in the format name=value. Indeed, the original specification indicates that this is the format to use but browsers do no such validation on cookie values. You can, in fact, specify a string without an equals sign and it will be stored just the same. Still, the most common usage is to specify a cookie value as name=value (and most interfaces support this exclusively).
When a cookie is present, and the optional rules allow, the cookie value is sent to the server with each subsequent request. The cookie value is stored in an HTTP header called Cookie and contains just the cookie value without any of the other options. Such as:
Cookie: value
If there are multiple cookies for the given request, then they are separated by a semicolon and space, such as:
Cookie: value1; value2; name1=value1
The next option is domain, which indicates the domain(s) for which the cookie should be sent. Another way to control when the Cookie header will be sent is to specify the path option. Similar to the domain option, path indicates a URL path that must exist in the requested resource before sending the Cookie header. When a cookie is created with an expiration date, that expiration date relates to the cookie identified by name-domain-path-secure. In order to change the expiration date of a cookie, you must specify the exact same tuple.
Keep in mind that the expiration date is checked against the system time on the computer that is running the browser. There is no way to verify that the system time is in sync with the server time and so errors may occur when there is a discrepancy between the system time and the server time.
There are a few reasons why a cookie might be automatically removed by the browser:
Session cookies are removed when the session is over (browser is closed).
Persistent cookies are removed when the expiration date and time have been reached.
If the browser’s cookie limit is reached, then cookies will be removed to make room for the most recently created cookie.
Cookie restrictions:-
There are a number of restrictions placed on cookies in order to prevent abuse and protect both the browser and the server from detrimental effects. There are two types of restrictions: number of cookies and total cookie size. The original specification placed a limit of 20 cookies per domain, which was followed by early browsers and continued up through Internet Explorer 7. During one of Microsoft’s updates, they increased the cookie limit in IE 7 to 50 cookies. IE 8 has a maximum of 50 cookies per domain as well. Firefox also has a limit of 50 cookies while Opera has a limit of 30 cookies. Safari and Chrome have no limit on the number of cookies per domain.
The maximum size for all cookies sent to the server has remained the same since the original cookie specification: 4 KB. Anything over that limit is truncated and won’t be sent to the server.
A web server specifies a cookie to be stored by sending an HTTP header called Set-Cookie. The format of the Set-Cookie header is a string as follows (parts in square brackets are optional):
Set-Cookie: value[; expires=date][; domain=domain][; path=path][; secure]
The first part of the header, the value, is typically a string in the format name=value. Indeed, the original specification indicates that this is the format to use but browsers do no such validation on cookie values. You can, in fact, specify a string without an equals sign and it will be stored just the same. Still, the most common usage is to specify a cookie value as name=value (and most interfaces support this exclusively).
When a cookie is present, and the optional rules allow, the cookie value is sent to the server with each subsequent request. The cookie value is stored in an HTTP header called Cookie and contains just the cookie value without any of the other options. Such as:
Cookie: value
If there are multiple cookies for the given request, then they are separated by a semicolon and space, such as:
Cookie: value1; value2; name1=value1
The next option is domain, which indicates the domain(s) for which the cookie should be sent. Another way to control when the Cookie header will be sent is to specify the path option. Similar to the domain option, path indicates a URL path that must exist in the requested resource before sending the Cookie header. When a cookie is created with an expiration date, that expiration date relates to the cookie identified by name-domain-path-secure. In order to change the expiration date of a cookie, you must specify the exact same tuple.
Keep in mind that the expiration date is checked against the system time on the computer that is running the browser. There is no way to verify that the system time is in sync with the server time and so errors may occur when there is a discrepancy between the system time and the server time.
There are a few reasons why a cookie might be automatically removed by the browser:
Session cookies are removed when the session is over (browser is closed).
Persistent cookies are removed when the expiration date and time have been reached.
If the browser’s cookie limit is reached, then cookies will be removed to make room for the most recently created cookie.
Cookie restrictions:-
There are a number of restrictions placed on cookies in order to prevent abuse and protect both the browser and the server from detrimental effects. There are two types of restrictions: number of cookies and total cookie size. The original specification placed a limit of 20 cookies per domain, which was followed by early browsers and continued up through Internet Explorer 7. During one of Microsoft’s updates, they increased the cookie limit in IE 7 to 50 cookies. IE 8 has a maximum of 50 cookies per domain as well. Firefox also has a limit of 50 cookies while Opera has a limit of 30 cookies. Safari and Chrome have no limit on the number of cookies per domain.
The maximum size for all cookies sent to the server has remained the same since the original cookie specification: 4 KB. Anything over that limit is truncated and won’t be sent to the server.
Monday, November 22, 2010
Web site Front-end optimization headaches
I have seen that typically 80-90% of page response time comes from other things than fetching HTML of the page :- fetching CSS, JavaScript, Images,banners which makes it number one thing to focus your optimization. Sharing few points, specially for front-end optimization. Hope this will be helpful.
1. Try to split initial payload .. first load all required js to load page then other scripts/things.
Lazyload,ajaxload,load scripts in parrller,load script after on windowload, these are different things and you must know about it.When the browser starts downloading an external script, it won’t start any additional downloads until the script has been completely downloaded, parsed, and executed. So load external scripts asynchronously only. we also need to care about inline JS, here we can use callback functions,timers,on window load etc see following chart , here few techniques are listed to load JS scripts.
2. Flushing the Document Early:-
As the server parses the PHP page, all output is written to STDOUT. Rather than being sent immediately, one character, word, or line at a time, the output is queued up and sent to the browser in larger chunks. This is more efficient because it results in fewer packets being sent from the server to the browser. Each packet sent incurs some network latency, so it’s usually better to send a small number of large packets, rather than a large number of small packets. Calling flush() causes anything queued up in STDOUT to be sent immediately. How- ever, simply flushing STDOUT isn’t enough to achieve the type of speedup experienced in the preceding example. The call to flush has to be made in the right place. idea is , flush before a long time taking process, it can be divide as module wise also like header,footer,sidebar,widgets etc.
3. Repaint and Reflow is costly:-
A repaint occurs when a visual change doesn't require recalculation of layout Changes to visibility, colors (text/background), background images, etc.
A reflow occurs when a visual change requires a change in layout Initial page load
browser resize,DOM structure change,layout style change layout information retrieved
4. Eliminate unnecessary cookies:-
Keep cookie sizes as low as possible to minimize the impact on the user response time,Be mindful of setting cookies at the appropriate domain level so other sub-domains are not affected
5. DOM manipulations are the slowest,Never update the DOM, if that's not possible, at least do it as infrequently as possible. Bunch up your updates to the DOM and save them for a later time.
6. Clone the DOM node you want to work with. Now you will be working with a clone of the real node, and the cloned node doesn't exist in the DOM. Updating the cloned node doesn't affect the DOM. When you are done with your manipulations, replace the original node with the cloned node.
7. However, note that the performance problems in point 4, because of the content and rendering reflow that the browser has to do. You might get similar benefits by simply hiding the element first, making the changes, and then showing it.
Saturday, November 20, 2010
MOD_PHP or FASTCGI ?
When we load PHP into Apache as a module (using mod_php), each Apache process we run will also contain a PHP interpreter which in turn will load all the compiled in libraries which themselves are not exactly small.
This means that even if the Apache process that just started will only serve images, it will contain a PHP interpreter with all assigned libraries. That in turn means that said Apache process uses a lot of memory and takes some time to start up (because PHP and all the shared libraries it's linked to need to be loaded). Wasted energy if the file that needs to be served in an image or a CSS file.
FastCGI in contrast loads the PHP interpreter into memory, keeps it there and Apache will only use these processes to serve the PHP requests.
That means that all the images and CSS, flashes and whatever other static content we may have can be served by a much smaller Apache process that does not contain a scripting language interpreter and that does not link in a bunch of extra libraries (think libxml, libmysqlclient, and so on).
Even if we only serve pages parsed by PHP - maybe because we process our stylesheets with PHP and because we do something with the served images - we are theoretically still better off with FastCGI as Apache will recycle its processes here and then (though that's configurable) while FastCGI processes stay there.
And if we go on and need to load-balance your application, FastCGI still can provide advantages: In the common load balancing scenario, we have a reverse proxy or a load balancer and a bunch of backend servers actually doing the work. In that case, if we use FastCGI, the backend servers will be running our PHP application and noting else. No web server loading an interpreter loading our script. Just the interpreter and our script. So we safe a whole lot of memory by not loading another web server in the backend (Yes. FastCGI works over the network).
This means that even if the Apache process that just started will only serve images, it will contain a PHP interpreter with all assigned libraries. That in turn means that said Apache process uses a lot of memory and takes some time to start up (because PHP and all the shared libraries it's linked to need to be loaded). Wasted energy if the file that needs to be served in an image or a CSS file.
FastCGI in contrast loads the PHP interpreter into memory, keeps it there and Apache will only use these processes to serve the PHP requests.
That means that all the images and CSS, flashes and whatever other static content we may have can be served by a much smaller Apache process that does not contain a scripting language interpreter and that does not link in a bunch of extra libraries (think libxml, libmysqlclient, and so on).
Even if we only serve pages parsed by PHP - maybe because we process our stylesheets with PHP and because we do something with the served images - we are theoretically still better off with FastCGI as Apache will recycle its processes here and then (though that's configurable) while FastCGI processes stay there.
And if we go on and need to load-balance your application, FastCGI still can provide advantages: In the common load balancing scenario, we have a reverse proxy or a load balancer and a bunch of backend servers actually doing the work. In that case, if we use FastCGI, the backend servers will be running our PHP application and noting else. No web server loading an interpreter loading our script. Just the interpreter and our script. So we safe a whole lot of memory by not loading another web server in the backend (Yes. FastCGI works over the network).
Monday, October 25, 2010
Why engineers fail and what makes a great engineer ?
Let's start with what makes great ... Today seen a presentation about Netflix company culture. The presentation told about how the company goes about hiring (and firing) of employees.
I don’t work at Netflix, of course (I work for shiksha.com), but I feel strongly that what makes a great employee and a great engineer is the same regardless of where you work. There are a few things that great engineers always do.
Always do it the right way
One of the challenges of software is to learn how to do it the right way. The right way is different depending upon what you’re working on and who you’re working for.Junior engineers tend to have the most trouble with this, but it does happen with senior-level people too. There’s an “emergency” project, or something that seems so different that it must have its own set of rules. That’s bogus.
Good engineers know that the right way applies to all situations and circumstances. If there’s not enough time to do something the right way, then there’s really not enough time to do it. Don’t make compromises, the quality of your work is what ultimately defines you as an engineer. Make sure that all of the code you write is done the right way 100% of the time. Expect excellence from yourself.
Be willing to suffer
This may sound silly, but good engineers are willing to suffer for their work. Show me a great engineer and I’ll show you someone that has, at various points in his or her career, spent days
trying to figure out a problem.Great engineers relish the challenge of a problem that keeps them up day and night, and knowing that it must be solved.
Not-so-great engineers call for help at the first sign of trouble. They routinely ask for help whenever something goes wrong rather than trying to fix it themselves. Their favorite line is, “can you look at this?” Great engineers first and foremost want to solve the problem on their own. Problem solving is a skill, a skill that great engineers take seriously.
Good engineers become great engineers by suffering. Suffering means not asking for help unless you absolutely cannot handle the task. Asking for help is a sign of defeat, so ring that bell infrequently lest you draw unwanted attention to yourself. Be willing to suffer. Spend time toiling over the problem. That’s how you learn.
I am not saying that you should never ask for help. I am saying that you should try to accomplish the task on your own first, and if you get stuck, then ask for help. Don’t simply ask for help every time without first trying to solve the problem yourself. Chances are, you’ll find that you could have figured it out on your own once you know the answer.
Never stop learning
Any engineer who claims that they don’t need to learn anything new is not someone with whom I’d like to work. In some careers, you can get away without learning anything new for years; technology changes too quickly to not pay attention. Your employer is paying you for your expertise and if that expertise goes stale, you become expendable. In order to be a great engineer you must first admit that you don’t know everything, and then you must seek out more knowledge in every way you can.
Identify someone in your current company or organization from which you can learn and attach yourself to him or her. Ask for advice on complex problems to see how they think. Show them solutions you've come up with and ask for a critique. If you can’t identify anyone in your organization that can serve as a mentor, then either you’re not looking hard enough or you’re at the wrong company. If you can’t grow in your current job then it’s time to look for another.
Read blogs. Attend conferences. Go to developer meetups. Great engineers never stop learning.
Share your knowledge
There are some who believe that their sole value is their knowledge, and by sharing that knowledge they therefore make themselves less valuable. Nothing could be farther from the truth. What makes you valuable is not your knowledge, it’s how you make use of your knowledge to create value for your employer. How better to create value from your knowledge than to share it with others?
I’ve interviewed at companies where hording knowledge seemed deeply-rooted at the organizational level. In that type of environment, a fierce competition develops between co-workers, and this opens the door to politics and backstabbing. I don’t want to work in an organization like that. You can’t learn if everyone is keeping information to themselves.
Great engineers want others to know what they know. They aren’t afraid of losing their position because someone else can do the same thing. Great engineers want to see their peers succeed and grow. Organizations rally around people who share information, and as they say in sports, people who make other people on the team better.
Lend a helping hand
Great engineers don’t consider any task to be “beneath” them. Always willing to lend a hand, you can see great engineers helping out junior engineers in between doing their own work. If something has to get done, and no one else is able to do it in time, great engineers volunteer to take on the work. They don’t scoff when asked to help on a project, whether it be small or menial or low-profile. Great engineers are team-focused and therefore are willing to do whatever it takes to help the team. Whether that be writing 1,000 lines of code or editing an image, great engineers jump at the chance to help out.
Take your time
Great engineers aren’t born, they are made. They’re made by following the advice in this post and by hard work. If you’re just starting out, there’s still plenty of time to become a great engineer. Patience is key. You don’t become a great engineer over night. For some it may take a couple years, for others it may take ten. There’s no one keeping score. Strong organizations recognize when someone has the potential to be a great engineer and will guide you along. You prove yourself through your work and how you make your team better. Focus and self-discipline go a long way towards becoming a great software engineer.
Discuss now why engineers fail ? This typically begins the conversation about the Peter principle. The Peter principle says that you’ll keep getting promoted until you finally end up in a job that you can’t do. This happens because the higher up in the organizational structure you move, the less your technical skills matter and the more your people skills matter. So whereas you began in a position that played to your strengths, you end up in one that plays to your weakenesses. This is precisely what Berkun found in his study, that designers were failing due to factors outside of their design skills. That is why designers fail. It’s also why software engineers fail.
software engineers, once they rise high enough in the organizational hierarchy, they need to learn how to work within the organizational structure. Oftentimes, that means gaining the
trust of business partners: engineers need to gain the trust of product managers. Gaining the trust of these business partners means being able to successfully negotiate, compromise, and work towards meeting a common goal without alienating people through your actions and speech. This is typically where people falter in their careers.
This year I’ve had to learn how to play the organizational game. I can honestly say it’s been far more challenging than anything I’ve done before. Dealing with people is much more difficult than dealing with technology, that’s for sure. You need to understand what each person responds to in terms of approach. Some people will easily cave when pressure is applied, others need to be convinced through logical argument while another set may require emotional persuasion. And of course, all of this must be done while making sure that all of these people still respect you and don’t feel manipulated.
Fortunately, my interest and research in social interaction has really helped me thusfar. Understanding what drives people and how to communicate effectively have been key to me. If you have aspirations of moving up in your company, then it would behoove you to also start researching these topics. The only way to really get ahead in business is a better understanding of people. Hard skill jobs such as engineers and designers are commodities that can easily be outsourced if necessary; soft skill jobs requiring you to work with and inspire others will always be in high demand and, as a bonus, can never be outsourced. Mastering people skills ensures employability, and more importantly, ensures that you won’t fail.
original post @ http://www.nczonline.net
I don’t work at Netflix, of course (I work for shiksha.com), but I feel strongly that what makes a great employee and a great engineer is the same regardless of where you work. There are a few things that great engineers always do.
Always do it the right way
One of the challenges of software is to learn how to do it the right way. The right way is different depending upon what you’re working on and who you’re working for.Junior engineers tend to have the most trouble with this, but it does happen with senior-level people too. There’s an “emergency” project, or something that seems so different that it must have its own set of rules. That’s bogus.
Good engineers know that the right way applies to all situations and circumstances. If there’s not enough time to do something the right way, then there’s really not enough time to do it. Don’t make compromises, the quality of your work is what ultimately defines you as an engineer. Make sure that all of the code you write is done the right way 100% of the time. Expect excellence from yourself.
Be willing to suffer
This may sound silly, but good engineers are willing to suffer for their work. Show me a great engineer and I’ll show you someone that has, at various points in his or her career, spent days
trying to figure out a problem.Great engineers relish the challenge of a problem that keeps them up day and night, and knowing that it must be solved.
Not-so-great engineers call for help at the first sign of trouble. They routinely ask for help whenever something goes wrong rather than trying to fix it themselves. Their favorite line is, “can you look at this?” Great engineers first and foremost want to solve the problem on their own. Problem solving is a skill, a skill that great engineers take seriously.
Good engineers become great engineers by suffering. Suffering means not asking for help unless you absolutely cannot handle the task. Asking for help is a sign of defeat, so ring that bell infrequently lest you draw unwanted attention to yourself. Be willing to suffer. Spend time toiling over the problem. That’s how you learn.
I am not saying that you should never ask for help. I am saying that you should try to accomplish the task on your own first, and if you get stuck, then ask for help. Don’t simply ask for help every time without first trying to solve the problem yourself. Chances are, you’ll find that you could have figured it out on your own once you know the answer.
Never stop learning
Any engineer who claims that they don’t need to learn anything new is not someone with whom I’d like to work. In some careers, you can get away without learning anything new for years; technology changes too quickly to not pay attention. Your employer is paying you for your expertise and if that expertise goes stale, you become expendable. In order to be a great engineer you must first admit that you don’t know everything, and then you must seek out more knowledge in every way you can.
Identify someone in your current company or organization from which you can learn and attach yourself to him or her. Ask for advice on complex problems to see how they think. Show them solutions you've come up with and ask for a critique. If you can’t identify anyone in your organization that can serve as a mentor, then either you’re not looking hard enough or you’re at the wrong company. If you can’t grow in your current job then it’s time to look for another.
Read blogs. Attend conferences. Go to developer meetups. Great engineers never stop learning.
Share your knowledge
There are some who believe that their sole value is their knowledge, and by sharing that knowledge they therefore make themselves less valuable. Nothing could be farther from the truth. What makes you valuable is not your knowledge, it’s how you make use of your knowledge to create value for your employer. How better to create value from your knowledge than to share it with others?
I’ve interviewed at companies where hording knowledge seemed deeply-rooted at the organizational level. In that type of environment, a fierce competition develops between co-workers, and this opens the door to politics and backstabbing. I don’t want to work in an organization like that. You can’t learn if everyone is keeping information to themselves.
Great engineers want others to know what they know. They aren’t afraid of losing their position because someone else can do the same thing. Great engineers want to see their peers succeed and grow. Organizations rally around people who share information, and as they say in sports, people who make other people on the team better.
Lend a helping hand
Great engineers don’t consider any task to be “beneath” them. Always willing to lend a hand, you can see great engineers helping out junior engineers in between doing their own work. If something has to get done, and no one else is able to do it in time, great engineers volunteer to take on the work. They don’t scoff when asked to help on a project, whether it be small or menial or low-profile. Great engineers are team-focused and therefore are willing to do whatever it takes to help the team. Whether that be writing 1,000 lines of code or editing an image, great engineers jump at the chance to help out.
Take your time
Great engineers aren’t born, they are made. They’re made by following the advice in this post and by hard work. If you’re just starting out, there’s still plenty of time to become a great engineer. Patience is key. You don’t become a great engineer over night. For some it may take a couple years, for others it may take ten. There’s no one keeping score. Strong organizations recognize when someone has the potential to be a great engineer and will guide you along. You prove yourself through your work and how you make your team better. Focus and self-discipline go a long way towards becoming a great software engineer.
Discuss now why engineers fail ? This typically begins the conversation about the Peter principle. The Peter principle says that you’ll keep getting promoted until you finally end up in a job that you can’t do. This happens because the higher up in the organizational structure you move, the less your technical skills matter and the more your people skills matter. So whereas you began in a position that played to your strengths, you end up in one that plays to your weakenesses. This is precisely what Berkun found in his study, that designers were failing due to factors outside of their design skills. That is why designers fail. It’s also why software engineers fail.
software engineers, once they rise high enough in the organizational hierarchy, they need to learn how to work within the organizational structure. Oftentimes, that means gaining the
trust of business partners: engineers need to gain the trust of product managers. Gaining the trust of these business partners means being able to successfully negotiate, compromise, and work towards meeting a common goal without alienating people through your actions and speech. This is typically where people falter in their careers.
This year I’ve had to learn how to play the organizational game. I can honestly say it’s been far more challenging than anything I’ve done before. Dealing with people is much more difficult than dealing with technology, that’s for sure. You need to understand what each person responds to in terms of approach. Some people will easily cave when pressure is applied, others need to be convinced through logical argument while another set may require emotional persuasion. And of course, all of this must be done while making sure that all of these people still respect you and don’t feel manipulated.
Fortunately, my interest and research in social interaction has really helped me thusfar. Understanding what drives people and how to communicate effectively have been key to me. If you have aspirations of moving up in your company, then it would behoove you to also start researching these topics. The only way to really get ahead in business is a better understanding of people. Hard skill jobs such as engineers and designers are commodities that can easily be outsourced if necessary; soft skill jobs requiring you to work with and inspire others will always be in high demand and, as a bonus, can never be outsourced. Mastering people skills ensures employability, and more importantly, ensures that you won’t fail.
original post @ http://www.nczonline.net
Sunday, October 24, 2010
Why use OAuth ? what are the benefits ?
OAuth is open protocol to allow secure API authorization in a simple and standard method from desktop and web applications. In short, it means that a user of your service can provide you limited access to a third party account of theirs. OAuth is often described as a valet key that your users can give you to access their accounts on other services. For example, a user using Flickr (the service provider) would provide Snapfish (the consumer) with read only access to their Flickr account. This lets Snapfish access photos in the user's Flickr account so they can order prints.
It's all in the tokens
How does this happen without asking the user to give up their Flickr password? The flow would start by Snapfish obtaining a consumer key and secret and using them to generate an authorization link to Flickr. Once the user follows the authorization link, they are asked to log in on Flickr's site. Once logged in they can choose to grant Snapfish access to their Flickr account. Flickr then marks the request token as having been authorized by the user. Snapfish uses the request token to obtain an access token which can be used by to make requests to Flickr on behalf of the user. This diagram may help visualize it easier. C = Consumer, SP = Service Provider
How does this happen without asking the user to give up their Flickr password? The flow would start by Snapfish obtaining a consumer key and secret and using them to generate an authorization link to Flickr. Once the user follows the authorization link, they are asked to log in on Flickr's site. Once logged in they can choose to grant Snapfish access to their Flickr account. Flickr then marks the request token as having been authorized by the user. Snapfish uses the request token to obtain an access token which can be used by to make requests to Flickr on behalf of the user. This diagram may help visualize it easier. C = Consumer, SP = Service Provider
Generating a valid OAuth request
It turns out that generating an OAuth request is very simple but debugging it is a pain. Every OAuth request contains certain parameters. These include:
- oauth_consumer_key
- oauth_token
- oauth_nonce
- oauth_timestamp
- oauth_signature method
- oauth_version
- oauth_signature
These can be passed in as GET or POST parameters or in the Authorization header. You'll most likely be passing in other additional parameters based on the API you're accessing. I think it's enough to understand OAuth and why we used it frequently. More details are available on http://oauth.net/documentation/getting-started
Friday, October 22, 2010
PHP Application Framework Battle ... CodeIgniter vs. Symfony
I'm familiar with many of the PHP frameworks like symfony,zend,cakePHP,codeigniter and last one which i think it will be good enough specially to design rich web and large scale applications, that is YII http://www.yiiframework.com/features/ at last, I would like share a funny framework built in PHP and that's source code you can even tweet :P checkout this one http://twitto.org/ :-)
We have seen on web that there are so many solutions but i notice that people always ask which ones is good ? Does such a framework exist who do all jobs well ? Usually my first answer is .. your question is a bit like going to the hardware store and having a conversation like:-
You: I'd like to buy some tools. Staff Member: Ok, great. What are you building? You: Don't know yet. But I'm sure I'll need some tools.
Second it's obviously that there are so many metrics of what good can possibly mean like, disk size on server, amount of code generated for the client, difficulty of installing / configuring process on the server, etc...
There is absolutely no point in solving a problem until you have a problem. Just code vanilla PHP until you decide some particular task is too hard/messy/etc and a framework will actually help you and, this may go against the "must-have-framework" crowd, but honestly I think for trivial tasks you're typically better rolling your own.
So again ball is on your side .. what are your problems and why you need it ? would you like fancy Web 2.0 site ? or scalability is major concern as your site hits ? Most likely in the form of a big complex MVC framework with plenty of layers that abstracts away your database, your HTML, your Javascript and in the end your application itself. If it is a really good framework it will provide a dozen things you'll never need. I am obviously not a fan of such frameworks. I like stuff I can understand in an instant. Both because it lets me be productive right away and because 6 months from now when I come back to fix something, again I will only need an instant to figure out what is going on.
Why MVC ? I don't want make you confuse but i found that it's up to you how much you would like build your application scalable and modular like security and all features that's you want to apply.. see my previous one posts ..
http://ravirajsblog.blogspot.com/2008/12/world-of-object-oriented-programming.html
http://ravirajsblog.blogspot.com/2008/12/principles-of-mvc-design-pattern.html
http://ravirajsblog.blogspot.com/2008/12/3-tier-architecture.html
I like MVC but Just make sure you avoid the temptation of creating a single monolithic controller. A web application by its very nature is a series of small discrete requests. If you send all of your requests through a single controller on a single machine you have just defeated this very important architecture. Discreteness gives you scalability and modularity.
You can break large problems up into a series of very small and modular solutions and you can deploy these across as many servers as you like. You need to tie them together to some extent most likely through some backend datastore, but keep them as separate as possible. This means you want your views and controllers very close to each other and you want to keep your controllers as small as possible.
So design your goals first.
1. Clean and simple design
* HTML should look like HTML
* Keep the PHP code in the views extremely simple: function calls, simple loops and variable substitutions should be all you need
2. Secure
* Input validation using pecl/filter as a data firewall
* When possible, avoid layers and other complexities to make code easier to audit
3. Fast
* Avoid include_once and require_once
* Use APC and apc_store/apc_fetch for caching data that rarely changes
* Stay with procedural style unless something is truly an object
* Avoid locks at all costs
Now come to title of post .. one to one "CodeIgniter vs Symfony" comparison .. overall, symfony is more sophisticated (ie. harder to use ... it has 7 times more codebase than codeigniter) while codeIgniter seems more geared to low level of programming and environment but yes symfony has complete features of set that makes it much stronger and scalable.
Finally i summarized and here are few my concerns that you should keep in mind while choosing a framework.
1. choose PHP version > 5.3
In PHP, an object is destroyed when it goes out of scope. This is normally when the script stops executing or when the function it was created within ends. unset($my_variable); we need to be fine allowing the destructor to handle closing the DB in most situations. The garbage collector or GC in PHP will do all the work you need once the variable goes out of scope or there are zero references to it. To be clear, the GC got a major rework in PHP 5.3.
2. Use PHP's PDO for model http://php.net/pdo
3. Use open source for rich UI work like http://developer.yahoo.net/yui/
4. Use PHP's pecl/filter extension for automagically sanitize all user data
5. Use auto include configuration setting in PHP to avoid include one and required once see http://php.net/manual/en/ini.core.php#ini.auto-prepend-file
6. Use PHP's inbuilt APIs for opt code caching http://php.net/apc_fetch
Clean separation of your views, controller logic and backend model logic is easy to do with PHP. Using above ideas, we should be able to build a clean framework aimed specifically at our requirements instead of trying to refactor a much larger and more complex external framework.
Many frameworks may look very appealing at first glance because they seem to reduce web application development to a couple of trivial steps leading to some code generation and often automatic schema detection, but these same shortcuts are likely to be your bottlenecks as well since they achieve this simplicity by sacrificing flexibility and performance. Nothing is going to build your application for you, no matter what it promises. You are going to have to build it yourself. Instead of starting by fixing the mistakes in some foreign framework and refactoring all the things that don't apply to your environment spend your time building a lean and reusable pattern that fits your requirements directly. In the end I think you will find that your homegrown small framework has saved you time and aggravation and you end up with a better product.
We have seen on web that there are so many solutions but i notice that people always ask which ones is good ? Does such a framework exist who do all jobs well ? Usually my first answer is .. your question is a bit like going to the hardware store and having a conversation like:-
You: I'd like to buy some tools. Staff Member: Ok, great. What are you building? You: Don't know yet. But I'm sure I'll need some tools.
Second it's obviously that there are so many metrics of what good can possibly mean like, disk size on server, amount of code generated for the client, difficulty of installing / configuring process on the server, etc...
There is absolutely no point in solving a problem until you have a problem. Just code vanilla PHP until you decide some particular task is too hard/messy/etc and a framework will actually help you and, this may go against the "must-have-framework" crowd, but honestly I think for trivial tasks you're typically better rolling your own.
So again ball is on your side .. what are your problems and why you need it ? would you like fancy Web 2.0 site ? or scalability is major concern as your site hits ? Most likely in the form of a big complex MVC framework with plenty of layers that abstracts away your database, your HTML, your Javascript and in the end your application itself. If it is a really good framework it will provide a dozen things you'll never need. I am obviously not a fan of such frameworks. I like stuff I can understand in an instant. Both because it lets me be productive right away and because 6 months from now when I come back to fix something, again I will only need an instant to figure out what is going on.
Why MVC ? I don't want make you confuse but i found that it's up to you how much you would like build your application scalable and modular like security and all features that's you want to apply.. see my previous one posts ..
http://ravirajsblog.blogspot.com/2008/12/world-of-object-oriented-programming.html
http://ravirajsblog.blogspot.com/2008/12/principles-of-mvc-design-pattern.html
http://ravirajsblog.blogspot.com/2008/12/3-tier-architecture.html
I like MVC but Just make sure you avoid the temptation of creating a single monolithic controller. A web application by its very nature is a series of small discrete requests. If you send all of your requests through a single controller on a single machine you have just defeated this very important architecture. Discreteness gives you scalability and modularity.
You can break large problems up into a series of very small and modular solutions and you can deploy these across as many servers as you like. You need to tie them together to some extent most likely through some backend datastore, but keep them as separate as possible. This means you want your views and controllers very close to each other and you want to keep your controllers as small as possible.
So design your goals first.
1. Clean and simple design
* HTML should look like HTML
* Keep the PHP code in the views extremely simple: function calls, simple loops and variable substitutions should be all you need
2. Secure
* Input validation using pecl/filter as a data firewall
* When possible, avoid layers and other complexities to make code easier to audit
3. Fast
* Avoid include_once and require_once
* Use APC and apc_store/apc_fetch for caching data that rarely changes
* Stay with procedural style unless something is truly an object
* Avoid locks at all costs
Now come to title of post .. one to one "CodeIgniter vs Symfony" comparison .. overall, symfony is more sophisticated (ie. harder to use ... it has 7 times more codebase than codeigniter) while codeIgniter seems more geared to low level of programming and environment but yes symfony has complete features of set that makes it much stronger and scalable.
Finally i summarized and here are few my concerns that you should keep in mind while choosing a framework.
1. choose PHP version > 5.3
In PHP, an object is destroyed when it goes out of scope. This is normally when the script stops executing or when the function it was created within ends. unset($my_variable); we need to be fine allowing the destructor to handle closing the DB in most situations. The garbage collector or GC in PHP will do all the work you need once the variable goes out of scope or there are zero references to it. To be clear, the GC got a major rework in PHP 5.3.
2. Use PHP's PDO for model http://php.net/pdo
3. Use open source for rich UI work like http://developer.yahoo.net/yui/
4. Use PHP's pecl/filter extension for automagically sanitize all user data
5. Use auto include configuration setting in PHP to avoid include one and required once see http://php.net/manual/en/ini.core.php#ini.auto-prepend-file
6. Use PHP's inbuilt APIs for opt code caching http://php.net/apc_fetch
Clean separation of your views, controller logic and backend model logic is easy to do with PHP. Using above ideas, we should be able to build a clean framework aimed specifically at our requirements instead of trying to refactor a much larger and more complex external framework.
Many frameworks may look very appealing at first glance because they seem to reduce web application development to a couple of trivial steps leading to some code generation and often automatic schema detection, but these same shortcuts are likely to be your bottlenecks as well since they achieve this simplicity by sacrificing flexibility and performance. Nothing is going to build your application for you, no matter what it promises. You are going to have to build it yourself. Instead of starting by fixing the mistakes in some foreign framework and refactoring all the things that don't apply to your environment spend your time building a lean and reusable pattern that fits your requirements directly. In the end I think you will find that your homegrown small framework has saved you time and aggravation and you end up with a better product.
Sunday, October 3, 2010
What does “> /dev/null 2>&1″ mean in Linux ?
I remember being confused for a very long time about the trailing garbage in commands I saw in Unix systems, especially while watching compilers do their work. Nobody I asked could tell me what the funny greater-thans, ampersands and numbers after the commands meant, and search engines never turned up anything but examples of it being used without explanation. In this article I’ll explain those weird commands.
Here’s an example command:
wibble > /dev/null 2>&1
Output redirection
The greater-thans (
>) in commands like these redirect the program’s output somewhere. In this case, something is being redirected into /dev/null, and something is being redirected into &1.Standard in, out, and error
There are three standard sources of input and output for a program. Standard input usually comes from the keyboard if it’s an interactive program, or from another program if it’s processing the other program’s output. The program usually prints to standard output, and sometimes prints to standard error. These three file descriptors (you can think of them as “data pipes”) are often called STDIN, STDOUT, and STDERR.
Sometimes they’re not named, they’re numbered! The built-in numberings for them are 0, 1, and 2, in that order. By default, if you don’t name or number one explicitly, you’re talking about STDOUT.
Given that context, you can see the command above is redirecting standard output into
/dev/null, which is a place you can dump anything you don’t want (often called the bit-bucket), then redirecting standard error into standard output (you have to put an & in front of the destination when you do this).The short explanation, therefore, is “all output from this command should be shoved into a black hole.” That’s one good way to make a program be really quiet
Saturday, September 18, 2010
The BIG software blog: Scalability patterns and an interesting story...
The BIG software blog: Scalability patterns and an interesting story...: "I read a pretty interesting war-story from solving scalability the pragmatic and incremental way. http://highscalability.com/blog/2010/8/23/..."
Subscribe to:
Posts (Atom)