codeigniter session memcache anti Bots and timestamped cache

Here you can see three different hacks that we can use in CI to make our life easier. However all these are based on my local box implementation and still these patches need to push live.

1. Memcache session handling available with CodeIgniter, So first thing to do was to filter bots that frequently visit my site and deny them sessions, instead a cookie will do them just fine.

function __detectVisit() {

       $this->CI->load->library('user_agent');

       $agent = strtolower($this->CI->input->user_agent());

       $bot_strings = array(

           "google", "bot", "yahoo", "spider", "archiver", "curl",

           "python", "nambu", "twitt", "perl", "sphere", "PEAR",

           "java", "wordpress", "radian", "crawl", "yandex", "eventbox",

           "monitor", "mechanize", "facebookexternal", "bingbot"

       );

       foreach($bot_strings as $bot) {

               if(strpos($agent, $bot) !== false) {

                       return "bot";

               }

       }

       return "normal";

}

The next step was to build namespaces adjusted with some of CodeIgniter’s built in Session handling mechanisms.

function __build_namespace($sess_id, $ip_addr = 0, $user_agent = '') {
$this->namespace .= $sess_id;
if($this->sess_match_ip == TRUE && $ip_addr > 0)
$this->namespace .= '#'.ip2long($ip_addr);
if($this->sess_match_useragent == TRUE && $user_agent != '')
$this->namespace .= '#'.md5($user_agent);
}


2. Second hack is bypass CI to server HTML cached pages. Here is code and its very easy to understand. 



<?php  if ( ! defined('BASEPATH')) exit('No direct script access allowed');


class MY_Output extends CI_Output {


function _display($output = '') {
$elapsed = $BM->elapsed_time('total_execution_time_start', 'total_execution_time_end');
$output = str_replace('{elapsed_time}', $elapsed, $output);


$memory = ( ! function_exists('memory_get_usage')) ? '0' : round(memory_get_usage()/1024/1024, 2).'MB';
$output = str_replace('{memory_usage}', $memory, $output);


// Grab the super object.  We'll need it in a moment...
$CI =& get_instance();


// TRAP OUTPUT
if(!empty($output) && ! $CI->access->logged_in() && class_exists('Memcache')) {
$output .= '<!-- DYNAMICALLY GENERATED -->';
$ns = "outputcachememc#".$_SERVER['HTTP_HOST'] . $_SERVER['REQUEST_URI'];
$m = new Memcache;
$m->addServer('127.0.0.1', '11211', 1);
$m->set($ns, $output, MEMCACHE_COMPRESSED, 60*5);


$output = str_replace('{elapsed_time_cc}', $elapsed, $output);
}
$output = str_replace('{elapsed_time_cc}', $elapsed, $output);


if (method_exists($CI, '_output')) {
$CI->_output($output);
} else {
echo $output;
}


log_message('debug', "Final output sent to browser");
log_message('debug', "Total execution time: ".$elapsed);
}


}


/* End of file Output.php */
/* Location: ./system/libraries/Output.php */


3.  Third and last hack is about get API that pull content from memcached.



public function get($key)
    {
        //How long do we allow for requests to hold for a cache miss. Just in case
        //a request thread dies because mysql went away or something
        $maxWait = 10;
        //How many times should a request poll for the key. CPU/response time tradeoff.
        $iterations = 50;

        $sleeptime = ($maxWait / $iterations) * 1000000;
        $v = $this->_cache->get($key);
        if (!$this->isSuccess()) {
            $lockKey = "::lock::$key";
            //Of all the requests slipping through varnish, only one may pass from here ...
            if (!$this->_cache->add($lockKey,  1, $maxWait)) {
                $i = 0;
                //... The rest will block here, polling until the key is removed ...
                while(true) {
                    usleep($sleeptime);
                    $i++;
                    if (!$this->_cache->get($lockKey) || $i == $iterations) {
                        break;
                    }
                }
                // ... then hopefully get a cache hit here. Technically, one could change
                //this to $this->get($key), which would recursively repeat the process.
                //In my case, I considered this unsafe as this should never happen except
                //if an E_FATAL_ERROR occurred or something similar.
                $v = $this->_cache->get($key);
            }
        }
        return $v;
    }


Let me know if you guys have any query or concern.


Happy Coding !!!

PHP APC locking mechanism

The locking mechanisms that APC can currently use are:

1. pthread locks
2. pthread mutexes
3. pthread locks
4. pthread mutexes
5. Slim reader/writer locks (Windows only)
6. spinlocks

Stick to the default locking mechanism unless APC won't compile. Brian Shire at Facebook tested locking mechanism performance and presented the results.

As of APC 3.1.9, the PECL installer ask about enabling three additional options: memory protection, pthread read/write locks and pthread mutexes, which correspond to --enable-apc-memprotect, --enable-apc-pthreadrwlocks and --enable-apc-pthreadmutex. The first two are labeled experimental and disabled by default; the latter is enabled.

There are another two options.

1. enable-apc-filehits
2. enable-apc-spinlocks

The first option (--enable-apc-filehits) enables gathering information for the "filehits" option of apc_cache_info. Basically, you can use it to figure out which files are pulled from the cache for each request if you're debugging cache-related problems. If cache_type is "filehits", information about which files have been served from the bytecode cache for the current request will be returned. This feature must be enabled at compile time using --enable-filehits.

The second option (--enable-apc-spinlocks), spinlocks are a processor-cycle inefficient way of ensuring only one process accesses a resource at any given time. APC uses locks when dealing with shared memory. APC places the cache in shared memory so that all the PHP processes can, well, share the cache, and locks ensure that the processes don't trip over each other while doing it.

Another important setting is apc.shm_segments APC allows you to create multiple shared memory segments using the apc.shm_segments setting to get around shared memory segment size limits.The default is 1 segment, 32MB in size.

You should check the kernel max size for shared memory.

sysctl -a | grep shmmax

This could explain apache crash when you set a too big number (but you should have something about it in error.log). If you've configured APC to use memory mapped files (with the --enable-mmap option), the shm_segments setting is ignored unless you specify a file mask via the mmap_file_mask setting. Which could be fixed by giving a value to apc.mmap_file_mask configuration setting.

some versions of APC (3.0.14) seems to ignore the value of apc.shm_segments and go with the apc.shm_size, and also it accepts values larger than the maximum allowed size for each segment.

When you allocate more the 32M apache should hang. Below is some standard APC settings which will be useful.

• apc.enabled=1
• apc.shm_segments=3
• apc.shm_size=32
• apc.ttl=7200
• apc.user_ttl=7200
• apc.num_files_hint=2048
• apc.mmap_file_mask=/tmp/apc.XXXXXX
• apc.enable_cli=1
• apc.max_file_size=10M

XDebug is not suitable for production environments

XDebug PHP extension, is a great, open-source debugger and profiler for PHP. It is a good alternative to the commercial Zend server. XDebug besides its profiling capabilities, it adds full stack traces to PHP error messages when enabled.These include Fatal Errors and Warnings.


however it is not suitable for production environments as Zend server is. 


Few people recommend that using XDebug on demand (remote debugging) does not harm server performance but there are major flaws here.


1. Basically  Xdebug comes with simple command line client for the DBGP protocol. We append  XDEBUG_SESSION_START in url and php script get that query string and xdebug start debugging and print output.


2. XDebug keeps traces in memory, which can lead to “out-of-memory” errors in

PHP

3. Imagine if php had to create a 100MB file for every request. By turning on xdebug.profiler_enable_trigger, you can at least selectively run the profiler.

4. Even when the profiler is off, the other primary function of xdebug (detailed stack traces on errors) causes your php instance to track lots of extra information slowing down execution. On a recent real world example where I turned on xdebug temporarily (with profiler off) and then shut it off, xdebug caused average response time to go from 400ms to 700ms.

5. There is still a bit (of overhead), about 10%. Xdebug 2.2 (currently SVN-only) have some improvements for this with (xdebug.coverage_enable=0) checkout this https://twitter.com/xdebug/status/50262398933798912

FAMOUS PERSONALITIES From Unnao U.P.

FAMOUS PERSONALITIES From Unnao U.P.

RAO RAM BUX SINGH
	Also known as Babu Rao Ram Bux Singh was a great freedom fighter of Biaswara, Unnao. He was talukdar of Daundia Khera, and fought for the freedom of the country against the British. But ultimately his forces were defeated and he was cought at Varanasi and hanged to death in Daundia Khera on 28th of December 1858.
	dd
CHANDRIKA BUX SINGH
	Born in 1923, son of the King of Bethar State, Ajit Singh, Chandrika Bux Singh came to the thrown after the untimely death of his father. He gave a tough fight to the British during the First war of Independence in 1857. Afterwards also he kept on his struggle. He killed the British Commander Murray and his wife and as a consequence he was arrested alongwith his family. The British decided to kill him in the most cruel way. He alongwith his family and associated was sentenced 'Kalapani ki Saja' as per the judgement on 28th Dec.1859. On 29th Dec,1859, it was ordered that his entire property be auctioned alongwith the above punishment. On his way to Kalapani, he was killed by the British on 30th Dec.1859.
Shams-ul-ulema mohd.abdul jalil usmani (born 1862 at Neotani, Unnao)
	Shams-ul-ulema mohammad abdul jalil usmani was bron in neotani and he was the professor of persian in college benaras persentely called it benaras hindu university.he was a versatile scholar in arabic, persian and urdu.he was among the six shams-ul-ulmas(grand scholar) in India.This honor was conferred upon him by the britsh governor lord elgin on the dated 22th june 1897 for his educationist views and intellectualities.He was a true scholar and patriot and had a deep vision to education for all classes. He got constructed two palatial building in neotani which are still intact and loan reminder of his love to native place. His grandsons are still living in nyotini.
MAULANA HASRAT MOHANI (born 1881 at Mohan, Unnao)
	Was a freedom fighter and Urdu Poet. After his graduation in 1903, inspired by Rao Ram Bux Singh, he threw himself into the freedom struggle. He took up the Editing of the newspaper 'Urdu-A-Muallah' in 1913 and continued his struggle through his articles. As a consequence he was kept in imprisonment from 1914 to 1918 but he continued his writing work in jail. He was nominated the member of the State Assembly in 1946. On 15th May 1951, this patriot breathed his last at Lucknow and left a void which is difficult to fill.
CHANDRA SHEKHAR AZAD
	A great freedom fighter and revolutionary, he belonged to village Badarqa where his parental house still exists. He had vowed that he would not be caught alive by British forces. On 27th Feb. 1931, he was rounded up by British forces at Alfred Park Allahabad. When he realised that it would not be possible for him to escape, he shot himself by his Mauser pistol and fulfilled his vow. February 27th. The significance of this date lies in the fact that today in the year 1931 a valiant patriot by the name of Chandrashekhar Azad attained martyrdom while fighting the British so that we Indians could rejoice in freedom. He joined Gandhi’s Non-Cooperation Movement but the violence perpetrated by the British infuriated him to such an extent that he turned a revolutionary. He joined the Hindustan Socialist Republican Association (HSRA) as a freedom fighter and later became its leader.Azad was involved in the Kakori train robbery, the Central Legislative Assembly bomb incident, the Delhi conspiracy and the shooting of British Police Superintendent Saunders at Lahore (1928) to avenge the killing of Lala Lajpat Rai. On February 27, 1931 he shot himself dead while bravely fighting the British. He was betrayed by an Indian. He lived upto his self-pride in not surrendering and in not succumbing to the enemy’s bullets. His fearlessness is signified by the fact that he continued fighting an army of British soldiers for several hours before sacrificing his own life when there was no way out, thus setting a rare example of courage and devotion to the country. He was truly a One-Man Army. The question that we all need to ask ourselves today is whether or not we are protecting the freedom that is soaked with the blood of revolutionaries like Chandrashekhar Azad, Bhagat Singh, Rajguru, Sukhdev and the list goes on an on. How easily do we take this independence for granted and become happily oblivious to the fact that we would still have been slaves had it not been for brave young men like Azad. It is different that we are still besieged by evils like poverty, population, prostitution etc.
VISHAMBHAR DAYAL TRIPATHI (5.10.1899 - 18.11.1959)
	Born at Bangarmau, Unnao , he was a great man in all aspects of life. He was a brilliant scholar, being a Gold Medallist from Banaras Hindu University. His enormous sacrifices for the country can be compared with any great leader or freedom fighter of the Nation. He was a close associate of Subash Chandra Bose He was several times sent to jail during the freedom struggle.He was jailed for 6 months during the 'Namak Satyagrah Aandolan' in 1930. He was fined Rs. 450 and sent to jail for 4 months in 1931 in connection with the 'Pipri Kand'.He was again jailed for 6 months for his participation in 'Lagan bandi Aandolan' in 1932.He was jailed for 6 months in 1933 during the 'Savinay Awagya Aandolan'.He was jailed from 1942 - 1945 during the 'Krips Mission'.
	He was- The Chairman of the Committee which was sent to Andman Nicobar to assess the possibility of development over there. The Chairman of the Zamindari Abolition Committee. The Chairman of the recognition Committee of High School & Intermediate Board U.P. The founder of D.S.N. College, Unnao and opened many other educational institutions in the District. In 1946 Vishambhar Dayal Tripathi was elected to the Legislative Assembly and was the Member of Parliament from 1952 to 1959 till his death (18th Nov.1959)
Padma Vibhushan Pt. UMA SHANKER DIXIT
	Born on 12th January 1901 at village Ugu of Unnao district, he got his education from Kanpur. Since his student life he joined the freedom movement and was the Secretary of the District Congress Committee Kanpur during the period when Sh. Ganesh SDhanker Vidyarthi was the President of the Committee.  He served the Country as the Home Minister, Health Minister and Governor of Karnatak & West Bengal. He also served as treasurer of All India Congress Committee, and Managing Director of Associated Journals at Lucknow. He founded a Girls Intermediate College at his village Ugu in the memory of his mother.
Pt. DWARIKA PRASAD MISRA
	A great freedom fighter & diplomat, he belonged to village Padari in Unnao. As a poet he composed Mahakavya- 'Krishnair'. He became the Chief Minister of M.P. after Ravi Shanker Shukla.
SURYA KANT TRIPATHI 'NIRALA' (21.2.1897 - 15.10.1961)
	'Nirala', a 'Chhayawadi' Poet, was born in Madinipur district of West Bengal. Thereon he shifted to Lucknow and then to Village Gadhakola of District Unnao, to which his father originally belonged. He was well versed in Hindi, Bangla, English & Sanskrit and was greatly inspired by Ram Krishna Paramhans, Swami Vivekanand & Ravindra Nath Tagore. His wrote strongly against social injustice & exploitation in the society. It will not be wrong to say that he was well ahead of his time in his work. There is a Park(Nirala Uddyan), an Auditorium (Nirala Prekshagrah) and a Degree College (Mahapran Nirala Degree College) in the district after his name.
	'Janmbhumi', 'Parimal', 'Geetika', 'Anamika' , 'Tulsidas', 'Kukurmutta', 'Adima', 'Bela', 'Nai Pattey', 'Archana', 'Aradhana', 'Geet Gunj', 'Sandya Kakli' ( All Poetry) 'Apsara', 'Alka', 'Prabhawati', 'Nirupama', 'Kulli Bhat', 'Billesur Bakriha', 'Choti ki Pakar', 'Kale Karname' (All Novels) 'Lily', 'Sakhi', 'Sukul ki Bibi', 'Devi', 'Chaturi Chamaar' (All Story books) 'Bangbhasha ka Uchcharan', 'Ravindra-Kavita-Kannan', 'Prabandh-Padya', 'Prabandh-Pratima', 'Chabuk', 'Chayan', 'Sangrah' (All Essays) 'Mahabharat' (Epic Katha) 'Anand Math', 'Vish-Vriksh', 'Krishna kant ka Vil', 'Kapal Kundala', 'Durgesh Nandini', 'Raj Singh', 'Raj Rani', 'Devi Chaudharani', 'Yuglanguliya', 'Chandrasekhar', 'Rajni', 'Sri Ramkrishna Vachnamrit','Bhatrat Main Vivekanand', 'Rajyog' (All Translations)
BHAGWATI CHARAN VERMA (30.8.1903 - 5.10.1980)
	A great writer,Born at Safipur, Unnao, Was Hindi Advisor at Akashwani, Lucknow in 1950 He was the member of Rajya Sabha in 1978
	Awards 'Sahitya Akademy Award' in 1961 for his book 'Bhule Bisre Chitra'. 'Sahitya Vachaspati Upadhi' in 1969 'Padma Bhushan' in 1971.
	Publication 'Bhule Bisre Chitra' 'Chitralekha' (The Film 'Chitrlekha' was based on this novel.) 'Prashan Aur Marichika' 'Chanakaya' 'Seedhi Sachchi Baten'
NAND DULARE BAJPAYEE (27.8.1906 - 21.8.1967)
	Born at Magrayer, Unnao, was a famous Critic.Was Vice Chancellor at Vikram University, Ujjain.
	Was Editor of :-'Bharat' during 1930-33 'Soor Sagar' during 1933-36'Ram Charit Manas' during 1937-39 Was Head of Hindi Deptt. at Kashi Hindu Vishwavidhalaya from 1941-1947.
	Publication  'Hindi Sahitya' 'Biiswin Shatabdi'  Aadhunik Sahitya' 'Naya Sahitya Naye Prashna' 'Kavi Nirala''Kavi Prasad''Maha Kavi Tulsidas'                                                                     Top
RAM VILAS SHARMA (born 10.10.1912 at Unchagaon Sani, Unnao)
	Hindi Scholar, MA English Literature, PhD. Retd.Prof.& Head Dept. of English, Balwant Rajput College,Agra Former Director K.M.Munshi Hindi Vidyapeeth, Agra Former Gen.Secy. 'Akhil Bhartiya Pragatisheel Lekhak Sangh' (1949-53) Editor of 'Samalochak' (1958-89)
	Awards      Reciepient of 'Sahitya Akademy Award' in 1970. 'Vyas Samman' from K.K. Birla Foundation in 1991. 'Bharat Bharti Samman' from U.P. Govt. 'Shalaka Samman' from Hindi Academy, Delhi & and many other awards. But as a noble gesture he refused the prize money of many awards with a request that it may be utilised in the promotion of literacy.
	Publications    More than 30 Hindi Publications. 'Prem Chand Aur Unka Yug' in 1952. 'Pragatisheel Sahitya Ki Samasyayen' in 1955 'Nirala Ki Sahitya Sadhana' ( 3 Vol.) during 1969-76 'Acharya Ram Chand Shukla Aur Hindi Aalochana' in 1973 'Bharat Mein Angareji Raj Aur Marxvad' in 1982 'Roop Tarang Aur Pragatisheel Kavita Ki Vaycharik Prishthbhoomi' in 1990 'Bhartiya Sahitya Ki Bhoomika in 1996. 'Itihas Darshan' in 1997. (The above are all critics & history of literature.) 'Pragati Aur Parampara' in 1953 'Aastha Aur Saundarya' in 1960 'Bhasha Aur Samaj' in 1961 'Bharat Ki Prachin Bhasha Parivar Aur Hindi' in 1979 (The above are all essays) 'Apni Dharti Apne Log' (Autobiography : 3 Vol.) in 1996 'Swadhinta Sangram - Badelte Pariprekshya' in 1992.
SHIV MANGAL SINGH SUMAN (born 5.8.1915 at Jhagarpur, Unnao)
	Hindi Poet, M.A.,PhD in Hindi (both from B.H.U.) Presently Exec. President Kalidas Academy, Ujjain. Career    V.C., Vikram University, Ujjain during 1968-78. Vice-President U.P. Hindi Sansthan, Lucknow , Press & Cultural Attache, Indian Embassy, Kathmandu Nepal during 1956-61 President, Association of Indian Universities, New Delhi during 1977-78 Visited Nairobi (Kenya), Mauritius, Nepal, Afganistan, erstwhile USSR, France, England, Mangolia, Australia and Several other Countries.
	Awards  'Deva Puraskar' in 1958   'Soviet Land Nehru Award' in 1974  'Sahitya Akademy Award' in       1974'Shikhar Samman' in 1993 from M.P. Govt.'Bharat Bharti' Award in 1993 and several others.
	Hons.    Conferred 'Padam Shri' by the Govt. of India in 1974                D.Litt. by Bhagalpur University in 1973 & by Jabalpur University in 1983.
	Publications 'Hillol' in 1939 'Jeevan ke Gaan' in 1942 'Yug ka Mol' in 1945 'Pralay Srajan' in 1950 'Vishwas Badalta hi Gaya' in 1948 'Vidhya Himalaya' in 1960 'Mitti Ki Baarat' in 1972'Vani ki Vyatha' in 1980'Kate Anguthon ki Vandanvaren' in 1991 The above are all Poetry. 'Mahadevi ki Kavya Sadhana' in 1951'Geeti Kavya : Udyam Aur Vikas' 1995' (Both Critic) 'Prakriti Purush Kalidas' in 1961 (Play)
BHUPENDRA NATH SHUKLA (born 16.5.1918 at Hafizabad , Unnao)
	Hindi Writer, BA,LLB Participated in 1942 movement and imprisoned. Former Secretary & President, City Congress Committee, Unnao Member UP Congress Committee for 20 years. Former President Bar Association, Unnao & Trustee of a number of Educational and social organisations. Former President Zila Parishad Unnao. Founder Member of V.D.Tripathi Hindi Vidyapeeth Unnao.
	Awards Reciepient of 'Bhartendu Harishchandra Award' for Drama.(thrice) 'Sahitya Bhushan Award' by U.P. Hindi Sansthan. 'Sahitya Manishi Award' by Nirala Mahavidyalaya Unnao.'Sahityashree', 'Nirala Samman', 'Naveen Samman'.
	Publications  'Brah Ke Geet' in 1943  'Killol', in 1988, 'Madhvi' in1986 (both Poetry) 'Gaon ka Aadmi' in 1988, 'Dhanpati' 1989, 'Shripati', 1991 (all novels)'Yug Seemayen' in 1987,  'Pryambada' 1991, 'Shaktibodh' 1988 (all plays)'Yug Nirmata Nehru' in 1989 (Biography) 'Bahadur Shah Zafar' in 1981 'Valley of Death' (an English Novel based on Ayodhya) in 1994 'Vaidehi Parinaya', Yugantar''Lashon Ka Maqbara' (a Hindi Novel based on Ayodhya) in 1991 'Siyaar Daroga' (Short story for children) in 1994
MAIRAJ ZAIDI (born 22.11.1949 at Chaudhrana, Unnao)
	Stage Plays  'Agra Baazar' directed by Habib Tanveer (68 shows ). 'Charan Das Choor' directed by Habib Tanveer (8 shows). Contribution : He acted in these Plays. 'Curfew' adopted from Novel 'Shahar Mein Curfew' written by Vibhuti Narain Rao 'Tamacha' 'Raj Darshan'
	Contribution : Direction, Screen Play, Acting & Dialogue Writing. T.V. Serials :'Raja Ka Baaja' directed by Sayeed Mirza (27 episodes have been telecast on DD-I) 'Farz' : Nimbus Productions (17 episodes.)'Chamatkaar' directed by Partho Ghosh. (6 episodes) 'Rishtey' (Telecast on Sony) Contribution: Dialogue Writing. 'Ghadar - 1857' based on The First war of Independence directed by Sanjay Khan. Contribution: Research Story, Screen Play & Dialogue Writing. Documentary Films: 'Pehli Baar' directed by Sayeed Mirza (Telecast on Bhopal Doordarshan) Contribution: Dialogue Writing.
Pd. PRATAP NARAIN MISHRA ( 1856 - 1894 )
	Born at Baijegaon, Unnao, a renowned Poet and Writer of 'Bhartendu Mandal', he has more that 50 books under his name. He was the editor of 'Brahmin', a mothly magazine. All his publications have been compiled into 'Pratap Narain Mishra Granthawali' by 'Nagri Pracharini Sabha', Kashi. He was well versed in Sanskrit, Urdu, Farsi, English and Bangla. He was a humourous person and had a lifestyle of his own. Publications :'Prem Pushpawali', 'Man ki Lehar', 'Trapyantaam', 'Vraidla-swagat', 'Shaiv Sarwasva', 'Kavita Lehari' 'Hathi Hameer', 'Kali Kautuk', 'Bharat-Durdasha' (All Plays)
JAGDAMBIKA Pd. MISHRA 'Hitaishi' ( 1895 - 1956 )
	'Shahidon ki chitaon par judenge har baras mele.' 'Watan par marne walon ka yahi baki nishan hoga..' Above are the popular lines of his poem 'Matra Geeta' which was banned by the British Government. Born at Ganj Moradabad, Unnao he was popular for applying 'Savaiya - Chhand' in 'Khari boli' for which he got appreciation from great personalities of his time namely Acharya Mahavir Pd. Dwivedi and Acharya Ram Chandra Shulka. He had a strong hold of Sanskrit, Bangla and Farsi.
	Publications: 'Kallolini' 'Matra Geeta' 'Baikali' 'Darshana'
RAMAI KAKA ( 2.2.1915 - 18.4.1982 )
	Born at Rawatpur village of Unnao in a ordinary agriculturist family, Ramai Kaka was a Comedy Poet and an Artist. After passing his matriculation, he worked as Inspector in Planning department. Due to his illness he took early retirement from service in 1938. He was offered to perform on 'Aakashvani' in 1940. His performance as an actor in the programme 'Bahire Baba' on Aakashvani Lucknow earned his great fame. His programmes were also broadcasted on BBC London. During his 35 years of service in Aakashvani, he wrote Plays, comedy plays, 'Nautanki', and a number of Poems. He was a simple person who held high moral standards
	Publications: 'Bauchhar''Bhinsar''Netaji''Fuhar''Harpati Tarwar''Gulchharra' 'Hasya ke chhente' 'Mati ke bol' Awards: He was honoured with a special award of Rs.5000/- by 'Rajya Hindi Sansthan' for his contribution to Hindi literature.
Suhail Zaidi (born 31.12.1974 at Chaudhrana, Unnao)
	Born at Chaudhrana, Unnao in a Talukedar family, a simple person who held high moral standards. Producer, Project Head & Executive Producer. T.V. Serials As Project Head “Jhansi Ki Rani” ( ZEE tv) “Chand Ke Paar Chalo” ( NDTV Imagine) “Kumkum” ( Star Plus)
	T.V. Serials As Producer  'Maryada' (395 episodes have been telecast on DD-I) 'Talaash' : (52 episodes have been telecast on DD-1.) 'Hairat' (13 episodes have been telecast on DD-1) T.V. Serials As Executive Producer “Zameer” (52 episodes have been telecast on DD-1) “Safar” ( 39 episodes have been telecast on DD-1)
	Production:   “Aaj Tak News” (India Today Group) “Metro Club” (a daily youth magazine have been telecast on DD-2)

Review on MYSQL Locks

We have seen so many times locked tables in show process list, obviously it's panic to see locked tables as such cases finally stop your application causes Apache's threads count huge number, In this post we discuss about type of locks and how can we avoid them. Before going start  I’m happy to say I’m not a MySQL expert in every aspect of MySQL, specifically internals where I have not had the detailed time to read the code, and understanding all internal workings.

The main important thing is instrumenting and monitor your mysql in better way.

1. Slow Query Log
2. General Query Log
3. Binary Log
4. Connectors
5. Process-list
6. Engine status
7. Status variables

There are two classes of locks:

READ LOCK (SHARED LOCK)
Read locks as the name suggests, are locks taken on resources to provide consistent read by blocking writes to that resource. The read locks are also known as shared locks because many clients can acquire read locks on the same resource simultaneously. The other thing to know is that, while read lock is in place, no write lock can be acquired on that resource.
Following is how MySQL will grant read lock:

if there is no write lock set on the resource grant the read lock immediately
if there is a write lock on the resource, put the lock in the read lock queue


WRITE LOCK (EXCLUSIVE LOCK)
Write lock is taken on a resource when it needs to be modified. There can be only one write lock on a resource at a given time. While a write lock is held on a resource, all other read lock requests must also wait.
Following is how MySQL will grant write lock:

if there is no (read/write) lock on the resource, grant the write lock immediately
if there is a lock on the resource, put the write lock in the write lock queue

PRIORITY OF READ AND WRITE LOCK
Write lock has a higher priority than read lock. When a resource is unlocked, and if there are lock requests waiting in the queue, then the lock is granted in the following manner:

grant the lock first to the request waiting in the write lock queue
if there is no lock request for the resource in the write lock queue, then grant the lock to the first request in the read lock queue

Locking in MyISAM

MyISAM supports table-level locking and does not support row-level locking. Because of this behavior, MyISAM is typically well suited for applications which have a high percentage of reads as compared to writes. But because MyISAM supports locking at a high level, the memory needed for locking is typically less, say when you compare it to row-level locks.

However, MyISAM does allow concurrent INSERTs in some situations!

CONCURRENT INSERTS
If there are no holes in MyISAM table, then INSERTs happen at the end of the table, and in such a situation INSERTs and SELECTs are allowed to happen concurrently on the same table, with INSERT needing no write locks. But there is a caveat, if there are many INSERT requests, then the INSERTs are done serially, which implies that only one INSERT can execute concurrently with SELECTs on the same table. Now what exactly are hole? Holes are produced when rows are deleted
and/or  updated in the middle of a MyISAM table.
 
Locking in InnoDB
 
InnoDB unlike MyISAM supports both table-level locking as well as row-level locking, which allows for a more fine-grained control over the resource to be locked. Row-level locking allows InnoDB to be extremely efficient in case of heavy write-load.
InnoDB also has a very nice feature MVCC, which allows for non-locking consistent reads. This is achieved by having different snapshots of data available to different transactions. However, if you have the transaction isolation level  set to SERIALIZABLE then plain SELECTs are automatically converted to locking SELECTs.

However, AUTO_INCREMENT columns need table-level locks.

TABLE-LEVEL AUTO_INC LOCKS

INSERTs into AUTO_INCREMENT column need table-level lock for the duration of the INSERT statement. This is so that the INSERTs are safe for statement-based replication. However, as of MySQL >= 5.1, which also introduced row-based replication, this behavior is configurable using the variable innodb_autoinc_lock_mode, but you still need to be using  row-based replication for the INSERTs to be safe for replication. Set the value of innodb_autoinc_lock_mode to 2, and
then INSERT like statements will not take any table-level locks.


Imagine The Scenario


0.00 seconds A select query (ie. read-only) accesses the table, it will take around 2 seconds to complete.
0.01 seconds Another select query accesses the table, it takes no time and will complete in an instant as it can run in  parallel.
0.02 seconds An insert, delete or update query (ie. write) attempts to write to that very same table before the first select has completed.
0.03 seconds A select query comes in, again waiting for the first select to complete.
0.04 seconds Another select query comes in, again waiting for the first select to complete.
0.05 seconds Yet another select query comes in, again waiting for the first select to complete.
1.99 seconds (~1000 queries later) Yet another select query comes in, again waiting for the first select to complete. And so on and so forth.

The queries in red are blocked and have to wait for the first select to complete before they can be executed. The selects in red are blocked by the write operation in orange which is in turn blocked by
the long-running select in green. This is what causes the table_locks_waited value to grow.

The following items describe some ways to avoid or reduce contention caused by table locking:

• Try to get the SELECT statements to run faster so that they lock tables for a shorter time. You might have to create some summary tables to do this.
• Start mysqld with --low-priority-updates. For storage engines that use only table-level locking (such as MyISAM, MEMORY,and MERGE), this gives all statements that update (modify) a table lower priority than SELECT statements. In this case,the second SELECT statement in the preceding scenario
• would execute before the UPDATE statement, and would not need to wait for the first SELECT to finish.
• To specify that all updates issued in a specific connection should be done with low priority, set the
• low_priority_updates server system variable equal to 1.
• To give a specific INSERT, UPDATE, or DELETE statement lower priority, use the LOW_PRIORITY attribute.
• To give a specific SELECT statement higher priority, use the HIGH_PRIORITY attribute. 
• Start mysqld with a low value for the max_write_lock_count system variable to force MySQL to temporarily elevate the priority of all SELECT statements that are waiting for a table
• after a specific number of inserts to the table occur. This permits READ locks after a certain number of WRITE locks.
• If you have problems with INSERT combined with SELECT, consider switching to MyISAM tables, which support concurrent SELECT and INSERT statements.
• If you mix inserts and deletes on the same table, INSERT DELAYED may be of great help.
• If you have problems with mixed SELECT and DELETE statements, the LIMIT option to DELETE may help.
• Using SQL_BUFFER_RESULT with SELECT statements can help to make the duration of table locks shorter.
• You could change the locking code in mysys/thr_lock.c to use a single queue. In this case, write locks and read locks would have the same priority, which might help some applications.

Instant’s Auto complete Suggestions Review

Over the past couple of years, auto-complete has become famous over the web. Facebook, YouTube, Google, Bing, MSDN, LinkedIn,Apple and other websites try to complete text phrases as soon as we typing.

Auto-complete definitely makes for a nice user experience, but it can be a challenge to implement efficiently. In many cases, an efficient implementation requires the use of interesting algorithms and data structures. In this blog post, I will describe all about it.

Basic Idea

Basically Auto complete is combination of HTML,JS,CSS and data source. It could be JS string or a file or data fetching from db connection. we usually fetch data from source on the basis of JS events like onkeyup etc. 

Engineering Challenges

however we notice that Facebook or google's auto suggestions are working very fast or with good speed.  I encountered first time with speed issue when i worked for www.peerpower.com. Actually we had planed to use Auto suggester frequently to show master data in drop downs. see registration page http://www.peerpower.com/register.php. That time i implemented incremental caching feature at browser side but later i found that still i don't touch facebook and google standards.

However now some open sourse caching plugins are avalable by default. see http://docs.jquery.com/Plugins/Autocomplete and http://flexbox.codeplex.com.

Typically following are the key points which play game in efficient manner.

1.  Data structure to retrieve big data
2.  client side caching from memcache indexed javascript array (for example first letter as index, so friends['a'] = [andrey, albert] )
3. client side storage techniques (normally all modern browser support it )
4.  warm up caches as early as it can
5. build an index of names -> dom elements, do dom manipulation offline and attach the results with only people that match the searched term

Data structure to retrieve big data

Ternary search trees Trie is used to retrive data for auto-complete lookup but needs too much memory. Solution for it PATRICIA (Practical Algorithm to Retrieve Information Coded in Alphanumeric). http://www.csse.monash.edu.au/~lloyd/tildeAlgDS/Tree/PATRICIA/

Both are data structues so you can read them about in details separately. 

Other Challenges

Suggestions Can Vary By Region & Language

Not everyone sees the same suggestions. For example if i am in delhi then i will see delhi based results with high priority.

Previously Searched Suggestions

Big demand in suggestor product to track how are things going.

How Suggestions Are Ranked

Big question if you plan to put suggestor at home page.Popularity is a factor, but some less popular searches might be shown above more popular ones.

Deduplicating & Spelling Corrections

Its basic functionality that everyone dreams in auto suggestor :P

Freshness

If there are terms that suddenly spike in popularity in the short term, these can appear as suggestions, even if they haven’t gained long-term popularity.

Dev team with no QA resources

Right now in my organization, there are completely independent roles for QA and Dev. QA vs Development ratio is always an open issue, specially in crucial prod releases :P


In my office, we are thinking about DEV without QA i.e. push releases on prod without/partial help of QA. 


Overall idea is avoid QA using either TDD, peer programming or peer code review etc. 


Here are few key notes ( my worries :P however they are very subjective, but based on experience)

1. Can developers really be effective testers, with their intimate knowledge of the application?
2. Can developers could be a good "bug finders" rather than engineers.
3. People are often blind to their own shortcomings or mistakes, and developed code is best validated when tested by third parties. Of course, proper monitoring and management of this process could mitigate the process...but it's a concern.
4. Can developer think high visibility prospective. Sometimes a customer relationship can be impaired by simple usability flaws -- a somewhat common mistake for an immature developer.
5. Increases the chance for rework
6. Leads time involved more on testing application or module

However following are few parameters which helps.


How much unit testing the developers are doing. The more they do, the less QA dependency.


How much the developers are writing from scratch versus leveraging existing libraries. If there is a lot of pre-existing code in use then its disappointing call.


How dynamic the development is. If you are writing a UI where relatively small developer tweaks cause a large change to the testable surface then it would be pain.


How mission critical the feature is. It need a lot more testing because bugs are hard to fix in the field and very bad when they happen.