Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Thursday, November 24, 2011

ZMQ socket creation test

Working with Harish aka ZMQ guru :)

We needed to use ZMQ for a lot of backend queueing operations from the serving PHP code, I started inspecting the ZMQ-PHP bindings (docs : http://php.zero.mq/ ) .

On close inspections we found that the sockets do not persist over multiple PHP requests*** but persist for multiple requests to the queue within the same request. So we have to make new sockets equal to the different queues we will be connecting for every request. To test whether this is optimal, Harish suggested a test as follows : 

<?php
for($i = 0; $i < 10000; $i ++) {
    
    $pid = pcntl_fork ();
    if ($pid == - 1) {
        
        die ( 'Fork failed' );
        break;
        
    } elseif ($pid == 0) {
        
        $context = new ZMQContext ();
        $socket = new ZMQSocket ( $context, ZMQ::SOCKET_PUSH, 'mySock' );
        
        $socket->setSockOpt ( ZMQ::SOCKOPT_HWM, 5 );
        
        $dsn = "tcp://1.1.1.1:22222";
        
        $socket->connect ( $dsn );
        
        break;
        
    } else {
        pcntl_waitpid ( $pid, $status );
    }
}
?>

We timed it using :

jknair@linuxbox:~/Projects/queue$ time php zmqFork.php 

real 1m32.174s
user 0m26.186s
sys 0m49.355s

and then without the ZMQ stuff :

<?php
for($i = 0; $i < 10000; $i ++) {
    
    $pid = pcntl_fork ();
    if ($pid == - 1) {
        
        die ( 'Fork failed' );
        break;
        
    } elseif ($pid == 0) {  

        break;
        
    } else {
        pcntl_waitpid ( $pid, $status );
    }
}
?>

After timing it :

jknair@linuxbox:~/Projects/queue$ time php zmqFork.php 

real 1m23.990s
user 0m10.629s
sys 0m37.450s

Approximately 10 seconds for 10000 ZMQ sockets.
1ms for a ZMQ socket !!! No problemo :) 


***Update : the sockets persist within the same preforked apache/nginx process that can server multiple php requests but do  not persist across mutiple processes 

Sunday, August 7, 2011

Node.js multi-node Proxy

For those who have no idea what is node.js , evented I/O for V8 , google's super fast javascript engine that powers chrome. Trust me its a completely different paradigm of coding. I wrote this simple node.js script that acts a proxy to multiple websevers and  multi-node for spawing multiple processes using all the cores on the machine. 

var http = require('http'),
request = require('request')

var port = "8080";

var address = ["1.1.1.1","1.1.1.2"]  //multiple node.js webservers delivering content with a delay of 200 ms
 
var server = http.createServer(function (req, res) {

var curr = address.shift(); //round robin the webservers

request.get({uri: "http://"+curr+":"+port+"/" ,
             maxSockets : 100 // the dreaded LINE which i did not include initially
}, function (err, response, html) {
        res.end(html);  
});

address.push(curr);

})

var nodes = require("multi-node").listen({
        port: 8090,
        nodes:30,  //no. of multiple child processes 
    }, server);


So now starts the long analysis in which small mistakes would lead me to various pitfalls. Obviously benchmarking this script was extremely essential. ApacheBench is stubborn to end on the first error so turned to siege as the benchmarking tool to emulate realtime traffic. Running this on a server with 16 cores and 32gb RAM. 

Nodes requestsPerSecond LoadAvg on the ProxyServer

30    1902.79                8.93
60    3724.89                33.26
100   6553.07                78.10
263   6560.79                223.43 ( had to increase ulimit beyond 70000 started getting EMFILE too many open files error) 
300   6800.36                260.87
500   6124.38                345.38 ( ulimit at 300000)



And we see that with more threads RPS is growing and so is the LoadAvg. Node is supposed to provide non-blocking I/O so increase in performance with threads more than the cores was not expected. After a "ps ax | grep node" which proved there were multiple processes running and lsof of each process showed me that all the processes are listening on the same port. Hey wait a minute I  was under the impression that only one process can bind to a socket for listening (mistake !!!). After a ton of reading socket manuals I was right : only one process can bind to a socket but forking "child_process" can listen on the same port. The OS acts as a load balancer delegating the requests coming on the port. So added the "f" flag and :
jknair@linuxbox:~/Projects/NodeHttpProxy$ ps axf | grep Node
 8783 pts/0    S      0:00      \_ node NodeHttpProxy.js
 8784 pts/0    S      0:00      |   \_ node /home/jknair/Projects/NodeHttpProxy/NodeHttpProxy.js
 8785 pts/0    S      0:00      |   \_ node /home/jknair/Projects/NodeHttpProxy/NodeHttpProxy.js


Now that I know they are child_processes still trying to figure out why the increase in RPS. After going beyond 200 nodes "Too many open files" error started popping up and then lsof-ing each pid there were many UNIX Sockets that were open. The loadavg could be because of high processor contention due to these unix sockets trying to maintain some state among all the processes. But reading through the underlying libraries didn't heed any such state maintaining features.
In vain I posted it on Google groups and then a solution to all the problems. Apparently the http.Agent defaults to a connection pool of 5 sockets for making the http requests to the backend servers. So a change to that dreaded line increasing the maxSockets :  4.5K RPS on 15 threads and 6.5K RPS using 30 threads and loadavg below 10. This was really fun to go about figuring out the underlying issue. 


UPDATE : after my colleague Harish went through this post he pointed out a lot of things I have missed out to mention.


1)  the last tests I was maxing out on network bandwidth.
2)  after removing the delay from the webservers I was getting  a 14K RPS.
3) I still have to push the proxy to max out on CPU and report test results when I able to achieve the same

Saturday, October 2, 2010

AWK, SED, GREP, FIND

The basics of these 4 utilities is a must know !!! I have found a few tutorials which atleast taught me the basics which HELPS a lot !!!




GREP : http://www.readylines.com/linux/grep/basics You will find a lot on grep but this one of the most basic one I could find.


Well after all this I couldnt resist but googling up python appended to each of these commands and well a few things to read ....




Sunday, August 22, 2010

C.S.I

Rahil messaged me "we raised our sponsorship amount to nearly 1 lakh !!!" I couldn't have been more happier. It was great to know the juniors have started to put all their efforts in growing the Frcrce C.S.I to greater heights. Then he lied about missing me in the council. I miss the running around and the chance to teach and learn among technical illiterates like me. The hustle bustle and the brunt of professors down our throats and being held responsible for screw ups and no gratitude for the accomplishments over the years are the most cherished memories of my 4 year technical massacre.
Some images of the C.S.I - OS installation seminar with Sasha Mac-ing away. Best of luck guys
Location Fr.C.R.C.E Bandra (w). Purpose was to make it easy for ignorant users to learn easy ways to install different OS's in an attempt to learn different systems and analyse their pros and cons.










Wednesday, August 18, 2010

Mouse


NOW IT WORKS !!!

P.S : thanks JJ for the mouse

Sunday, July 18, 2010

mountRB : Disk mounter for Rhythmbox

A very much needed feature that is missing in Rhythmbox the default music player in Ubuntu , mounting the drives that are part of the music library in Rhythmbox.

Having a huge music library and spread out in multiple drives it was quite irritating to mount the drives again and again at every startup whenever rhythms had to be outboxed :) . So I wrote a python script to parse through the rhythmbox library and mount the drives mapping the labels to the respective drives and a bash script to call this python script and also startup rhythmbox. A shortcut to launch the bash script added to the gnome panel made music a one click stop .



MAKE SURE TO HAVE LABELS ON EVERY DRIVE !!!

njoy !!!

PS: my first attempt