Pun intended. Today I am doing totally unscientific (but quite useful) comparison between a few well-known HTTP performance tools.

System: FreeBSD 8.2, x86_84 (L3426, no HT, no TB). Both client and servers were on the same host.

Load generators tested:

• httperf (0.9.0)
  
• siege (2.70)
  
• Apache Benchmark, ab (2.3)
  
• Pronk (git master; ghc 7.0.3)

Server side is node.js (0.6.6) with the following code:

var app = require('http').createServer(handler);

app.listen(8081, "127.0.0.1");

function handler (req, res) {
  res.writeHead(200);
  return res.end('Success');
}

All load generators were first set up for single-connection operation (no concurrency), then I threw some concurrency in to up the ante.

Test results:

Tool	Keep-Alive?	Concur­rency	Requests made	Test duration	RPS shown	Notes
httperf	YES	1	500 K	41 s	12.1 K	httperf's CPU usage was about 10% less than Node's: ~60% vs 70%
httperf	NO	1	500 K	93 s	5.3 K
siege	YES	1	600 K	71 s	8.2 K	siege's CPU usage was about 30% less than Node's: ~50% vs 75%
siege	NO	1	300 K	88 s	3.9 K
ab	NO1	1	500 K	104 s	4.8 K	ab's CPU usage was about 3x less than Node's: ~25% vs 80%
pronk	YES2	1	100 K	112 s	0.9 K3 std=1.5 K	pronk's CPU usage was about 20x greater than Node's: ~101% vs 4%
httperf	YES	100	500 K	39 s	12.7 K	httperf's CPU usage was on par with Node.JS's
httperf	NO	2584	500 K	69 s	7.2 K
siege	YES	100	300 K	21 s	11.7 K	siege's CPU usage was about 5x less than Node's: ~20% vs 100%
siege	NO	100	300 K	43 s	7.4 K	siege's CPU usage was about 2x less than Node's: ~50% vs 100%
ab	NO1	100	500 K	63 s	7.8 K	ab's CPU usage was about 2x less than Node's: ~40% vs 99%
pronk	YES2	100	500 K	81 s	7.0 K std=5.1 K	pronk's CPU usage was about 3x greater than Node's: ~160% vs 60%
pronk	YES2	10	500 K	110 s	4.7 K std=5.3 K

^[1] ab utilizes HTTP/1.0 when calling the server, and in this mode the Keep-Alive header is ignored by Node.JS, quite surprisingly. Effectively, ab's "-k" option (to enable HTTP keep-alives) is a no-op if used against Node.JS based servers.
^[2] pronk can't be configured to disable keep-alive mode.
^[3] pronk is very inefficient when -c is low (less than a number of cores).
^[4] httperf cannot be configured to utilize a fixed number of connections. So concurrency is what's measured after trying to connect with --rate <rate>.


I learned something new today.

httperf is not always right for Node.JS

httperf once again shown itself being the best testing tool for high-performance web servers... when it is not used against high-performance GC language based non-keep-alive servers. In such unlucky scenario GC pauses trigger creation of new FDs, which causes httperf to choke a bit with managing its select() masks^[4]. So, httperf might not be able to squeeze the max performance out of non-keepalive, low latency Node.JS applications. ab is somewhat better, but not without its own quirks.

ab cannot fully test Node.JS

Apache Benchmark utilizes HTTP/1.0 when talking to the server. When "-k" option is given, it just adds "Connection: Keep-Alive" header. Unfortunately, Node.JS server ignores keep-alive if not invoked through an HTTP/1.1 connection^[2]. This removes an opportunity to test how Node.JS based applications work in a keep-alive mode.

siege is a decent tool

siege starts a tad slower in a single-user, but becomes more competitive in higher concurrency setting. No quirks discovered.

pronk is fun, but quirky

The relatively new pronk is not well suited for testing efficient (low latency) high performance servers. The reason is two-fold. First, pronk does not have this mode where it creates new connections for each new HTTP request (in this weirdness pronk is a complete opposite of ab, which can't keep it alive). Yet this non-keepalive operation is the most important mode to be tested in the high-load setting, since clients come and go in waves, and there is comparatively not too much opportunity to reuse the connection. Second, pronk is woefully inefficient (0.9 kRPS?!) for a single connection (-c 1). Although situation improves rather quickly with supplying higher -c values.

So, I take from that that I can't use pronk for testing Coser, our internal web server. To load it properly I had to resort to getting three physical boxes to run httperf in parallel in HTTP/1.0 non-keepalive mode.

Node.JS is flaky [on FreeBSD]

pronk -c 10000 causes it to quit:

events.js:48
        throw arguments[1]; // Unhandled 'error' event
        ^
Error: accept Unknown system errno 23
    at errnoException (net.js:632:11)
    at TCP.onconnection (net.js:817:24)