Lev Walkin (lionet) wrote,
Lev Walkin

Stress-testing httperf, siege, apache benchmark, and pronk

Pun intended. Today I am doing totally unscientific (but quite useful) comparison between a few well-known HTTP performance tools.

System: FreeBSD 8.2, x86_84 (L3426, no HT, no TB). Both client and servers were on the same host.

Load generators tested:
Server side is node.js (0.6.6) with the following code:
var app = require('http').createServer(handler);

app.listen(8081, "");

function handler (req, res) {
  return res.end('Success');

All load generators were first set up for single-connection operation (no concurrency), then I threw some concurrency in to up the ante.

Test results:
ToolKeep-Alive?Concur­rencyRequests madeTest durationRPS shownNotes
httperfYES1500 K41 s12.1 Khttperf's CPU usage was about 10% less than Node's: ~60% vs 70%
httperfNO1500 K93 s5.3 K
siegeYES1600 K71 s8.2 Ksiege's CPU usage was about 30% less than Node's: ~50% vs 75%
siegeNO1300 K88 s3.9 K
abNO11500 K104 s4.8 Kab's CPU usage was about 3x less than Node's: ~25% vs 80%
pronkYES21100 K112 s0.9 K3
std=1.5 K
pronk's CPU usage was about 20x greater than Node's: ~101% vs 4%
httperfYES100500 K39 s12.7 Khttperf's CPU usage was on par with Node.JS's
httperfNO2584500 K69 s7.2 K
siegeYES100300 K21 s11.7 Ksiege's CPU usage was about 5x less than Node's: ~20% vs 100%
siegeNO100300 K43 s7.4 Ksiege's CPU usage was about 2x less than Node's: ~50% vs 100%
abNO1100500 K63 s7.8 Kab's CPU usage was about 2x less than Node's: ~40% vs 99%
pronkYES2100500 K81 s7.0 K
std=5.1 K
pronk's CPU usage was about 3x greater than Node's: ~160% vs 60%
pronkYES210500 K110 s4.7 K
std=5.3 K

[1] ab utilizes HTTP/1.0 when calling the server, and in this mode the Keep-Alive header is ignored by Node.JS, quite surprisingly. Effectively, ab's "-k" option (to enable HTTP keep-alives) is a no-op if used against Node.JS based servers.
[2] pronk can't be configured to disable keep-alive mode.
[3] pronk is very inefficient when -c is low (less than a number of cores).
[4] httperf cannot be configured to utilize a fixed number of connections. So concurrency is what's measured after trying to connect with --rate <rate>.

I learned something new today.

httperf is not always right for Node.JS

httperf once again shown itself being the best testing tool for high-performance web servers... when it is not used against high-performance GC language based non-keep-alive servers. In such unlucky scenario GC pauses trigger creation of new FDs, which causes httperf to choke a bit with managing its select() masks[4]. So, httperf might not be able to squeeze the max performance out of non-keepalive, low latency Node.JS applications. ab is somewhat better, but not without its own quirks.

ab cannot fully test Node.JS

Apache Benchmark utilizes HTTP/1.0 when talking to the server. When "-k" option is given, it just adds "Connection: Keep-Alive" header. Unfortunately, Node.JS server ignores keep-alive if not invoked through an HTTP/1.1 connection[2]. This removes an opportunity to test how Node.JS based applications work in a keep-alive mode.

siege is a decent tool

siege starts a tad slower in a single-user, but becomes more competitive in higher concurrency setting. No quirks discovered.

pronk is fun, but quirky

The relatively new pronk is not well suited for testing efficient (low latency) high performance servers. The reason is two-fold. First, pronk does not have this mode where it creates new connections for each new HTTP request (in this weirdness pronk is a complete opposite of ab, which can't keep it alive). Yet this non-keepalive operation is the most important mode to be tested in the high-load setting, since clients come and go in waves, and there is comparatively not too much opportunity to reuse the connection. Second, pronk is woefully inefficient (0.9 kRPS?!) for a single connection (-c 1). Although situation improves rather quickly with supplying higher -c values.

So, I take from that that I can't use pronk for testing Coser, our internal web server. To load it properly I had to resort to getting three physical boxes to run httperf in parallel in HTTP/1.0 non-keepalive mode.

Node.JS is flaky [on FreeBSD]

pronk -c 10000 causes it to quit:
        throw arguments[1]; // Unhandled 'error' event
Error: accept Unknown system errno 23
    at errnoException (net.js:632:11)
    at TCP.onconnection (net.js:817:24)
Tags: echo, haskell, js-kit
  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded