Web server administration eventually reaches a moment when you need to test the performance of a solution, tune its settings, or check a hypothesis. Solving these tasks requires picking the right benchmarking tool, which is what this article is about. All the examples are based on the Angie server, but they can apply to other products too. This article covers simple, readily available utilities for quick tests.
Video version
For convenience, a video version of this article is available on YouTube.
Preparing the web server and the environment
Repeatability and reliability of benchmark results depend on environment and server preparation. One caveat up front: load must be directed at a server under your control, otherwise the testing can turn into a denial-of-service attack.
For Angie, the basic server parameters worth controlling are the ones that significantly affect test results (for HTTP):
- local logging of requests (
access_log) and errors (error_log) puts significant load on the disk subsystem; - number of worker processes (
worker_processes); - dynamic compression settings (
gzip, brotli, zstd); - cipher suites and certificate type (when using HTTPS);
- protocol in use (HTTP 1.1, HTTP/2, HTTP/3);
- loaded dynamic modules;
keep-aliveconnection settings (toward the client and the backend);- buffer sizes when proxying (
proxy_buffers); - server-side caching (
proxy_cache); - request limits (
limit_reqandlimit_conn); - the underlying operating system (libraries in use);
- network stack settings.
The settings and operating modes above must be repeated and applied the same way when comparing different configurations, versions, or products against each other. It also follows that you cannot rely on default settings: they may implement different functionality and target different goals. For example, many web servers and databases ship with defaults tuned to save RAM and run on weak hardware, which fits poorly with performance.
Besides the server side, the client side (the load source) is worth controlling too. Typical problems: hitting the load generator's own performance ceiling, exhausting network interface bandwidth, background load on the system. You can also run out of ephemeral ports on the client or server (for example, with the net.ipv4.tcp_tw_reuse option disabled), hit open file descriptor limits, and other logical limits. The first test run may stay within these limits, while the next one starts throwing errors from sockets in the TIME-WAIT state. As a result, benchmarks show lopsided results in favor of one of the participants purely because of environment preparation mistakes.
So any benchmark is not so much about running tests as it is about analyzing results and understanding the product's behavior and its interaction with its surroundings (the operating system, the hardware, the environment). That is exactly why performance testing is such a useful and interesting activity.
Benchmark types: closed-loop and open-loop
Before reviewing the utilities for testing web server performance, here is a classification of such tests by how they handle responses:
- closed-loop;
- open-loop.
Utilities that run in a closed-loop send requests at the rate at which they receive responses from the server. So when the server starts responding more slowly under load, the request rate drops. In effect, a closed loop depends heavily on server response time (latency), but does not always give complete data on server throughput. Examples of this type: ab, wrk, wrk2, h2load.
Open-loop testing tools generate load according to a configured request rate, without waiting for responses and without depending on server response time. If the load exceeds the server's capacity, expect errors in the responses (timeouts, connection errors, and so on). This mode emulates real user load more realistically. Examples of open-loop tools: Yandex.Tank, httperf, Vegeta. Some tools have several operating modes, for example Grafana k6 and Nighthawk.
The ab utility (Apache Benchmark)
Apache Benchmark (ab) is a simple utility with decent capabilities. It ships in the apache2-utils package (for Ubuntu).
A basic test runs with a single command:
ab -c 10 -l -k -n 5000 http://localhost/
The parameters of this command:
‑c 10— request concurrency level (emulating several clients);‑l— allow responses of varying length (otherwise differences are counted as errors);‑k— use keep-alive connections (disabled by default);‑n 5000— number of requests in the test (50000);http://localhost/— the request URL.
Since ab is a single-threaded utility, this is an emulation of parallel clients rather than an actual run of several threads. The utility is therefore limited by the performance of a single CPU core. When testing Angie, you can hit that limit even with a single worker process on the server.
The test result looks like this:
Benchmarking localhost (be patient)
Completed 500 requests
...
Completed 5000 requests
Finished 5000 requests
Server Software: Angie/1.11.4
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: Variable
Concurrency Level: 10
Time taken for tests: 8.129 seconds
Complete requests: 5000
Failed requests: 0
Keep-Alive requests: 5000
Total transferred: 54581671 bytes
HTML transferred: 53641671 bytes
Requests per second: 615.08 [#/sec] (mean)
Time per request: 16.258 [ms] (mean)
Time per request: 1.626 [ms] (mean, across all concurrent requests)
Transfer rate: 6557.09 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 2 16 29.4 13 205
Waiting: 2 16 29.4 13 205
Total: 2 16 29.4 13 205
Percentage of the requests served within a certain time (ms)
50% 13
66% 14
75% 14
80% 15
90% 17
95% 36
98% 158
99% 167
100% 205 (longest request)
The first part of the report shows the test settings and the totals. The main values of interest are requests per second (Requests per second) and response time (Time per request). Also compare the throughput used (Transfer rate) against the capacity of the network interface. If that value approaches the interface maximum, the test is invalid.
The response time distribution (percentiles) matters too. These values show the response time within which x% of responses fit. For example, in the results above the 95th percentile is 36 ms, meaning 95% of responses were received in 36 ms or less. Accordingly, the 100th percentile is the response time of the slowest request, and the 50th is the median.
When errors occur, the utility reports responses with a code other than 200 (non-200 replies). To ignore socket errors you can add the ‑r flag; otherwise the test stops at the first such error.
The ab utility supports HTTP/1.1 and TLS 1.2. It can use various HTTP methods (‑m), HTTP authentication (‑A), and mTLS (authentication with a client certificate — ‑E).
During testing it is often useful to set request HTTP headers (‑H); for example, to announce compression support:
ab -n 200 -c 4 -H "Accept-Encoding: gzip,deflate,br" https://localhost/
Overall, ab is useful for testing over HTTP/1.1 and TLS 1.2. At the same time, its load generation is limited to one thread, which is not enough for fast Angie scenarios that run several worker processes. Partly you can work around this limit by running several ab instances in parallel. For example, like this:
apt install parallel
parallel --ungroup -j 2 'ab -n 1000 -c 10 {}' ::: 'http://localhost/' 'http://localhost/'
The test results will then need extra processing. With that, let's move to a more advanced tool: wrk.
The wrk/wrkx utilities
The wrk project has a long history. It is a multithreaded utility for load testing and benchmarking web servers and applications. The project has not been developed for over five years now, but there are active forks with important improvements over the original. The wrk2 version adds load control (the ‑R flag, which sets RPS) and measures response time more accurately.
The variant we will use below is wrkx, a fork of wrk2 by Alexey Rybak. wrkx includes several patches for stability and the ability to bind threads to CPUs (CPU affinity).
To use wrkx you need to build it from source, which is easy enough. An example of the process for Ubuntu 24.04:
apt install git build-essential libssl-dev zlib1g-dev
git clone https://github.com/devhands-io/wrkx.git
cd wrkx
make
If everything went well, you get a wrk binary in the cloned project directory. When running it, be careful not to confuse the regular wrk installed in the system (from the package of the same name) with this built file.
A simple wrkx test can run like this (note the path to the file):
./wrk -t8 -c48 -d20s -R2000 --timeout 5s http://localhost/
The parameters of this command:
‑t8— run 8 threads to generate load;‑c48— number of parallel connections across all threads;‑d20s— test duration;‑R2000— total load of 2000 RPS (requests per second);‑timeout 5s— server response wait time;http://localhost/— request URL.
So the test defines a duration and a request rate cap. By default wrk opens keep-alive connections. Thanks to multithreading, wrk can generate high load. Like ab, wrk can set request headers:
./wrk -t2 -c48 -d30s -R2000 -H "Accept-Encoding: gzip" --timeout 5s http://localhost
Now the results include detailed response time distribution parameters (part of the output omitted):
Running 30s test @ http://localhost
2 threads and 48 connections
Thread calibration: mean lat.: 3844.563ms, rate sampling interval: 13623ms
Thread calibration: mean lat.: 3788.619ms, rate sampling interval: 13557ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 14.90s 3.98s 22.95s 56.98%
Req/Sec 244.50 0.50 245.00 100.00%
Latency Distribution (HdrHistogram - Recorded Latency)
50.000% 15.68s
75.000% 17.97s
90.000% 19.97s
99.000% 22.50s
99.900% 22.77s
99.990% 22.92s
99.999% 22.97s
100.000% 22.97s
Detailed Percentile spectrum:
Value Percentile TotalCount 1/(1-Percentile)
7155.711 0.000000 1 1.00
9789.439 0.100000 951 1.1
...
22970.367 0.999902 9421 10240.00
22970.367 1.000000 9421 inf
#[Mean = 14897.478, StdDeviation = 3975.822]
#[Max = 22953.984, Total count = 9421]
#[Buckets = 27, SubBuckets = 2048]
----------------------------------------------------------
14874 requests in 30.04s, 150.82MB read
Requests/sec: 495.20
Transfer/sec: 5.02MB
Detailed response time data can be used to visualize an HDR histogram.
The wrk utility works with HTTP/1.1; TLS capabilities depend on the library version used at build time. Beyond simple tests, wrk supports Lua scripts for more complex scenarios.
What if you need to test the server over HTTP/2? The h2load tool handles that.
The h2load utility
All the tools covered above could only work with HTTP/1.1. Yet today HTTP/2 is the most popular protocol for front-end servers. You can create a test using HTTP/2 with the h2load utility from the nghttp2 project. For Ubuntu 24.04 a package is available:
apt install nghttp2-client
Running the test is similar to the other utilities:
h2load -n 500000 -c 100 -t 4 https://localhost/
By default h2load uses HTTP/2. The parameters of this test in detail:
-n 500000— number of requests;-c 100— number of parallel connections;-t 4— number of threads;https://localhost/— request URI.
To use HTTP/1.1 you can pass the ‑h1 parameter. In experimental mode the utility supports HTTP/3, enabled with the ‑h3 flag.
Let's look at the test results:
starting benchmark...
spawning thread #0: 5 total client(s). 2500 total requests
spawning thread #1: 5 total client(s). 2500 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
...
progress: 100% done
finished in 8.88s, 563.37 req/s, 5.75MB/s
requests: 5000 total, 5000 started, 5000 done, 4997 succeeded, 3 failed, 0 errored, 0 timeout
status codes: 4997 2xx, 0 3xx, 0 4xx, 3 5xx
traffic: 51.07MB (53546209) total, 371.03KB (379934) headers (space savings 43.70%), 50.53MB (52985839) data
min max mean sd +/- sd
time for request: 895us 1.13s 17.13ms 58.48ms 95.32%
time for connect: 2.23ms 6.40ms 4.30ms 1.46ms 60.00%
time to 1st byte: 7.09ms 27.95ms 13.57ms 7.88ms 80.00%
req/s : 56.34 63.28 58.42 2.56 80.00%
The results include overall statistics on server response codes (status codes) and the observed RPS (req/s), throughput (MB/s), plus response time statistics (the final table). Among the extra data you can see header compression efficiency (space savings) in the traffic line.
The utility is especially useful for comparing HTTP/1.1 and HTTP/2 behavior, and it is highly efficient, on par with wrkx.
Summary
Each of the utilities reviewed has its own strengths and weaknesses, and each handles its own tasks well. The most universal tool is wrkx. For working with newer protocols, h2load is the recommendation. Out of scope for this article are more complex testing systems, for example k6 from Grafana, Apache JMeter, or Yandex.Tank. They can implement more complex test scenarios and come with a broad set of capabilities.