Contact Us

Home / Engineering Blog / Performance

PerformanceEngineering notes

Tuning Angie for High Load

Server performance and tuning have come up more than once in other articles in this series. This article collects the main techniques and approaches for tuning Angie under high load, and points to dedicated articles for the details of each topic.

Video version

A video version of this article is available on YouTube.

General approach

Tuning a server for high load requires an individual approach. The load pattern, optimization goals, hardware and application specifics — all of these shape the decisions. For example, recommendations for cutting latency and recommendations for raising throughput can contradict each other. This article surveys the options for tuning a system and a server for different use cases. With that in mind, do not copy settings blindly: understand the context and verify on a test bench.

We will assume the operating system is the standard choice today — Linux. Let's start with system-level settings.

Linux system tuning

On the operating system side, what matters under high load is the network stack. The settings covered here belong to the kernel, which handles traffic up to layer 4 of the OSI model. This is true for a default configuration, without moving the network stack to user space (for example, with DPDK).

View the current values with:

sysctl -a

Settings can be changed interactively (for testing) or through a configuration file (permanently). The recommended approach is a separate file for custom settings in /etc/sysctl.d/:

nano /etc/sysctl.d/99-custom.conf

Apply settings from configuration files with the sysctl command:

sysctl --system

Temporary changes are also made through sysctl:

sysctl -w net.ipv4.tcp_slow_start_after_idle=0

Now the settings that can help:

  • net.ipv4.tcp_slow_start_after_idle = 0 — keeps the slow-start state after idle periods;
  • net.ipv4.tcp_notsent_lowat = 65536 — TCP send buffer size;
  • net.ipv4.ip_local_port_range = 1024 65535 — local port range (limits the number of outbound connections to upstreams);
  • net.ipv4.tcp_tw_reuse = 1 — reuse of sockets in the TIME_WAIT state;
  • net.netfilter.nf_conntrack_max = 1048576 — conntrack table size;
  • net.core.netdev_max_backlog= 10000 — frame queue for processing on the network interface;
  • net.core.somaxconn = 16000 — maximum size of the accept queue for connections on a socket;
  • net.ipv4.tcp_abort_on_overflow = 1 — fast reset of connections when the accept queue overflows;
  • net.ipv4.tcp_max_syn_backlog = 1024 — parameter affecting the maximum number of connections in the SYN queue;
  • net.core.rmem_max = 8388608 — maximum TCP receive buffer size;
  • net.core.wmem_max = 8388608 — same, but for sending;
  • net.ipv4.tcp_rmem = 4096 131072 8388608 — memory autotuning for receiving data on a TCP socket (minimum, default, maximum);
  • net.ipv4.tcp_wmem = 4096 16384 8388608 — same, but for sending.

The parameters are described in detail in the documentation. You can also look into the various TCP congestion control algorithms, since modern kernels offer several options. The main subject of this article is tuning the Angie server, so let's move on to its configuration.

General settings

The server's main work is done by worker processes, so their count matters. By default, the number of worker_processes equals the number of processors in the system:

worker_processes  auto;

For most applications this setting is optimal. On large NUMA systems it can pay to give the server a single NUMA node, which means limiting the process count. Processes can also be pinned to individual processors; automatic distribution looks like this:

worker_cpu_affinity auto;

Pinning to CPUs cuts context-switching overhead. But pinning can hurt the server when other applications on the system compete for the CPU.

The next setting caps the number of open file descriptors (worker_rlimit_nofile) per worker process:

worker_rlimit_nofile 30000;

This setting also caps the number of connections a worker process can handle. The connection limit itself is set by the worker_connections directive:

events {
  worker_connections  8192;
}

Keep in mind that with HTTP/2 and HTTP/3 a single connection carries multiple parallel requests, controlled by the http2_max_concurrent_streams and http3_max_concurrent_streams directives respectively.

If the PCRE library supports JIT compilation, you can enable it:

pcre_jit on;

That covers the general server settings; next, the network parameters.

Network parameters

The server exposes various TCP socket options to cut latency or improve throughput. For example, the tcp_nodelay directive sends data as early as possible (enabled by default):

http {
  tcp_nodelay on;
}

When using the sendfile mechanism, it pays to enable the tcp_nopush option, which lets the file be sent in full packets:

http {
  tcp_nopush on;
}

The sendfile directive enables the mechanism of the same name and cuts load when sending files.

http {
  sendfile on;
}

However, the sendfile mechanism works only with plain HTTP, or with Kernel TLS enabled (a kernel module that implements TLS).

In the general case (without sendfile), serving static files depends heavily on the output_buffers directive, which sets the number and size of buffers used to read files from disk. Two buffers are enough; increase the size so data is read from disk in larger chunks.

http {
  output_buffers 2 128k;
}

Next: use an efficient method of distributing connections across worker processes — reuseport, enabled as a socket option in the listen directive, for example:

server {
  listen 80 default_server reuseport;
}

The reuseport parameter can be specified only once per socket in the server configuration. The listen directive also has many options that fine-tune sockets. For example, note these options:

  • fastopen=32 — enables TCP Fast Open; the number is the maximum number of connections that have not completed the handshake — check that this is safe for your application;
  • backlog=1024 — length of the accept queue; helps ride out spikes of new connections. This setting is tightly coupled with the kernel parameters net.core.somaxconn, net.ipv4.tcp_max_syn_backlog and net.ipv4.tcp_abort_on_overflow; details are covered here and here;
  • deferred — enables TCP_DEFER_ACCEPT for the socket on Linux, which delays (by up to 1 second) connection processing until the client sends data, cutting context switches while waiting for data in the socket buffer;
  • rcvbuf=128k, sndbuf=128k — socket buffer sizes for receiving and sending data;
  • so_keepalive=15:5:3 — enables SO_KEEPALIVE (TCP keep-alive), not to be confused with the HTTP keep-alive mechanism. Optional parameters are also given (they override the system keep-alive settings): keepidle:keepintvl:keepcnt (idle period, probe send interval in seconds, and probe count).

For HTTP/3 specifically, Angie can route QUIC packets with eBPF (requires Linux 5.7 or newer). The developers describe the technology in an article. QUIC can also use a UDP send optimization: quic_gso (GSO — generic segmentation offload), which lowers CPU load. The configuration looks like this:

events {}

quic_bpf on;

http {
  quic_gso on;
  ...
}

For the details of configuring and using HTTP/2 and HTTP/3, see the earlier article in this series.

Under load the server usually holds many connections, some of which linger waiting for final teardown. The reset_timedout_connection directive speeds up dropping such connections:

http {
  reset_timedout_connection on;
}

The most important optimization for HTTP/HTTPS is keep-alive connections. Here we mean client connections to the server. Keep-alive to proxied servers is also useful and is covered below. It greatly reduces TCP and TLS handshakes and uses the link's bandwidth more efficiently. A keep-alive connection stays up until the timeout (keepalive_timeout) expires or the request limit (keepalive_requests) is reached. Example:

http {
  keepalive_timeout  120;
  keepalive_requests  2000;
}

Also look at timeout settings for client interactions. The limits for receiving a request (client_body_timeout, client_header_timeout) and for sending (send_timeout) are configured separately. By default they are 60 seconds, which can be excessive today. When choosing values, account for slow clients.

http {
  send_timeout      10;
  client_body_timeout   10;
  client_header_timeout  10;
}

Next, the HTTPS settings.

TLS configuration

The TLS load splits into two parts: the handshake (asymmetric phase) and the main data transfer (symmetric phase).

When tuning TLS, start with session caching. TLS 1.2 and TLS 1.3 use different mechanisms: ssl_session_cache (1.2) and ssl_session_tickets (1.3). Also set a reasonably long session lifetime:

http {
  ssl_session_cache    shared:SSL:10m;
  ssl_session_tickets   on;
  ssl_session_timeout  28h;
}

Usage of the shared memory zone (ssl_session_cache) can be monitored in Angie Console Light or through the API.

The handshake phase can also use different certificate types: RSA and ECDSA. The certificate type does matter, but the effect depends on many factors: TLS version, library build, and so on. Test results are published in an earlier article in this series.

For the symmetric data transfer phase, the choice of cipher matters. In practice most clients will use TLS 1.3, where the set of allowed ciphers is tightly restricted. The symmetric phase will usually run AES, which gives maximum throughput. Narrowing the cipher list would likely conflict with the range of supported clients, so it is rarely practical. That earlier article also compares symmetric cipher options.

TLS performance also depends on buffer size (the ssl_buffer_size directive). By default the directive has the maximum allowed value (16 KB), which minimizes overhead when transferring large volumes of traffic. Lower it only to minimize the time to receive and decrypt the first part of a document (for example, to start parsing HTML); in most configurations the maximum buffer size is optimal:

ssl_buffer_size 16k;

We covered the sendfile mechanism for optimizing file sends earlier. With HTTPS, sendfile requires TLS to run in the kernel. If your system supports Kernel TLS, you can get sendfile benefits over HTTPS:

http {
  ssl_conf_command Options KTLS;
}

However, sendfile works efficiently with HTTPS only over HTTP/1.1. Combining sendfile and kTLS with HTTP/2 will cause a performance drop.

Angie has many file-handling optimizations worth reviewing.

File handling

In operation, Angie actively works with files of all kinds: serving static files, writing logs, temporary buffering files, certificate and key files, the on-disk response cache.

First, to cut file-open syscalls, use the open file cache (open_file_cache):

http {
  open_file_cache          max=10000 inactive=10s;
  open_file_cache_valid    30s;
  open_file_cache_errors   on;
  open_file_cache_min_uses 2;
}

This configuration declares a cache of 10,000 file descriptors with a 30-second validity period. A file descriptor enters the cache after two uses. Lookup errors are also cached, which can produce 404 responses even when the file exists on disk (if the missing file was cached). Problems can also arise when files are rewritten non-atomically — the server will then serve corrupted files.

If log file names are set through variables, you can cache them as well:

http {
  open_log_file_cache   max=100 inactive=60s min_uses=2;
}

There is a similar cache for TLS certificates whose names come from variables (ssl_certificate_cache):

http {
  ssl_certificate_cache max=1000 inactive=20s valid=1m;
}

Local log file writes can also be optimized with buffering and compression:

access_log /var/log/angie/access.log main_ext buffer=16k gzip=3 flush=10s;

This example buffers 16 KB at a time and writes the log compressed at level 3. That shifts load off the disk subsystem at the cost of extra CPU. If the server is short on CPU, use buffering without compression. Another way to reduce disk load is shipping logs over the network with syslog.

Angie often proxies requests to other servers, so let's look at optimizations for that interaction.

Working with proxied servers

This section assumes HTTP proxying (the Proxy module); other protocols usually have similar directives and settings.

First, set timeouts for backends. Exact values depend on the application, but shorter settings let you detect a dead server faster and route the request elsewhere. Example configuration:

location / {
  proxy_connect_timeout      5;
  proxy_send_timeout         10;
  proxy_read_timeout         10;
}

When tuning these timeouts — sending a request (proxy_send_timeout) and receiving a response (proxy_read_timeout) — study how your servers behave. Dropping the connection from the Angie side may not stop the backend from finishing its work.

Next: response buffering (enabled by default, the proxy_buffering directive). This mechanism frees the server as quickly as possible without waiting for the response to reach the client. Memory buffers have a limit (proxy_buffers); once exceeded, the response is written to disk and served from a temporary file. To avoid frequent disk buffering, raise the buffer size so most responses fit. Example configuration:

location / {
  proxy_temp_file_write_size 64k;
  proxy_buffer_size          4k;
  proxy_buffers              64 4k;
  proxy_busy_buffers_size    32k;
}

The total buffer size here is 64×4 KB, the initial header buffer (proxy_buffer_size) is 4 KB, and writing to the temporary file (proxy_temp_file_write_size) happens in 64 KB blocks.

Backend communication usually goes over the network, so — as with client connections — keep-alive makes it more efficient. Consider this example:

upstream http_backend {
    server 192.168.0.100:8080;
    keepalive 16;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}

server {
    location /http/ {
        proxy_pass http://http_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

This uses HTTP/1.1 and an upstream block to configure persistent connections to the server (the keepalive directive). When enabling persistent connections to servers, make sure they can hold enough simultaneous connections. In the configuration above, each worker process keeps 16 persistent connections; multiply by the process count for the total. It is also possible for one Angie worker to take all available backend connections, leaving the others unable to connect.

To reduce load on proxied servers, you can use server caching — covered next.

Caching

A proxy server has two kinds of caching: server-side and client-side.

Start with server-side caching. Angie can save a server response to a disk file and serve it instead of sending a request upstream. Such a cache cuts load on the server sharply, but it is based on element time-to-live (TTL) and offers no other invalidation mechanisms. Server caching is covered in detail in a dedicated article in this series.

Here is a configuration example:

http {
  proxy_cache_valid 1m;
  proxy_cache_key $scheme$host$request_uri;
  proxy_cache_path /cache levels=1:2 keys_zone=one:10m:file=/etc/angie/cache.state inactive=48h max_size=800m;

  server {

    location / {
      proxy_cache one;
      proxy_cache_valid 200 1h;
      proxy_cache_lock on;
      proxy_cache_min_uses 2;
      proxy_ignore_headers "Cache-Control" "Expires";
      proxy_cache_use_stale updating error timeout invalid_header http_500 http_502 http_504;
      proxy_cache_background_update on;
    }
}

This configuration creates one cache zone (proxy_cache_path), saves state to a file (file=/etc/angie/cache.state), and allows serving a stale cache element in some cases (proxy_cache_use_stale). The cache lifetime for requests is set to one minute, though even a short TTL (microcaching, 1–5 seconds) can reduce load without seriously affecting site logic. Cache locking (proxy_cache_lock) ensures only one request goes out to refresh a cache element, but it can add a 500 ms delay for requests waiting for a response to appear in the cache; you can also set a maximum lock time with proxy_cache_lock_timeout. Targeted cache element removal is available through the third-party Cache Purge module (details in the article on server caching in this series).

Second: client-side caching can also cut server load by reducing traffic volume and request count. The browser owns the client cache; the server's job is to set correct response headers. This topic is covered in detail in the article on client caching in this series. Often a single header is enough to allow "permanent" caching of static files. For example:

location /static/ {
  add_header Cache-Control "max-age=31536000, public, no-transform, immutable";
}

With this header, the client stores the response and stops sending repeat requests. To invalidate such a cache, change the request URI.

Text resources are almost always compressed in transit — let's look at that in more detail.

Compression of text resources

In a typical web application, most requests return text responses. Text compresses extremely well, which cuts traffic by roughly 3× or more.

Compression in Angie is available in dynamic and static modes. Static compression clearly saves resources, while dynamic compression costs CPU.

By default Angie supports only Gzip compression; third-party modules add Brotli and Zstandard. Start with a Gzip configuration example:

http {
  gzip  on;
  gzip_static on;
  gzip_types text/plain text/css text/xml application/javascript application/json image/svg+xml application/font-ttf;
  gzip_comp_level 3;
  gzip_proxied    any;
  gzip_min_length 1000;
  gzip_vary on;
}

This example uses both compression modes: dynamic (gzip) and static (gzip_static). It also lists the MIME types to compress (gzip_types), the minimum response size to compress (gzip_min_length), and enables the Vary header for compressed responses (gzip_vary).

The key performance setting is the compression level (gzip_comp_level). For most cases, low values (1 to 3) give the best trade-off between response size and CPU load. Do not set high values (7–9) — load rises sharply while compression improves only a few percent.

With the third-party Brotli module the configuration is similar:

load_module modules/ngx_http_brotli_static_module.so;
load_module modules/ngx_http_brotli_filter_module.so;

http {
  brotli_static    on;
  brotli        on;
  brotli_comp_level 3;
  brotli_types  text/plain text/xml text/css application/javascript application/json image/x-icon image/svg+xml;
}

For static mode one more optimization applies: use the directive (gzip_static or brotli_static) only in locations that have statically compressed files. For example:

location /js/ {
  brotli_static    on;
  gzip_static on;
}

Compression tuning is covered in detail in a dedicated article in this series; a separate article compares popular text compression formats by the performance they deliver.

Summary

This article surveyed the main aspects of tuning Angie configuration for high load. You can now run your own audit of your server settings and find new ways to improve performance. Detailed breakdowns of individual directives and edge cases live in dedicated articles in this series.

← Back to all engineering notes