Contact Us

Home / Engineering Blog / Networking

NetworkingEngineering notes

L4 Traffic Balancing and Proxying in Angie

Angie's capabilities as a web server and reverse proxy are well known. But besides working at level L7 (HTTP), we can also apply it at L4 (TCP and UDP) as a load balancer and proxy server. That is exactly the functionality this article covers.

Video version

A video version of this article is available on YouTube.

How the HTTP and stream modules differ

I will assume you have used Nginx or Angie as an HTTP server, because we will build on the server configuration in the HTTP module context. All directives for working at the HTTP level are defined inside the http block. For L4 there is a separate module and its own configuration block: stream.

The key difference between the HTTP and stream modules is that they operate at different layers of the OSI (Open Systems Interconnection) model. The HTTP module belongs to L7 — the application layer — and implements the HTTP protocol. When the http module works, the server deals with individual HTTP requests and forms a response according to the configuration.

The stream module works at L4 — the transport layer — and proxies TCP and UDP protocols. For the stream module there is no notion of a "request" in the general case. You can think of the stream module as not attempting to recognize protocols above L4; the content of the traffic stays a black box. The exception is modules with limited parsing of application-layer protocols (MQTT Preread, SSL Preread, RDP Preread). That is why in the stream module the term "connection" is normally used rather than "request".

In terms of functionality, the limitations of the stream module are defined by working at the L4 level; otherwise the configuration and behavior are similar to the corresponding modules of the HTTP context.

Do not confuse same-named modules and variables from the http context and the stream module. They may share directive names while differing in the available parameters and the ways they can be applied. In the Angie documentation the modules are split by context: "HTTP module" and "Stream module". The alphabetical indexes of directives and variables also indicate the context.

Proxying TCP and UDP applications

Let's start exploring the stream module with a simple task: proxy a TCP application (MySQL) through Angie in order to enforce a connection limit and an access list based on the client IP address.

The first thing to do is define a server to bind to the TCP socket in the stream context:

stream {
  server {
      listen 6306;
  }
}

In this example we created a server listening on TCP port 6306. The listen directive supports a wide list of additional parameters. For example, you can enable socket creation for each worker process (reuseport) or specify SSL mode (ssl). The ways to specify a socket are the same as in HTTP mode (IP with port, port only, port range, separate sockets for IPv4 and IPv6, UNIX sockets).

The second thing to specify is which server the traffic should be forwarded to. That is the job of the proxy module and its proxy_pass directive:

server {
  listen 6306;
  proxy_pass 192.168.1.2:3306;
}

Note that no scheme (http(s)://) is given before the server address.

The minimal proxying configuration is now in place, and we need to implement the access control requirement and the per-IP connection limit:

stream {
  limit_conn_zone $binary_remote_addr zone=addr:10m;

  server {
    listen 6306;
    proxy_pass 192.168.1.2:3306;

    limit_conn addr 1;

    allow 192.168.1.0/24;
    deny  all;
  }
}

All requirements are covered: we limit connections to one per client IP address and allow access only from the 192.168.1.0/24 subnet. Unlike the HTTP module, stream does not write an access.log by default. This can be fixed by defining a log format and enabling it in the stream context:

stream {
    log_format basic '$remote_addr [$time_local] '
                 '$protocol $status $bytes_sent $bytes_received '
                 '$session_time ' "$upstream_addr";
    access_log /var/log/angie/stream-access.log basic;
}

Entries of the following kind will now appear in stream-access.log:

192.168.122.1 [08/Aug/2025:17:18:57 +0300] TCP 200 180902 3535 301.392 127.0.0.1:82

Such records appear after the session with the client ends and show overall traffic statistics (recall that there are no requests here). For convenience, artificial session status codes are used for $status, with values chosen to parallel HTTP response codes: 200, 400, 403, 500, 502, 503.

Proxying UDP services is broadly similar to working with TCP, but has its own specifics. For example, tracking UDP sessions requires the reuseport parameter. You can also explicitly specify how many responses from the server are expected before a UDP session is considered complete. Consider the following example:

server {
    listen 553;
    listen 553 udp reuseport;
    proxy_pass 192.168.1.1:53;
    proxy_responses 1;
}

In this configuration Angie accepts both TCP and UDP traffic on port 553 and proxies it to the server 192.168.1.1:53. It explicitly states that exactly one response from the server is expected when the request goes over UDP. It is also worth mentioning that traffic is not converted from TCP to UDP or back. Proxying always uses the protocol the client connected to Angie with.

Working with TLS

At L4 the full range of Angie's TLS capabilities is available. Let's take a typical task: wrap an application's plaintext traffic in TLS (implement TLS termination) so it can be transmitted securely over public networks. This can be done as follows:

stream {
  server {
    listen 443 ssl;
    ssl_certificate /etc/angie/ssl/1.crt;
    ssl_certificate_key /etc/angie/ssl/1.key;
    proxy_pass 192.168.1.2:80;
  }
}

As in the HTTP module, it is enough to enable TLS on the listen directive with the ssl parameter and specify a certificate with a key. The already decrypted traffic is then proxied to the server.

The example above did not use domain-based server selection (there is no server_name directive). Suppose the task is to distribute traffic on port 443 across different applications: some domains will be served by Angie itself in the HTTP module, while others will be proxied to other servers at L4. The task can be solved — it is enough to use the server_name and pass directives:

http {
    server {
        listen 8000;
        location / {
            root html;
        }
    }
}

stream {
    server {
        listen 443 ssl;
        server_name test.ru;
        ssl_certificate     /etc/angie/ssl/1.crt;
        ssl_certificate_key /etc/angie/ssl/1.key;
        pass 127.0.0.1:8000;
    }
    server {
        listen 443 ssl;
        server_name serv.test.ru;
        ssl_certificate     /etc/angie/ssl/1.crt;
        ssl_certificate_key /etc/angie/ssl/1.key;
        proxy_pass 192.168.1.2:9000;
    }
}

In the example above Angie opens a 443 TCP socket to accept connections and terminate TLS. The server name is determined from the SNI extension: for the domain test.ru the pass directive takes effect. That directive hands the client connection to another module of the server (HTTP in our case), and request processing happens in the HTTP module on the server with socket 8000. For the domain serv.test.ru TLS termination and L4 proxying to socket 192.168.1.2:9000 take place.

But what if termination at Angie is not needed, yet we still want to distribute traffic by domain name? That is what the ssl_preread module is for. It implements the pre-read phase of the request and can obtain the TLS version, the domain name, and the list of protocols over ALPN (for example, h2 or h3). These values are written into the corresponding variables: $ssl_preread_protocol, $ssl_preread_server_name and $ssl_preread_alpn_protocols. On top of these variables you can build flexible traffic routing. Consider the example:

map $ssl_preread_protocol $upstream {
    ""        192.168.1.2:22;
    "TLSv1.2" 192.168.1.2:443;
    default   192.168.1.2:443;
}

server {
    listen 8443;
    ssl_preread on;
    proxy_pass $upstream;
}

There is no TLS termination here (traffic is passed through as-is), but the ssl_preread mode is enabled. The map block creates a variable $upstream that depends on $ssl_preread_protocol. If the protocol version has an empty value, traffic is proxied to the SSH server (192.168.1.2:22); if it is TLSv1.2 or anything else, traffic is proxied to 192.168.1.2:443. The same scheme can be built on the server name or on the protocols in the ALPN extension.

TLS re-encryption is of course also possible — that is, talking to the proxied server over TLS. The setup is similar to proxying with the HTTP module:

server {
    listen 443 ssl;
    ssl_certificate /etc/angie/ssl/1.crt;
    ssl_certificate_key /etc/angie/ssl/1.key;
    proxy_ssl on;
    proxy_ssl_verify on;
    proxy_ssl_name test.ru;
    proxy_ssl_server_name on;
    proxy_ssl_trusted_certificate /etc/angie/ssl/trust.crt;
    proxy_pass 192.168.1.2:443;
}

The example shows proxying to the server 192.168.1.2:443 over TLS. The certificate of the proxied server is verified (proxy_ssl_verify) with respect to the name test.ru (proxy_ssl_name) and the trusted certificate (proxy_ssl_trusted_certificate).

Finally, we get to the balancing features for TCP and UDP applications.

Balancing at the transport level

If you are familiar with balancing HTTP applications in Angie, moving to L4 will be straightforward. All balancing algorithms and modes are supported except ip_hash and the sticky cookie mode.

The list of available features (the PRO mark means the feature exists only in the commercial Angie PRO version):

  • Round Robin algorithm;
  • Least Connection algorithm;
  • Hash algorithm;
  • Random Least Connection algorithm;
  • passive server checks (max_fails);
  • the slow_start, resolve options for servers;
  • Sticky Route mode;
  • Least Time (random) algorithm (PRO);
  • Feedback algorithm (PRO);
  • Sticky Learn and Sticky Learn Remote action modes (PRO);
  • active server checks (PRO);
  • the state, drain, backup_switch options for servers (PRO).

To set up balancing, as usual, define an upstream block describing a group of servers. We can then proxy to that group by referring to it by name:

stream {
  upstream backend {
    server 192.168.0.1:8000;
    server 192.168.0.2:8000;
  }
  server {
    listen 9090;
    proxy_pass backend;
  }
}

This is the simplest configuration for balancing traffic between two servers. Round Robin is used by default and the weights are equal.

To select the desired balancing algorithm, use the directives of the upstream module: hash, random, least_conn, feedback (PRO), least_time (PRO). Also keep in mind the parameters of the server directive. Suppose we need least-connections balancing with session affinity by domain and a limit on the number of connections to each server. The resulting configuration is this:

map $ssl_preread_server_name $route {
  test.ru            a;
  test2.ru           b;
  default           "";
}

upstream backend {
  least_conn;
  server 192.168.0.1:443 sid=a max_fails=2 fail_timeout=10s max_conns=100;
  server 192.168.0.2:443 sid=b max_fails=2 fail_timeout=10s max_conns=100;
  sticky route $route;
}

server {
  listen 443;
  ssl_preread on;
  proxy_pass backend;
}

Note that in the configuration above the servers have the max_fails and fail_timeout parameters set, which are responsible for passive checks. In the context of L4 balancing max_fails controls the number of connection attempts to the server (not requests, as in the HTTP module), while fail_timeout as usual defines the time within which those attempts have to be made for the server to be considered down (until another fail_timeout period has passed).

One more difference from balancing HTTP services is worth pointing out here. The failover setting (proxy_next_upstream) has a boolean type: on or off. When it is on (the default), the switch to the next server happens if connecting to the selected one is impossible.

On the other hand, there are features that the HTTP module does not have: a limit on the rate of reading the response from the server (proxy_download_rate) and a limit on the rate of reading data from the client (proxy_upload_rate).

Summary

We reviewed the main scenarios for using Angie as a TCP and UDP proxy and load balancer, and identified the differences between the stream module and the HTTP module. It is fair to say that Angie's L4 capabilities have nearly caught up with those of the HTTP module, except for the natural limitations of working at a lower level.

← Back to all engineering notes