Contact Us

Home / Engineering Blog / DevOps

DevOpsEngineering notes

Using Angie as a Reverse HTTP Proxy

Historically, the reverse proxy role can be called the original purpose of Nginx, and therefore of Angie. In this article we look at why it is "reverse" and which proxying details matter when you configure it.

Video version

A video version of this article is available on YouTube.

What is a reverse proxy

The term proxy server comes from proxy (intermediary). A client sends requests to remote servers not directly but through a proxy server. Common uses of a proxy are caching internet resources, controlling traffic, and hiding the client's address. So a regular proxy has a limited circle of clients and an unlimited list of servers to request. A classic example of a proxy implementation on Linux is squid. A proxy server used to be a near-mandatory part of any office network, because it saved precious internet bandwidth and traffic.

A reverse proxy solves a different task: it accepts connections from an unlimited circle of clients and forwards requests to a strictly limited set of servers (usually called upstreams or backends). Various web servers can act as a reverse proxy: Angie, Nginx, Apache, and so on.

The tasks of a reverse proxy vary widely:

  • efficient handling of a large number of client connections;
  • providing a secure connection (TLS);
  • access restriction;
  • caching backend responses;
  • dynamic response compression;
  • load balancing;
  • request rate limiting;
  • routing requests to different servers.

Many web applications assume a reverse proxy server in their architecture to handle such tasks. Below we look at popular Angie configurations for the reverse proxy role.

Proxying over HTTP

HTTP(S) proxying is handled by the http_proxy module. With this module you can expose a web service through Angie. The main directive responsible for proxying a request is proxy_pass. A typical location configuration for proxying, though, includes several extra directives.

Consider an example of a dynamic location with HTTP proxying.

location / {
  proxy_pass http://localhost:8000;
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Real-IP $remote_addr;
}

This configuration is typical for servers where the frontend and the backend of the application run within the same system. Requests in the location are proxied to the TCP socket localhost:8000 over HTTP. As the backend server address you can use a UNIX socket or an upstream group name. If the proxy_pass parameter contains a URI in addition to the server address (remember that / is also a URI), various side effects appear, which we look at separately. In that case a separate connection to the backend is created, and it is closed as soon as the request has been processed. By default the response is buffered in memory or on disk (if the memory buffers run out).

Then come several proxy_set_header directives, which pass extra request data to the backend. The first directive sets the Host header so that the backend can split servers by domain. By default this header is filled with the address given in proxy_pass (a server or an upstream name).

The connection to the backend is made from Angie, so the real client's IP has to be passed separately. For that we pass the X-Forwarded-For header with a special value from the $proxy_add_x_forwarded_for variable, which holds the original request header X-Forwarded-For and the $remote_addr variable joined by a comma. Some backends key on the X-Real-IP header, so we pass the client's original IP in that header.

Also note that some backend response headers are hidden from the client: Date, Server, X-Pad and X-Accel-*. To pass them through, specify the proxy_pass_header directive with the corresponding header names.

Client request headers (or the request body) can also be blocked from reaching the backend entirely, with the proxy_pass_request_headers and proxy_pass_request_body directives respectively.

How the URI in location and proxy_pass interact

To fully understand how the proxy_pass directive works you need to look at how the request URI is passed to the backend. The URI that gets passed depends on whether the server address in proxy_pass contains a URI and, in some cases, on the prefix in location.

The simplest configuration is when proxy_pass contains only a server address, with no URI, for example:

proxy_pass http://127.0.0.1:8000;

In this variant the request URI is passed to the backend unchanged.

Any location with proxy_pass inside (and any other *_pass) has a quirk: if the prefix in location ends with a slash and the request has no slash, you get a 301 redirect to the URI with a trailing slash.

location /test/ {
    proxy_pass http://127.0.0.1:8000;
}

Here the request GET /test is redirected with code 301 to /test/. To avoid this behavior you can create a separate location = /test, or turn the redirect off with the auto_redirect off directive.

Behavior gets more complex if proxy_pass specifies a URI. For example:

location /test/ {
    proxy_pass http://127.0.0.1:8000/real/;
}

At a minimum, remember that the URI in location and the URI in proxy_pass interact with each other, and the result can be surprising. When in doubt, check the documentation.

Using keep-alive connections with the backend

The proxying examples in this article used local backend addresses. But a web application's infrastructure is often spread across several servers, and proxying goes to a remote machine. That is, there is a real network between Angie and the server, with non-zero latency (RTT).

Remember that by default Angie opens a connection to the proxied server for every request, so keep-alive support toward the backend is off. This mode causes no problems when proxying locally, but over a real network you have to account for the TCP handshake overhead (1 RTT) added to every request.

There is also the behavior of TCP connections, which only reach their working throughput after some time of data transfer (congestion control algorithms). The higher the network latency between Angie and the backend, the more these factors affect the resulting response time for the client. Keep-alive matters especially when proxying over HTTPS, because on top of TCP there is a TLS handshake (1–2 RTT).

Keep-alive support on the Angie side can be enabled with the following configuration, but of course the backend must support keep-alive too. Here you need an upstream block (even when using a single server) with the keepalive directive. By the way, the upstream block can serve other purposes as well — for example, limiting the number of connections to the backend; details are in the description of the server directive. Connection lifetime and request count are controlled by keepalive_timeout and keepalive_requests respectively. In the location itself, where proxying happens, you must set the Connection header and use HTTP/1.1.

upstream backend {
    server 127.0.0.1:8080;
    keepalive 100;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}
...
location / {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_pass  http://backend;
}

When data travels over a network, a secure connection may be required to rule out the risk of interception. In that case you can set up proxying over HTTPS (covered below). In this example we started with request headers when talking to the backend, but you can also affect response headers — we look at that next.

Working with response headers and cookies

Earlier in the configuration we used the proxy_set_header directive, which sets a request header from Angie to the backend.

To set a response header from Angie to the client, use the add_header directive.

When using these directives, remember the override rule at different levels. These directives are allowed to appear more than once in the same block, and inheritance works like this.

If a directive is set at a higher level and not set in the nested block, the value is inherited from the higher level. But if the directive is set both at the top level and at the nested level, the nested directive does not inherit and is used as written in its block.

Consider the following example.

http {
    add_header X-HTTP-TEST 1;
    server {
        listen 80;
        location / {
            return 200 "OK";
        }
    }
}

Here, when testing the server, we get the header inherited from the http context:

curl --head localhost | grep TEST

X-HTTP-TEST: 1

Now the next example:

http {
    add_header X-HTTP-TEST 1;
    server {
        listen 80;
   add_header X-SERVER-TEST 1;
        location / {
            return 200 "OK";
        }
    }
}

Testing shows that the directive does not stack with the http level; it is overridden at the server level:

curl --head localhost | grep TEST

X-SERVER-TEST: 1

With cookies, Angie lets you change their properties. For example, if the backend passes an incorrect domain=localhost, you can rewrite it with the proxy_cookie_domain directive:

proxy_cookie_domain localhost $host;

The directive supports regular expressions and variables. Flags can be added to cookies (for example httponly) via proxy_cookie_flags. Finally, the path attribute can be changed with proxy_cookie_path.

Handling backend errors and redirects

Unfortunately, the backend does not always answer with a successful 200. On an error the backend may respond with its own error page, which can be undesirable both from a security point of view and for the overall impression of the service. To intercept such errors and handle them in Angie itself (with the error_page directive) it is enough to enable proxy_intercept_errors.

proxy_intercept_errors on;

The second kind of special backend response is redirects. Sometimes the Location header value needs a change (a different host or path). Such changes happen by default based on the prefix in location and the URI in proxy_pass. The proxy_redirect directive controls them; its default behavior (the default value) is easiest to show with two configurations that behave identically. The first one uses the default value, the second explicitly configures the redirect address to change from http://upstream:port/two/ to http://upstream:port/one/.

location /one/ {
    proxy_pass     http://upstream:port/two/;
    proxy_redirect default;
}

location /one/ {
    proxy_pass     http://upstream:port/two/;
    proxy_redirect http://upstream:port/two/ /one/;
}

If the replacement string has no server address and port, the values from proxy_pass are used.

Proxying over HTTPS

Using the HTTPS protocol between Angie and the backend server improves security, but adds overhead to connection setup and traffic encryption. In practice, HTTPS inside an infrastructure is usually required because there is no trust in the network between Angie and the backend, or because of regulatory constraints. The goals when configuring it are the same as when configuring HTTPS for clients: minimize the number of full TLS handshakes and use efficient cipher algorithms. There are also differences from client connections: we fully control the software, so compatibility is not a concern. We can also skip certificate validation (use self-signed certificates) if the server does not need to be authenticated. Depending on those requirements, the configuration can look like this:

location / {
  proxy_pass https://192.168.0.2;
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_ssl_name $host;
  proxy_ssl_server_name on;
  proxy_ssl_verify       off;
  proxy_ssl_session_reuse on;
  proxy_ssl_protocols    TLSv1.3;
}

The configuration above uses keep-alive connections and TLS session resumption (proxy_ssl_session_reuse), passes the domain name (proxy_ssl_server_name and proxy_ssl_name), and disables certificate validity checks (proxy_ssl_verify). The protocol is pinned to TLS 1.3 because we fully control the software on both ends of the connection. If your backend has different capabilities, adjust the cipher and protocol settings to match.

One more note: Angie supports the HTTP/3 protocol for connections to the backend. To use it, just set the protocol version:

proxy_http_version 3;

Note that the protocol versions used for client connections and for connections to the proxied server in Angie are unrelated to each other. With that, let's move on to general proxying settings: buffering and timeouts.

Configuring buffering and timeouts

By default the response from the server is read into a buffer so that the backend is released as fast as possible and the response can then be sent to the client. This way the load on the backend is optimized: it is freed as soon as possible. Buffering parameters can be changed, but the most common setting in this area is proxy_buffers. It takes two parameters — the number of buffers and the size of one buffer. If the response size exceeds the total buffer size, Angie writes part of the response to disk, which can slow down the response to the client. Example of proxy_buffers:

proxy_buffers 32 4k;

In some situations response buffering has to be disabled. Angie then reads only a small part of the response into a buffer and reads the rest of the backend response synchronously. That can be done with the proxy_buffering directive:

proxy_buffering off;

If buffering is needed but writing temporary files to disk must be disabled, set the proxy_max_temp_file_size directive.

proxy_max_temp_file_size 0;

When working with the proxied server, Angie tracks timeouts at various stages of request processing (we look at the specific directives below). By default the timeout value is 60 seconds, which is enough for a working configuration in most cases without tuning them. Under high load or with load balancing, though, such values can cause a lot of problems.

The point is that under load the backend's response time grows, and high timeouts build up a queue of requests. Angie keeps waiting for the response, so the client gets no result either. Even if the backend recovers, the whole stream of pending requests is sent to it and the application stays slow for a while longer. With shorter timeouts Angie returns an error to the client sooner, problem requests do not accumulate, and the application recovers faster. With load balancing, the switch to the next server also happens sooner. Timeouts are configured with the directives:

proxy_connect_timeout 10s;
proxy_read_timeout 10s;
proxy_send_timeout 10s;

The first one waits for the connection to the server, the second is the timeout while reading the response from the server (the interval between two reads), the third is like the second but for sending the request to the server. When tuning read/send timeout you have to account for the backend's specifics, because not all servers finish processing a request when the connection is dropped from the Angie side. Also, for slow applications or specific locations these settings can be raised so that everything works correctly.

To reduce the load on the backend you can use server-side caching.

Keeping local copies of files

The proxying module can store (proxy_store) local copies of files received from backends. This lets you keep a copy of immutable static content on the frontend.

Say the source files for the /js path live on a remote backend, but you want to be able to serve them from the local disk. That behavior can be configured like this.

location /js/ {
    root               /var/www;
    error_page         404 = @fetch;
}

location @fetch {
    internal;

    proxy_pass         http://192.168.0.2;
    proxy_store        on;
    proxy_store_access user:rw group:rw all:r;
    proxy_temp_path    /var/temp;

    root               /var/www;
}

This configuration works by handling the 404 error. If a file is not found in the local directory, it is passed to the internal location @fetch, which sends requests to the backend (proxy_pass) and at the same time saves the result into the local directory rooted at /var/www.

Summary

This article looked at various aspects of Angie in reverse proxy mode, including the details of tuning optimal interaction between the frontend server and the backend. Add load balancing and caching to that and you get a full picture of Angie's proxying capabilities.

← Back to all engineering notes