Text Compression in Angie: Static, Dynamic and Performance

Compressing text resources is a basic step in optimizing web application load times. There is a lot of text on the web: HTML, CSS, JS, JSON, SVG and more. The topic looks simple: enable gzip, add the content types to compress, and you are done. As always, the interesting part is in the details. Here is the full walkthrough.

A video version of this article is available on YouTube.

Compression negotiation mechanism

The process of negotiating a compression variant for the response is based on HTTP headers. In its request the client can send an Accept-Encoding header containing a list of options, for example:

Accept-Encoding: gzip, deflate, br, zstd

The server can then use any of the listed compression types and adds the corresponding response header:

Content-Encoding: gzip

Today gzip is the classic compressor for Angie, available in the standard build. But there are two more modern contenders that deserve a close look: brotli and zstd. By the way, the set of compression algorithms supported by browsers depends on the access protocol (HTTP or HTTPS). Over plain HTTP browsers advertise support for gzip and deflate, while the more advanced zstd and brotli are available only over HTTPS.

By default, when the Accept-Encoding header is absent, the server must respond with an uncompressed document. Keep this in mind when testing compression methods: remember to add the request header, for example with the ab utility:

ab -n 1000 -c 4 -H "Accept-Encoding: gzip" -k http://127.1/

When caching proxy servers are involved, compression can affect client compatibility. This is why compressed responses often carry a Vary header, which indicates that the response differs depending on request headers. For example:

Vary: Accept-Encoding

This example shows a response that depends on the Accept-Encoding request header. The header lets a cache serve a compressed response only to a client that is able to read it.

Dynamic and static compression

In principle, compression can be implemented in two ways: dynamic and static.

Dynamic compression is compression on the fly — performed directly before the response is sent to the client. In web server terms the compression happens in a filter, which is why the corresponding modules have "filter" in their names. In this variant, CPU resources are spent compressing every response. If the response itself is dynamic — a document from the backend, say — then it can only be compressed dynamically. Note that if the backend responds with an already compressed document, no second compression happens: the response is passed through unchanged. This makes it possible to shift the load around: if the backend compresses responses, the load stays on it; if it serves uncompressed responses, the frontend handles the compression. Dynamic compression puts higher demands on algorithm speed, so fast compressors with low compression level settings are used.

Static compression is creating a compressed version of the file on disk to serve to the user. With this method CPU resources are used only once — when the compressed copy is created. The web server itself takes no part in the compression: the compressed file is built beforehand with a utility. Static compression not only saves CPU resources but also allows the maximum compression ratio, since there is no need to save CPU during compression. In other words, you can use relatively slow algorithms with maximum compression settings.

Both methods and their use in different cases are covered below. Now for the compression algorithms available for web applications. Of course, the only compression options of interest are the ones supported by browsers and by the Angie web server.

The gzip compressor

The gzip compression algorithm, along with deflate, has the widest possible support among browsers and web servers. This compressor compresses text documents quite efficiently and is suitable for both the dynamic and the static variant. It is based on the Huffman algorithm and LZ77 coding. Support for the gzip compression format can therefore be considered a baseline and a good practice. However, Angie has to be configured correctly to get the right balance between CPU cost and compression level. Here is a typical gzip configuration:

http {
  gzip  on;
  gzip_types text/plain text/css text/xml application/javascript application/json image/svg+xml application/font-ttf;
  gzip_comp_level 3;
  gzip_proxied    any;
  gzip_min_length 1000;
  gzip_vary on;

  server {
    location ~* \.(?:js|css|svg|txt|ttf|xml)$ {
        gzip_static on;
    }
  }
}

Both compression modes are enabled here: dynamic (gzip) and static (gzip_static). The on value in the gzip_static directive means looking for compressed versions of files on disk (with the .gz suffix). If a compressed file is found, it is used as the response. It also matters to set the file modification date to match the uncompressed version, for better caching. Note that although gzip_static can be specified at the http level, it is set at the location level here, to avoid compressed-file lookups on every static request (for JPEG images, for example).

Next come the MIME types of the content that should be compressed. Note that type names can differ across systems, so check the mime.types file of the active Angie configuration. The key parameter for dynamic compression is the compression level (gzip_comp_level). Its value ranges from 1 to 9. To keep CPU load minimal, the default (1) can be left as is. If CPU resources are plentiful and the number of dynamically compressed requests is small, the level can be raised to 3–5; beyond that, compression costs grow quickly while the compression ratio changes little.

Compression of proxied requests is also allowed (gzip_proxied), a minimum response size for compression is set (gzip_min_length), and the Vary response header is added.

Besides the directives above, gzip can be disabled for specific clients by regular expression (gzip_disable), and the compression buffer can be tuned (gzip_buffers). In general, the compression configuration is best placed in the http context so it applies server-wide, though finer configuration at the server, location and if in location levels is allowed.

Compressed files for the static mode can be prepared with the standard gzip utility or with zopfli, a utility optimized for compression ratio. Using zopfli saves about 10-20% of the compressed file size. Example commands that keep the original file:

gzip -k9 'path/fname'
zopfli --gzip -i15 'path/fname'; touch -r 'path/fname' 'path/fname.gz'
zstd -f9 --format=gzip 'path/fname'

gzip compression gives maximum compatibility with different clients and is therefore practically mandatory for a standard Angie configuration. But there are more advanced compression formats built specifically for the web, brotli for example.

The brotli compressor

The brotli format is developed by Google; its main goal is the maximum compression ratio for text web content while keeping decompression fast. Compression speed can be much lower than that of standard gzip, though at low settings the two are comparable. The compression efficiency comes from a dictionary and other improvements to the DEFLATE algorithm.

To use brotli in Angie, third-party modules have to be installed, which is not a problem (they are in the Angie repository):

sudo apt install angie-module-brotli
sudo dnf install angie-module-brotli

The package contains two modules for static and dynamic compression. To use them, add the dynamic module loading directives to the main section (the main context) of the configuration file (at the top of angie.conf):

load_module modules/ngx_http_brotli_static_module.so;
load_module modules/ngx_http_brotli_filter_module.so;

Overall the compression configuration resembles the set of gzip module directives, with small differences. An example working configuration for dynamic and static compression:

http {
  brotli        on;
  brotli_comp_level 3;
  brotli_types  text/plain text/xml text/css application/javascript application/json image/x-icon image/svg+xml;

  server {
    location ~* \.(?:js|css|svg|txt|ttf|xml)$ {
        brotli_static    on;
    }
  }
}

The main choice to make is the compression level. In brotli the range is 1 to 11. The same values as gzip give a higher compression ratio and generally a lower compression speed. A speed comparison is presented at the end of the article. It also has to be taken into account that brotli is especially effective (in compression ratio) on large files (over 60 KB).

Two more directives are available for tuning: the minimum object size for compression (brotli_min_length) and the window size (brotli_window).

Statically compressed files are prepared with the standard brotli utility, which can be installed from a package on Ubuntu:

sudo apt install brotli
brotli -fZ 'path/fname'

A reminder here: browser support for brotli works only over HTTPS.

As a short interim summary, brotli is the best option for the static mode and quite good for the dynamic mode when the compression level is chosen carefully. But the list of compression options is not exhausted, so on to the Zstandard (zstd) algorithm.

The zstd compressor

The third compression option available on the web today is the Zstandard algorithm, implemented in the zstd utility. The algorithm is based on LZ77 combined with efficient entropy coding. For simplicity it is called zstd below. The goal of its development was high performance at a compression ratio comparable to the deflate algorithm. These qualities make zstd an excellent candidate for the dynamic compressor role.

zstd support in Angie comes from installing third-party modules from the repository:

sudo apt install angie-module-zstd
sudo dnf install angie-module-zstd

As with the brotli modules, the loading directives go into the config:

load_module modules/ngx_http_zstd_static_module.so;
load_module modules/ngx_http_zstd_filter_module.so;

The configuration is also similar to the other modules, except that the static mode most likely will not be needed here, so the setting is commented out (the ngx_http_zstd_static_module loading line can then be dropped as well):

http {
  zstd      on;
  #zstd_static  on;
  zstd_min_length   1000;
  zstd_comp_level   5;
  zstd_types        text/plain text/css text/xml application/javascript application/json image/x-icon image/svg+xml;
}

As usual, choosing the compression level for the dynamic mode is a trade-off between CPU load and compression ratio. The next step is a table comparing the different algorithms in dynamic mode.

Compression method priority in Angie

With the extra modules there are now three different compression algorithms, each with a static and a dynamic mode. The immediate question: what happens if the client supports every algorithm, and all algorithms and modes are enabled? Two rules apply.

1. Between the static and the dynamic mode, the static one wins.
2. Among different modules within one mode, the one loaded later in the configuration wins.

Since the gzip module (and gzip_static) are part of the standard build and need no loading, they get the lowest priority. The extra modules (installed from packages) are loaded dynamically, so their order in the configuration is under your control.

Following the logic of this article — brotli as the priority choice for static compression and zstd for dynamic — the module loading order is:

load_module modules/ngx_http_brotli_static_module.so;
load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_zstd_filter_module.so;

With static compression, brotli is checked first and gzip is used if it is missing. Dynamic compression uses this order: zstd, brotli, gzip.

Compressor performance comparison

To pick compression levels for the various modules, throughput has to be tested. To keep testing simple, a single Angie worker process was used, access went over HTTPS, and the load was generated with the ab utility, roughly like this:

ab -n 1000 -k -c 1 \
 -H "Accept-Encoding: gzip,br,zstd" https://site.com/css/1.css

Real CSS, HTML and JS files of large size (about 1 MB) were used as test files.

The test results turned out to be quite unexpected. The most detailed results are given for the CSS file (770 KB).

For the HTML file (1,8 MB) the results are similar.

To complete the picture, the results for the JS file (1,5 MB):

What conclusions can be drawn from the test results? First, zstd is clearly the speed leader: at medium compression ratios it is about twice as fast as gzip. The result is expected, since zstd was created for exactly such modes.

The second conclusion is that at low compression levels brotli beats gzip in both speed and compression ratio. This was rather unexpected: gzip should have handled the dynamic mode better than brotli. The choice of test files may have played a part. Either way, real data should be used when picking a compression ratio.

From a worker process perspective, dynamic compression is a blocking operation, so static compression methods should be used as much as possible, while the dynamic mode should use fast settings and algorithms. With HTTP/2 or HTTP/3, all requests from one client are handled by a single worker process — that is, effectively on one CPU core of the server — which makes the blocking problem worse.

When choosing a dynamic compression level, the sum of compression time and transfer time of the compressed file also has to be accounted for. For each network throughput value there is an optimal setting for a given compressor. There is a handy resource for finding these combinations (it covers gzip and brotli), but keep in mind that it uses outdated compressor versions.

Summary

This article covered three main options for compressing text resources in Angie.

Based on the dynamic mode test results, the standard gzip showed the weakest results. Using this compressor can be recommended either in static mode (with zopfli) or with minimal compression settings for compatibility.

The brotli algorithm leads in compression quality and is the best option for the static mode. In the dynamic mode it can also compete with gzip on speed while providing a better compression ratio.

Finally, zstd is the best option for dynamic compression thanks to its maximum speed, with compression quality sometimes even better than gzip.

All in all, an interesting combination is zstd for dynamic compression, brotli for static compression, and gzip (both dynamic and static) for compatibility with old clients or for plain HTTP.

You should now be able to pick the best option for your server, based on what today's web compression algorithms actually do.