This article continues our look at the tools the Angie web server offers for fighting DoS (and a bit of DDoS) attacks. Here we cover third-party modules and the Fail2Ban system.
Video version
A video version of this article is available on YouTube.
Blocking clients with Fail2Ban
Fail2Ban can block illegitimate users of various services. It already ships with rules for blocking clients that guess login credentials for SSH, FTP, HTTP authentication, databases, and so on. The service runs by parsing application logs on a schedule. Every enabled plugin (jail in Fail2Ban terminology) has its log locations and ban criteria specified. These logs are scanned regularly, and bans are issued based on the entries found in them.
For our purposes the right jail is nginx-limit-req. It works from the messages the Limit Req module writes to error.log, so we will assume it is already configured.
Installing Fail2Ban on Ubuntu is straightforward:
apt install fail2ban
For configuration, copy jail.conf to jail.local (in /etc/fail2ban/) and put your own settings in jail.local.
Bring the [nginx-limit-req] section in jail.local to the following form:
[nginx-limit-req]
port = http,https
enabled = true
filter = nginx-limit-req
action = iptables-multiport[name=ReqLimit, port="http,https", protocol=tcp]
logpath = /var/log/angie/*error.log
findtime = 600
bantime = 7200
maxretry = 4
This enables the search for limit-violation messages in /var/log/angie/*error.log. A ban requires 4 triggers (maxretry) within the last 600 seconds (findtime), and the ban lasts 7200 seconds (2 hours — bantime). The blocking method is the iptables-multiport action. It is a fairly targeted restriction: access is denied only on the listed ports (TCP 80 and 443). This action adds rules through iptables, one per offender. The scheme suits a small number of bans (up to 100–1000 addresses); at a larger scale, the iptables-ipset-proto4 action is recommended. To use ipset, remember to install the package:
apt install ipset
Now we are ready to test. Point your favorite tool at the site with the rate limit in place and you will get banned. You can verify it like this:
fail2ban-client status nginx-limit-req
Status for the jail: nginx-limit-req
|- Filter
| |- Currently failed: 0
| |- Total failed: 1992
| `- File list: /var/log/angie/error.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: 192.168.122.1
Finding the iptables rule is easy too:
iptables -nvL
...
Chain f2b-ReqLimit (1 references)
pkts bytes target prot opt in out source destination
0 0 REJECT 0 -- * * 192.168.122.1 0.0.0.0/0 reject-with icmp-port-unreachable
113 14563 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
The client address 192.168.122.1 is now blocked, and it will be unblocked in 2 hours. To unblock it immediately, run:
fail2ban-client set nginx-limit-req unbanip 192.168.122.1
To ban a client, use banip instead of unbanip.
This setup can fend off small DDoS attacks as long as the request-rate limits trigger. In a full-scale DDoS attack there will be a large number of bots, and none of them will exceed the limits set for normal users.
Geoblocking clients
The next way to block unwanted requests to the server is geoblocking based on the client's IP. Suppose your site is local: its main audience lives in a single country. Denial-of-service attacks, on the other hand, often use botnets assembled from different countries. The obvious solution: block every country except the ones where your audience is. You will lose some legitimate traffic and access for some search bots, but at least the site stays available to your main audience.
To implement geoblocking we will use the third-party GeoIP2 module. Install it from the Angie repository:
apt install angie-module-geoip2
Load it in the main context (versions for the http and stream contexts also exist):
load_module modules/ngx_http_geoip2_module.so;
Next you need a GeoIP database that maps IP ranges to locations. A database from this GitHub repository will do for our purposes. Download the country (Country) database. In my example the database file sits in /etc/angie/. Do the basic setup in the http context:
http {
geoip2 /etc/angie/GeoLite2-Country.mmdb {
auto_reload 5m;
$geoip2_metadata_country_build metadata build_epoch;
$geoip2_data_country_code default=RU source=$remote_addr country iso_code;
$geoip2_data_country_name country names en;
}
map $geoip2_data_country_code $allowed_country {
default no;
RU yes;
BY yes;
}
…
}
Here the geoip2 block loads the database from disk and sets the default country (RU). To map countries to an access flag, we create a map section that sets $allowed_country to yes for the country codes RU and BY — swap in the codes for your own audience's countries. We can now use the $allowed_country variable to control access at the site level or in other parts of the server. For example:
server {
…
if ($allowed_country = no) {
return 403;
}
}
This configuration returns 403 when the variable is $allowed_country = no. You could also use 444, but 403 makes it easier to sort out problems with clients that were blocked by mistake. Another blocking option: allow everything and block only specific countries. For example, log analysis shows a lot of bots from Romania — block Romania.
With this kind of blocking you have to account for unavoidable geolocation errors, as well as clients that use VPNs or proxies.
Finally, if none of these methods cut off the junk traffic, move on to filtering bots with the Testcookie module.
Testcookie module
The idea behind the third-party Testcookie module is to detect bot traffic from what the client can do: process cookies and run JavaScript. On the first access attempt the client receives a response with a cookie-setting header, or an HTML document with JavaScript whose result is also a cookie. On repeat requests the cookie value is checked, and access is granted when it is correct.
First, the standard module installation:
apt install angie-module-testcookie
Load it in the main context:
load_module modules/ngx_http_testcookie_access_module.so;
Now the module has to be configured in one of its operating modes. We will use the variant that relies on the JavaScript check. For a more detailed overview, see the write-up by the module's author and the documentation.
It makes sense to configure the checks at the server block (site) level:
server {
listen 80;
root /var/www/static;
testcookie off;
testcookie_name BPC;
testcookie_secret keepmescdret31hio31iocqiojcijcqijowc1ijwcq;
testcookie_session $remote_addr;
testcookie_arg attempt;
testcookie_max_attempts 3;
testcookie_fallback /cookies.html?backurl=http://$host$request_uri;
testcookie_get_only on;
testcookie_redirect_via_refresh on;
testcookie_refresh_encrypt_cookie on;
testcookie_refresh_encrypt_cookie_key random;
testcookie_refresh_encrypt_cookie_iv random;
testcookie_refresh_template '<html><body>setting cookie...<script type="text/javascript" src="/aes.min.js" ></script><script>function toNumbers(d){var e=[];d.replace(/(..)/g,function(d){e.push(parseInt(d,16))});return e}function toHex(){for(var d=[],d=1==arguments.length&&arguments[0].constructor==Array?arguments[0]:arguments,e="",f=0;f<d.length;f++)e+=(16>d[f]?"0":"")+d[f].toString(16);return e.toLowerCase()}var a=toNumbers("$testcookie_enc_key"),b=toNumbers("$testcookie_enc_iv"),c=toNumbers("$testcookie_enc_set");document.cookie="BPC="+toHex(slowAES.decrypt(c,2,a,b))+"; expires=Thu, 31-Dec-37 23:55:55 GMT; path=/";location.href="$testcookie_nexturl";</script></body></html>';
location = /aes.min.js {
}
location = /cookies.html {
}
}
In this configuration we use a check template with JavaScript implementing the slow AES algorithm (aes.min.js); get its code from this repository, then apply the patch so it works with the module.
From now on the site is reachable only with a correct BPC cookie. Test it in a browser. Keep in mind that the module also filters traffic from legitimate bots, so it is worth adding an IP whitelist for known legitimate bots. That is what the testcookie_whitelist block is for (in the same server context):
testcookie_whitelist {
8.8.8.8/32;
}
Legitimate bots can now visit the site without hindrance. In general, the author has frozen the module's development: the last commit was 4 years ago as of the time of writing. Even so, it is a good foundation for building a more sophisticated solution (as the large number of forks indirectly suggests).
Summary
Across two articles we have covered a fairly broad range of tools for fighting simple DoS attacks with Angie. These tools can be used individually or combined. Repelling a full-scale DDoS attack, of course, requires specialized tools: a cloud service that can absorb large volumes of traffic and carries a reliable bot-detection system based on formal attributes and bot behavior.