Block web scanners with ipset & iptables

Block web scanners with ipset & iptables

Anybody who runs an internet-facing webserver has seen their fair share of spammy scanners in the logs. It varies server to server, but some of mine get up to 15,000 scans per day.

Almost all of these are harmless network mappers, but they still annoy me. Many are compromised hosts or belong to hackers & organized crime rings. While it’s possible to create false positives, it’s probably safe to block all of these.

So I’m going to ban these spammy scanners from my servers automatically, for several weeks at a time. Once a malicious IP has been confirmed, it will stay blocked indefinitely Here’s how:

Configure Nginx to log default virtualhost traffic

First and foremost, we need to separate the legit traffic from the scans. The easiest way to do this is to point the default virtualhost at a separate log file. For nginx, a default server block can be added to the end of nginx.conf:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    server_name_in_redirect off;
    location / {
        return 404;
        access_log /var/log/nginx/spam.log; 

server {
    listen 443 ssl  default_server;
    listen [::]:443 ssl default_server;
    server_name _;
    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
    ssl_protocols TLSv1.3 TLSv1.2;
    server_name_in_redirect off;
    location / {
        return 404;
        access_log /var/log/nginx/spam.log; 

With this configuration, all traffic that does not have a valid hostname will be 404’d and logged to spam.log, keeping the actual server logs much cleaner. This logfile will also be used to generate our block list later.

Unless you have a well-known or high traffic site, this will filter about 95% of the scanner traffic, since they primarily scan IP ranges rather than by hostname.

Configure logrotate to disable compression

To make parsing historical data easier, we will disable compression on old web server log files. Make sure you have sufficient storage before doing this. For nginx, the file /etc/logrotate.d/nginx will have two lines commented out:

        rotate 14

Once applied, new log files will only be rotated, not gzipped. It is possible to unzip the compressed log files for analysis, but it is much easier to have them uncompressed.

Install and configure ipset

Ipset is a tool for managing hash tables in the kernel for fast lookup. Rather than creating hundreds or thousands of individual firewall rules, a single rule can perform a quick lookup on the table without impacting performance in a meaningful way.

First, the tools must be installed:

sudo apt install ipset ipset-persistent
sudo systemctl enable --now ipset.service

Then, the table can be created:

ipset create scanners hash:ip hashsize 4096 counters

This creates an ipset list with counters enabled, which will allow us to check the hit rate on each rule later.

Process Nginx logs to build banned IP list

A simple script is used to process the nginx log and generate the ipset list:

After execution, the ipset list will contain several IPs to block:

# ipset list -t

Name: scanners
Type: hash:ip
Revision: 5
Header: family inet hashsize 4096 maxelem 65536 counters bucketsize 12 initval 0xe1ff9cd2
Size in memory: 27632
References: 0
Number of entries: 359

Additionally, a cron job should be used to regularly refresh the list with any new malicious IPs from the nginx log file:


00 * * * *    root    /opt/

Configure iptables to lookup & block banned IPs

The iptables rules for the server should be modified to include the set lookups. For performance reasons, established connections should be allowed to ‘bypass’ the rules - only new connections should be examined.

An example for a web server:


-A INPUT -m state --state INVALID -j DROP
-A INPUT -m set --match-set scanners src -j DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -s -j ACCEPT

Load the ruleset, either using iptables-restore or by restarting the system (if iptables-persistent is installed).

sudo iptables-restore /etc/iptables/rules.v4

Before too long, the rules will begin to get matches, and therefore block requests.

# iptables -L -v -n

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in    out   source           destination         
1007K 1173M ACCEPT     all  --  *     *        state RELATED,ESTABLISHED
   12  1344 DROP       all  --  *     *        state INVALID 
  268 15228 DROP       all  --  *     *        match-set scanners src
23196 1392K ACCEPT     all  --  lo    *        
  524 27891 ACCEPT     tcp  --  *     *        tcp dpt:80
 2829  170K ACCEPT     tcp  --  *     *        tcp dpt:443
    7   412 ACCEPT     tcp  --  *     *        tcp dpt:22
 7826  688K DROP       all  --  *     *        

Likewise, the ipset list will begin to show hits on the counters:

# ipset list scanners | tail -n +9 | sort -k 3 -g -r  | awk '{print $1"\t"$3"\t"$5}' | less  1256    75360   432     25920  224     11648  108     4320  42      2520

This will grow over time as more requests are logged and dropped.

After 14 days, the nginx logs will be removed, and IPs will be flushed from the blocklist shortly after. Effectively, a banned IP will be blocked for just over two weeks.

It’s also possible to configure logrotate to retain additional logs, which would allow the list to grow over time and maintain a much larger list of banned hosts. This may or may not be desirable, depending on the threat model of the server.