Linux Router, Firewall and IDS Appliance

Linux Router, Firewall and IDS Appliance

Over the years, I’ve chewed through quite a few different routers, firewalls, even virtual appliances to connect my home network to the internet. Though most of these provided positive experiences, all of them had at least one point of friction, sometimes to the point of being a dealbreaker.

  • PFSense is a great platform, but has terrible ethics.
  • Sophos is proprietary and has an awful CLI.
  • Untangle feels more like an ad than a product.
  • Mikrotik is cheap as hell (for better or for worse).
  • Cisco.

Furthermore, many commercial security products ‘anonomously’ submit samples of your traffic for analysis. While this may have a net positive impact on security at large, it sets a dangerous precident that security and privacy are not compatible.

But, most of all, I want to use the same robust set of tools for securing my network that I already use for securing my servers.

The focus of this project is to build a super reliable, durable, and stable network device from tried and tested tech. This is not a project for pushing the limits or testing out flashy new stacks. This affinity for ‘boring’ technology will reflect on most of the choices made here, from the hardware to the way we configure services and daemons.

Objectives

The goal of this project is to build an appliance like device that will be feature comparable to a commercial ‘NGFW’ device, but affordable and based 100% on free and open source software. Here are some of the software components:

  • Debian Buster (v10.3)
  • bind9 DNS server
  • isc-dhcp-server DHCP server
  • iptables for port address translation and firewall functionality
  • Suricata Intrusion Detection system

Hardware

The first decision to make is the hardware requirements. I have a fairly basic network, but I think the core requirements will be the same for most home and small businesses.

  • Two or more Gigabit Ethernet adapters
  • Two or more CPU cores
  • 2-4 GiB of memory
  • 120 GiB of storage (32 or less would likely be fine, but I would like space for IDS logs)
  • Very low power usage
  • Few moving parts, ideally fanless

With this in mind, I decided the best option would be the Qotom Q190G4-S02 kit. It’s based on the Intel Celeron J1900 SoC platform, and has four Intel Gigabit ports. It certainly doesn’t have a lot of power, but most of what it needs to do is move packets between buffers very quickly, so it should do okay.

PFSense Users beware! Netgate’s position is that only devices with AES-NI instruction set support will be compatible with future versions. If you want to run PFSense, don’t buy anything based on the J1900 patform.

Cost breakdown, excluding taxes/duties/shipping:

Item Qty $CAD
Qotom Q190G4 S02 Barebone PC 1 249.95
Hynix 4G DDR3L SODIMM 1 21.99
MSATA 120GB Solid State Drive (off brand) 1 34.99
Software Licensing 0 0.00
TOTAL $306.93

Software

For this type of appliance-like hardware, I like to stick with Debian Stable. For the most part, installation was straightforward on this system, the only hiccup being that it does need the nonfree installer for the NICs, and if the BIOS can’t be updated immediately you may need to boot with acpi=off as indicated in this Debian bug report thread.

I also made some tweaks in the BIOS to make sure the device will power on every time it’s plugged in, whether it was previously powered on or not. It needs to operate like an appliance so this is important.

Once the OS is installed, I typically install the regular creature comforts, tmux, htop, nload, dns-utils and a few other nice to have tools. Then, the real work of configuring the device begins.

Automated Configuration

Since the system is all open source, it felt appropriate to build and release an Ansible role to provision and maintain my firewall.

Add it to your project:

git submodule add https://github.com/noahbailey/ansible-router

Or, if you prefer to set it up manually, or just want to see how the sausage is made, proceed to the next section.

Configuration

For the rest of this section, we’ll make these assumptions:

  • The ‘outside’ interface is named eth0
  • The ‘inside’ interface is named eth1
  • Our internal subnet range is 10.98.76.0/24, and is on a tagged VLAN 76.
  • Our external IPv4 address is 99.88.77.66 (Assigned through DHCP)

Interfaces

Before pushing packets, interfaces need addresses. Debian uses ifupdown2, the old fashioned but tried and true network init system.

Inside the /etc/network/interfaces file, we enter:

auto eth0
iface enp1s0 inet dhcp

auto eth1.76
iface eth1.76 inet static
    address 10.98.76.1/24
    vlan_raw_device eth1

This can be brought into effect by restarting the network service, and re-initializing all the interfaces. It’s very likely that SSH sessions will cut out at this point.

sudo service networking restart

If operating over SSH, you can also bring up the interfaces independently, for example to only reconfigure the WAN interface:

sudo ifdown eth0 && sudo ifup eth0

Once it’s been confirmed that all the interfaces are configured correctly, we can begin setting the device up as a firewall.

Kernel Network Stack

The very core of this device is its ability to move packets around. For that, we’ll use trusty IPtables to set up NAT.

First, enable IP forwarding:

/etc/sysctl.conf

net.ipv4.ip_forward=1

This allows packets to go across the device, rather than the device being the termination point for network connections.

Put this change into effect by either rebooting or reloading the network stack:

sudo sysctl --system

IPtables

While IPtables comes with Debian, we’ll install some tooling around it to make our lives easier:

sudo apt install -y iptables-persistent

What this package does is ensure that the rules are always loaded at boot time from /etc/iptables/rules.v4

Then, the rules can be added to the system:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
# -------> INPUT 
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m comment --comment "Default deny rule" -j REJECT --reject-with icmp-host-unreachable
# -------> FORWARD 
-A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT -m comment --comment "Outside->Inside"
-A FORWARD -i eth1 -o eth0 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment "Inside->Outside" -j ACCEPT
-A FORWARD -m comment --comment "Default deny rule" -j REJECT --reject-with icmp-host-unreachable
COMMIT
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT

These rules are a ‘sane default’ that will quickly get the device to a usable state as a firewall.

Once the rules are ready, we can either restart the iptables service or simply restore from the file:

sudo iptables-restore /etc/iptables/rules.v4

And, the active in-memory rules can be viewed:

sudo iptables -L -v

At this point, the device should be functional as a basic router. However, there is still some work to do.

DHCP Server

One of the network daemons we take for granted is DHCP. It’s always there in the background allowing our devices quick and easy autoconfig.

This network, like many others, will provide DHCP to clients using the isc-dhcp-server.

First, install the package:

sudo apt install -y isc-dhcp-server

The package installation will immediately fail. This is normal. By default dhcpd doesn’t have any configuration and the daemon will crash. This is to protect your network from rogue servers starting up automatically.

Next, the config file can be added:

option domain-name "example.com";
option domain-name-servers 10.98.76.1, 1.1.1.1;

default-lease-time 600;
max-lease-time 7200;

ddns-update-style none;

subnet 10.98.76.0 netmask 255.255.255.0 {
  authoritative;
  range 10.98.76.50 10.98.76.250;
  option routers 10.98.76.1;
}

Finally, the service can be started and enabled.

sudo service isc-dhcp-server start
sudo systemctl enable --now isc-dhcp-server

At this point, the firewall will begin issuing leases to clients on the local subnet.

DNS Server

Next, we configure a basic DNS server, starting by installing bind9.

sudo apt install bind9 bind9-host

Then, the server config file can be set up:

/etc/bind/named.conf.options

acl "trusted" {
	10.98.76.0/24;
	localhost;
};

options {
	directory "/var/cache/bind";
	forwarders {
		1.1.1.1;
		1.0.0.1;
	};
	dnssec-validation auto;
	auth-nxdomain no; 
	listen-on-v6 { any; };
	listen-on { any; };
	allow-recursion { trusted; };
	allow-query-cache { trusted; };
	allow-transfer { none; };
};

Before restarting the server, it’s good practice to check the config.

sudo named-checkconf

If that comes back clean, we’re good to go:

sudo service bind9 restart

At this point, all of the basic daemons are set up and ready to act as a firewall. But in terms of actual threat detection we can do better.

Intrusion Detection System

To me, an IDS is a must have. Without visibility and alerting it can be very difficult to respond to potential threats, and even harder to react. While an IPS takes time and effort to tune to the point it’s not regularly causing network disruptions, an IDS doesn’t interfere with traffic at all.

Think of an IPS as an automated machine gun robot that kills on sight, and an IDS as a system of lights and CCTV cameras.

IDS Install and Config

The latest Debian Stable (Buster) ships with a reasonably up to date Suricata. If desired, one could backport from Sid or Testing or install from the Ubuntu PPA repo.

sudo apt install suricata suricata-update

After installing, the only tweaks needed to get up and running are to change the interfaces and set up the network ranges.

vars: 
  address-groups: 
    HOME_NET: "[10.98.76.0/24]"
    EXTERNAL_NET: "!$HOME_NET"

And, further down,

af-packet: 
  - interface: eth0
    threads: auto 
  - interface: eth1
    threads: auto

The only other tweak to make to the default config is to disable the built in rules and let suricata-update manage rule updates.

default-rule-path: /var/lib/suricata/rules
rule-files: 
  - suricata.rules

The service should start up and run at this point.

sudo systemctl enable --now suricata

Definition Updates

To make sure we’re always getting up to date threat intel, the included suricata-update tool can be used. Out of the box there are a few feeds available which can be enabled.

sudo suricata-update enable-source et/open
sudo suricata-update enable-source oisf/trafficid
sudo suricata-update enable-source tgreen/hunting
sudo suricata-update enable-source sslbl/ssl-fp-blacklist
sudo suricata-update enable-source sslbl/ja3-fingerprints

Then we can pull the data from all these sources into the rules cache.

sudo suricata-update

If it succeeds the rules can be reloaded back into Suricata.

sudo kill -USR2 $(pidof suricata)

That’s great, but what about automated updates? Thankfully, these can be combined into an cron task.

/etc/cron.d/suricata-rules-updates

00 0,6,12,18 * * *  root  (suricata-update && kill -USR2 `pidof suricata`)

Monitoring

Collecting IDS Logs

Suricata records all logs into a json structured format, making it very easy to ship into Elasticsearch. For this, I install Filebeat and set up a data source.

filebeat.inputs:
...
  - type: log
    enabled: true
    json.keys_under_root: true
    paths:
      - "/var/log/suricata/eve.json"
...
output.logstash:
  hosts: ["10.98.76.10:5044"]

After restarting Filebeat, data immediately begins shipping to the log server and being indexed into the ElasticSearch database.

Metrics

I also choose to install Telegraf on this device to have detailed monitoring and health checks.

[[outputs.influxdb]]
  urls = ["http://10.98.76.11:8086"]

Most other default settings for Telegraf are fine.

Check IDS rules

Once the whole system is up and running, it’s time to test the alerting. The simplest way is to query a DNS host record that is known for its use in prolific ransomware cyberattacks.

dig a 3wzn5p2yiumh7akj.onion

This should almost immediately return an alert with description ET TROJAN Cryptowall .onion Proxy Domain in the logs. The full Elasticsearch document can be viewed in raw JSON format:

{
  "_index": "logstash-2020.02",
  "_type": "doc",
  "_id": "zGhnRnABD9IAo1WeYqEw",
  "_score": 1,
  "_source": {
    "offset": 888,
    "flow_id": 999,
    "alert": {
      "rev": 2,
      "signature_id": 2022048,
      "signature": "ET TROJAN Cryptowall .onion Proxy Domain",
      "severity": 1,
      "action": "allowed",
      "metadata": {
        "updated_at": [
          "2019_08_28"
        ],
        "created_at": [
          "2015_11_09"
        ]
      },
      "gid": 1,
      "category": "A Network Trojan was detected"
    },
    "tx_id": 0,
    "dest_port": 53,
    "prospector": {
      "type": "log"
    },
    "src_ip": "10.11.12.13",
    "beat": {
      "name": "router",
      "hostname": "router",
      "version": "6.8.6"
    },
    "dest_ip": "10.20.30.40",
    "flow": {
      "bytes_toserver": 105,
      "start": "2020-02-14T20:12:24.067505-0500",
      "pkts_toserver": 1,
      "pkts_toclient": 0,
      "bytes_toclient": 0
    },
    "event_type": "alert",
    "proto": "UDP",
    "source": "/var/log/suricata/eve.json",
    "input": {
      "type": "log"
    },
    "in_iface": "ens1.10",
    "timestamp": "2020-02-14T20:12:24.067505-0500",
    "geoip": {},
    "@version": "1",
    "app_proto": "dns",
    "host": {
      "name": "router"
    },
    "log": {
      "file": {
        "path": "/var/log/suricata/eve.json"
      }
    },
    "tags": [],
    "stream": 0,
    "@timestamp": "2020-02-15T01:12:24.242Z",
    "src_port": 34547
  },
  "fields": {
    "flow.start": [
      "2020-02-15T01:12:24.067Z"
    ],
    "@timestamp": [
      "2020-02-15T01:12:24.242Z"
    ]
  }
}

If this doesn’t produce an alert, check that all the services and logging agents are configured correctly.

Conclusion

So, after it’s all set up, what you are left with is a device that is 100% under your control. Because of the modularity of this, it could run on any distribution (with some modifications), and most components could be replaced. Automation tooling can also be replaced.

By building your own router/firewall/security appliance, you are reclaiming digital sovereignty in an age where it is being increasingly eroded.