Pf-dup on a PfSense for distributed IDS sensors
This is a lab-report of a network security monitoring experiment. Network intrusion detection system / network security monitoring architectures need to mind SPAN ports. Core switches and hardware load balancers don’t allow for many different mirror ports. But we usually aren’t interested in getting all traffic of a modern multi-Gigabit network. It can overload our network analysis tools so that it can get very hard to handle the data.
Pf-dup is a BSD Packet Filter feature, that allows for selective traffic duplication. It’s over a decade old, but I haven’t used it until recently. With this feature (or the Linux
iptables TEE target feature) we can perform targeted network analysis.
Prepare the (virtual) Switch network
The setup is relatively simple. The virtual topology has got a designated vSwitch (Internal) that switches vLAN 42:
You don’t need to do this with VMware of course. You may not even need a switch since the duplication will use an IP target. It’s however preferable and can keep mirrored traffic separated.
Two switch ports are interesting for this experiment:
pfsense_gateway - the FreeBSD host that routes the traffic and offers
- AutoGRC - the receiver of the mirrored traffic. A Ubuntu Linux system
– The other two may be ignored (are shut down)
– the pfsense_gatway host has got a WAN interface (em1), and a LAN interface (em2)
Prepare the pfSense gateway
Once connected we can configure this interface with the static IP 10.1.1.1/24.
We don’t need an upstream GW, because we don’t intend to route this.
Prepare the IDS sensor / NSM system (AutoGRC)
On the Linux receiver this should be straightforward:
[email protected]:~$ sudo ifconfig ens192 10.1.1.2\ netmask 255.255.255.0 up [email protected]:~$ sudo ifconfig ens192 promisc
Add the traffic duplication rule
One of the reasons why I keep up this website is, that other people documented their ideas as well. I found a Tao Security blog by Richard Bejtlich from 2005 about this
wan_if="em0" cp_if="em2" cp_ip="10.1.1.2" pass out on $wan_if dup-to ($cp_if $cp_ip) proto tcp from any port 80 to any pass in on $wan_if dup-to ($cp_if $cp_ip) proto tcp from any to any port 80
pfSense creates a copy of the generated rules in
/tmp/rules.debug. We can simply add two lines to our pf config (at the end):
pass out on em0 dup-to (em2 10.1.1.2) proto tcp from any port 80 to any pass in on em0 dup-to (em2 10.1.1.2) proto tcp from any to any port 80
- em0 is the WAN interface
- em2 is the OPT1 interface that’s directly connected to the receiver via the switch
- 10.1.1.2 is the Linux receiver’s (AutoGRC) static IP
Then we may enable the selective traffic duplication without interfering with the pfSense rules:
[2.5.1-RELEASE][[email protected]]/: pfctl -f /tmp/rules.debug
I asked about that specifically, because pfSense manages the pf rules.
Traffic capture test with tshark
On the Linux receiver system, we can test the traffic duplication using
[email protected]:~$ sudo !! sudo tshark -i ens192 Running as user "root" and group "root". This could be dangerous. Capturing on 'ens192' 1 0.000000000 188.40.24.X → 144.76.153.Y TCP 74 48831 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=308966310 TSecr=0 WS=128 2 0.000269907 144.76.153.Y→ 188.40.24.X TCP 74 80 → 48831 [SYN, ACK] Seq=0 Ack=1 Win=65228 Len=0 MSS=1460 WS=128 SACK_PERM=1 TSval=4241488132 TSecr=308966310 3 0.000532648 188.40.24.X → 144.76.153.Y TCP 66 48831 → 80 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=308966310 TSecr=4241488132 4 0.005588488 188.40.24.X → 144.76.153.Y HTTP 163 GET / HTTP/1.1 5 0.005655736 144.76.153.Y → 188.40.24.X TCP 66 80 → 48831 [ACK] Seq=1 Ack=98 Win=65664 Len=0 TSval=4241488142 TSecr=308966312 6 0.005662955 144.76.153.Y → 188.40.24.X HTTP 453 HTTP/1.1 301 Moved Permanently (text/html) 7 0.006027786 188.40.24.X → 144.76.153.Y TCP 66 48831 → 80 [ACK] Seq=98 Ack=388 Win=30336 Len=0 TSval=308966312 TSecr=4241488142
- Traffic from the WAN interface (these are public IPv4 IPs) gets duplicated
- The vSwitch topology works fine for this
- We see dest port 80 traffic, ingress, and egress (nothing else, although the excerpt here is very short)
Mission success. 2005 - 2021
Why not run all of that on the pfSense GW?
First of all running complex code (like network packet parsers) on perimeter systems isn’t in line with sane security objectives that are concerned about the (software and network) integrity. We are parsing arbitrary data on a critical system; with more or less random code.
Granted, if you need preventive functions you may not have a choice. But we should be concerned about the software integrity of central network gateways. Separating the systems is key: the resulting analysis can result in automated firewall rule changes as well.
We can think beyond pf(Sense): with such an approach it’s possible to perform targeted in-depth security tests, for example on TCP port 80. But we may not need to be that selective and could mirror more.
– Maybe in order to look for long-lasting connections, that indicate command and control (CnC) channels from Malware. Maybe with tools like Rita, Zeek or Suricata. Lots of options, which we can try out on a separate Linux system.
We could use a SPAN port and mirror Gigabits of traffic to a BSD system, that can distribute the traffic to specifically configured sensors. It’s also possible that the sensors reconfigure the traffic distribution on demand (it’s not complex).
There will be limits to the performance of such a system, but it’s much better to know what to look for than to drown in network data.
- (selective) traffic duplication from a pf(Sense) host is possible (low effort)
- ideally we have little code on a perimeter gateway
- installing lots of software on a “firewall” isn’t necessary
- traffic duplication (via
pf-dup) doesn’t need to interfere with pfSense’s rule management
- selective network traffic analysis can help security and network troubleshooting
- BSD is useful for that, and so is Linux
- VMware’s vSwitch tech (typical Enterprise IT) doesn’t have a problem with this
- FOSS network security tools (Rita, Zeek, Bro, Suri…) can be used independently, via this setup
- 30.05.2021: grammar and typos fixed
(I am not concerned about prevention systems here) ↩︎
Switched Port Analyzer usually mirror all traffic - that’s why they are also called mirror ports ↩︎