Pf-dup on a PfSense gateway for separate distributed IDS sensors

Tags: #<Tag:0x00007f8a165602f0> #<Tag:0x00007f8a16560200> #<Tag:0x00007f8a16560138> #<Tag:0x00007f8a16560070> #<Tag:0x00007f8a16560250> #<Tag:0x00007f8a165635b8> #<Tag:0x00007f8a1655fe18> #<Tag:0x00007f8a1655fd50> #<Tag:0x00007f8a1655fc60> #<Tag:0x00007f8a1655fb98>

Pf-dup on a PfSense for distributed IDS sensors

This is a lab-report of a network security monitoring experiment. Network intrusion detection system / network security monitoring architectures[1] need to mind SPAN[2] ports. Core switches and hardware load balancers don’t allow for many different mirror ports[3]. But we usually aren’t interested in getting all traffic of a modern multi-Gigabit network. It can overload our network analysis tools so that it can get very hard to handle the data.

Pf-dup[4] is a BSD Packet Filter feature, that allows for selective traffic duplication. It’s over a decade old, but I haven’t used it until recently. With this feature (or the Linux iptables TEE target feature) we can perform targeted network analysis[5].

Prepare the (virtual) Switch network

The setup is relatively simple. The virtual topology has got a designated vSwitch (Internal) that switches vLAN 42:

You don’t need to do this with VMware of course. You may not even need a switch since the duplication will use an IP target. It’s however preferable and can keep mirrored traffic separated.

Two switch ports are interesting for this experiment:

  1. pfsense_gateway - the FreeBSD host that routes the traffic and offers pf
  2. AutoGRC - the receiver of the mirrored traffic. A Ubuntu Linux system

– The other two may be ignored (are shut down)
– the pfsense_gatway host has got a WAN interface (em1), and a LAN interface (em2)

Prepare the pfSense gateway

Once connected we can configure this interface with the static IP 10.1.1.1/24.

We don’t need an upstream GW, because we don’t intend to route this.

Prepare the IDS sensor / NSM system (AutoGRC)

On the Linux receiver this should be straightforward:

[email protected]:~$ sudo ifconfig ens192 10.1.1.2\ 
netmask 255.255.255.0 up

[email protected]:~$ sudo ifconfig  ens192 promisc

Add the traffic duplication rule

One of the reasons why I keep up this website is, that other people documented their ideas as well. I found a Tao Security[6] blog by Richard Bejtlich from 2005 about this :slight_smile:

wan_if="em0"
cp_if="em2"
cp_ip="10.1.1.2"

pass out on $wan_if dup-to ($cp_if $cp_ip) proto tcp from any port 80 to any
pass in on $wan_if dup-to ($cp_if $cp_ip) proto tcp from any to any port 80

pfSense creates a copy of the generated rules in /tmp/rules.debug. We can simply add two lines to our pf config (at the end):

pass out on em0 dup-to (em2 10.1.1.2) proto tcp from any port 80 to any
pass in on em0 dup-to (em2 10.1.1.2) proto tcp from any to any port 80
  • em0 is the WAN interface
  • em2 is the OPT1 interface that’s directly connected to the receiver via the switch
    • 10.1.1.2 is the Linux receiver’s (AutoGRC) static IP

Then we may enable the selective traffic duplication without interfering with the pfSense rules:

[2.5.1-RELEASE][[email protected]]/: 
pfctl -f /tmp/rules.debug

References

I asked about that specifically, because pfSense manages the pf rules.

Traffic capture test with tshark

On the Linux receiver system, we can test the traffic duplication using tshark.

[email protected]:~$ sudo !!
sudo tshark -i ens192
Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens192'
    1 0.000000000 188.40.24.X → 144.76.153.Y TCP 74 48831 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=308966310 TSecr=0 WS=128
    2 0.000269907 144.76.153.Y→ 188.40.24.X TCP 74 80 → 48831 [SYN, ACK] Seq=0 Ack=1 Win=65228 Len=0 MSS=1460 WS=128 SACK_PERM=1 TSval=4241488132 TSecr=308966310
    3 0.000532648 188.40.24.X → 144.76.153.Y TCP 66 48831 → 80 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=308966310 TSecr=4241488132
    4 0.005588488 188.40.24.X → 144.76.153.Y HTTP 163 GET / HTTP/1.1
    5 0.005655736 144.76.153.Y → 188.40.24.X TCP 66 80 → 48831 [ACK] Seq=1 Ack=98 Win=65664 Len=0 TSval=4241488142 TSecr=308966312
    6 0.005662955 144.76.153.Y → 188.40.24.X HTTP 453 HTTP/1.1 301 Moved Permanently  (text/html)
    7 0.006027786 188.40.24.X → 144.76.153.Y TCP 66 48831 → 80 [ACK] Seq=98 Ack=388 Win=30336 Len=0 TSval=308966312 TSecr=4241488142
  • Traffic from the WAN interface (these are public IPv4 IPs) gets duplicated
  • The vSwitch topology works fine for this
  • We see dest port 80 traffic, ingress, and egress (nothing else, although the excerpt here is very short)

Mission success. 2005 - 2021 :slight_smile:

Why not run all of that on the pfSense GW?

First of all running complex code (like network packet parsers) on perimeter systems isn’t in line with sane security objectives that are concerned about the (software and network) integrity. We are parsing arbitrary data on a critical system; with more or less random code.
Granted, if you need preventive functions you may not have a choice. But we should be concerned about the software integrity of central network gateways. Separating the systems is key: the resulting analysis can result in automated firewall rule changes as well.

We can think beyond pf(Sense): with such an approach it’s possible to perform targeted in-depth security tests, for example on TCP port 80. But we may not need to be that selective and could mirror more.
– Maybe in order to look for long-lasting connections, that indicate command and control (CnC) channels from Malware. Maybe with tools like Rita[7], Zeek[8] or Suricata. Lots of options, which we can try out on a separate Linux system.

We could use a SPAN port and mirror Gigabits of traffic to a BSD system, that can distribute the traffic to specifically configured sensors. It’s also possible that the sensors reconfigure the traffic distribution on demand (it’s not complex).
There will be limits to the performance of such a system, but it’s much better to know what to look for than to drown in network data.

Summary


Mirroring WAN traffic over OPT1 with pf-dup

  • (selective) traffic duplication from a pf(Sense) host is possible (low effort)
    • ideally we have little code on a perimeter gateway
    • installing lots of software on a “firewall” isn’t necessary
  • traffic duplication (via pf-dup) doesn’t need to interfere with pfSense’s rule management
  • selective network traffic analysis can help security and network troubleshooting
    • BSD is useful for that, and so is Linux
    • VMware’s vSwitch tech (typical Enterprise IT) doesn’t have a problem with this
  • FOSS network security tools (Rita, Zeek, Bro, Suri…) can be used independently, via this setup
Changelog
  • 30.05.2021: grammar and typos fixed

  1. (I am not concerned about prevention systems here) ↩︎

  2. Switched Port Analyzer usually mirror all traffic - that’s why they are also called mirror ports ↩︎

  3. F5 ClonePools are the same. This can work in an Enterprise network setting. ↩︎

  4. Pf-dup manpage ↩︎

  5. For example generate Netflow records from mirrored traffic ↩︎

  6. Distributed Traffic Collection with Pf Dup-To ↩︎

  7. GitHub - activecm/rita: Real Intelligence Threat Analytics (RITA) is a framework for detecting command and control communication through network traffic analysis. ↩︎

  8. https://zeek.org/ ↩︎

This topic was automatically closed after 2 days. New replies are no longer allowed.