🔥 ipfw: The Other FreeBSD Firewall

Numbered rules where the first match wins, tables, logging and built-in traffic shaping, tested with real traffic, and the way not to lock yourself out

Video: ipfw (FreeBSD video)
Prefer to watch? This short video runs this page's main commands on a real FreeBSD 15.0 machine, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.

Two firewalls in one system

FreeBSD ships three firewalls (pf, ipfw and ipfilter, from the documentation). You met pf on the pf page. This page is about ipfw, the one that is FreeBSD's own: older, simpler in some ways and with one feature that the others lack, traffic shaping (dummynet) built in. Run one firewall per machine, not both: two sets of rules at once are a headache to reason about. So the first thing we do is turn pf off.

The big difference you must learn is this:

In ipfw, the numbers are the whole story. Everything on this page was run for real on FreeBSD 15.0-RELEASE in a virtual machine, with the web server of the jails page as the target and the laptop that runs the virtual machine as the "outside" machine (the same test setup as the pf page, with port 18080 forwarded to the web server).

Step 1: pf off, ipfw loaded, safely

root@practice:~ # pfctl -d
pf disabled
root@practice:~ # sysrc pf_enable=NO pflog_enable=NO
pf_enable: YES -> NO
pflog_enable: YES -> NO

pfctl -d disables pf now, and the sysrc line stops it starting at boot.

root@practice:~ # kldstat | grep ipfw
root@practice:~ # ipfw list
ipfw: retrieving config failed: Protocol not available
root@practice:~ # kenv net.inet.ip.fw.default_to_accept=1
net.inet.ip.fw.default_to_accept="1"
root@practice:~ # kldload ipfw
root@practice:~ # kldstat | grep ipfw
11    1 0xffffffff8309a000    26570 ipfw.ko
root@practice:~ # ipfw list
65535 count ip from any to any not // orphaned dynamic states counter
65535 allow ip from any to any
root@practice:~ # sysctl net.inet.ip.fw.default_to_accept net.inet.ip.fw.one_pass net.inet.ip.fw.verbose
net.inet.ip.fw.default_to_accept: 1
net.inet.ip.fw.one_pass: 1
net.inet.ip.fw.verbose: 0
laptop:~ $ curl -sI -m 5 http://127.0.0.1:18080 | head -1
HTTP/1.1 404 Not Found

A rule, taken apart

An ipfw rule looks like this:

ipfw add NUMBER ACTION PROTOCOL from SOURCE to DESTINATION [OPTIONS]

A default-deny ruleset

root@practice:~ # ipfw -q add 100 allow ip from any to any via lo0
root@practice:~ # ipfw -q add 150 check-state
root@practice:~ # ipfw -q add 200 allow tcp from any to any established
root@practice:~ # ipfw -q add 250 allow ip from me to any out via vtnet0 keep-state
root@practice:~ # ipfw -q add 300 allow tcp from any to me 22 in via vtnet0 setup keep-state
root@practice:~ # ipfw -q add 400 allow tcp from any to me 80 in via vtnet0 setup keep-state
root@practice:~ # ipfw -q add 65000 deny ip from any to any
root@practice:~ # ipfw list
00100 allow ip from any to any via lo0
00150 check-state :default
00200 allow tcp from any to any established
00250 allow ip from me to any out via vtnet0 keep-state :default
00300 allow tcp from any to me 22 in via vtnet0 setup keep-state :default
00400 allow tcp from any to me 80 in via vtnet0 setup keep-state :default
65000 deny ip from any to any
65535 count ip from any to any not // orphaned dynamic states counter
65535 allow ip from any to any

Reading the seven rules, in order:

ipfw list prints the rules in order. (:default is just the name ipfw gives to the connection state it remembers.) Compare what we saw with no rules, for a port with nothing listening (the forwarded port 18081):

laptop:~ $ time curl -sI -m 5 http://127.0.0.1:18081; echo "exit code: $?"

real	0m0.010s
user	0m0.002s
sys	0m0.008s
exit code: 56

The machine's own network stack answered at once: 0.01 seconds, exit code 56 (the connection was refused). Now with the ruleset:

laptop:~ $ curl -sI -m 5 http://127.0.0.1:18080 | head -1
HTTP/1.1 404 Not Found
laptop:~ $ time curl -sI -m 5 http://127.0.0.1:18081; echo "exit code: $?"

real	0m5.017s
user	0m0.003s
sys	0m0.012s
exit code: 28
root@practice:~ # ipfw -a list
00100   0     0 allow ip from any to any via lo0
00150   0     0 check-state :default
00200 696 44828 allow tcp from any to any established
00250   0     0 allow ip from me to any out via vtnet0 keep-state :default
00300   0     0 allow tcp from any to me 22 in via vtnet0 setup keep-state :default
00400   9   602 allow tcp from any to me 80 in via vtnet0 setup keep-state :default
65000   1    44 deny ip from any to any
65535   0     0 count ip from any to any not // orphaned dynamic states counter
65535 401 34708 allow ip from any to any

First match wins

Now the lesson that makes ipfw different. Rule 400 allows web traffic. What happens if we add a deny with a number below 400, and what if the number is above?

root@practice:~ # ipfw add 350 deny tcp from any to me 80 in via vtnet0
00350 deny tcp from any to me 80 in via vtnet0
laptop:~ $ time curl -sI -m 5 http://127.0.0.1:18080; echo "exit code: $?"

real	0m5.020s
user	0m0.006s
sys	0m0.012s
exit code: 28
root@practice:~ # ipfw delete 350
root@practice:~ # ipfw add 350 reset tcp from any to me 80 in via vtnet0
00350 reset tcp from any to me 80 in via vtnet0
laptop:~ $ time curl -sI -m 5 http://127.0.0.1:18080; echo "exit code: $?"

real	0m0.017s
user	0m0.003s
sys	0m0.014s
exit code: 56
root@practice:~ # ipfw delete 350
laptop:~ $ curl -sI -m 5 http://127.0.0.1:18080 | head -1
HTTP/1.1 404 Not Found
root@practice:~ # ipfw add 450 deny tcp from any to me 80 in via vtnet0
00450 deny tcp from any to me 80 in via vtnet0
laptop:~ $ curl -sI -m 5 http://127.0.0.1:18080 | head -1
HTTP/1.1 404 Not Found
root@practice:~ # ipfw -a list | grep -E "^00(400|450)"
00400  38  2488 allow tcp from any to me 80 in via vtnet0 setup keep-state :default
00450   0     0 deny tcp from any to me 80 in via vtnet0
root@practice:~ # ipfw delete 450

Now a deny after the allow: ipfw add 450 deny .... The page still loads, because rule 400 was matched first and the packet never got as far as 450. The counters prove it: rule 400 matched 38 packets, rule 450 matched none. In pf you would put a rule later to override an earlier one. In ipfw, a rule that is too late is dead code. If a rule seems to be ignored, look for a lower-numbered rule that matches first.

Tables: ban an address without touching the rules

root@practice:~ # ipfw table badguys create
root@practice:~ # ipfw table badguys add 10.0.2.2
added: 10.0.2.2/32 0
root@practice:~ # ipfw table badguys list
10.0.2.2/32 0
root@practice:~ # ipfw add 340 deny tcp from "table(badguys)" to me 80 in via vtnet0
00340 deny tcp from table(badguys) to me 80 in via vtnet0
laptop:~ $ time curl -sI -m 5 http://127.0.0.1:18080; echo "exit code: $?"

real	0m5.019s
user	0m0.007s
sys	0m0.011s
exit code: 28
root@practice:~ # ipfw table badguys delete 10.0.2.2
deleted: 10.0.2.2/32 0
root@practice:~ # ipfw table badguys list
laptop:~ $ curl -sI -m 5 http://127.0.0.1:18080 | head -1
HTTP/1.1 404 Not Found
root@practice:~ # ipfw table all list
--- table(badguys), set(0) ---

Logging

root@practice:~ # ipfw delete 340
root@practice:~ # ipfw add 340 deny log tcp from "table(badguys)" to me 80 in via vtnet0
00340 deny log tcp from table(badguys) to me 80 in via vtnet0
root@practice:~ # sysctl net.inet.ip.fw.verbose=1
net.inet.ip.fw.verbose: 0 -> 1
root@practice:~ # ipfw table badguys add 10.0.2.2
added: 10.0.2.2/32 0
laptop:~ $ curl -sI -m 3 http://127.0.0.1:18080; echo "exit code: $?"
exit code: 28
root@practice:~ # tail -3 /var/log/security
Oct  5 21:00:24 practice newsyslog[1048]: logfile first created
Oct  6 14:33:45 practice kernel: ipfw: 340 Deny TCP 10.0.2.2:59092 10.0.2.50:80 in via vtnet0
root@practice:~ # ipfw table badguys delete 10.0.2.2
deleted: 10.0.2.2/32 0
root@practice:~ # ipfw -a list | grep "^00340"
00340    1     44 deny log tcp from table(badguys) to me 80 in via vtnet0

Add the word log to a rule and ipfw writes a line for every packet that matches, to the system log, in /var/log/security. Two things to know: logging also needs the setting net.inet.ip.fw.verbose=1 (it was 0), and each logged rule has a limit on how many lines it writes (from the documentation, not run here). The line says what we want to know: the time, ipfw: 340 Deny TCP 10.0.2.2:59092 10.0.2.50:80 in via vtnet0: rule number, action, source address and port, destination, direction and interface. The counter of rule 340 shows it caught 1 packet. (Unlike pf, there is no separate log device: the lines go through syslogd like everything else.)

Traffic shaping with dummynet

This is the feature that makes many people choose ipfw. dummynet can limit the bandwidth of traffic, delay it, and even drop a percentage of it, all with simple rules. We made a 300 KB file on the web server and downloaded it from the outside, for the baseline:

root@practice:~ # mkdir -p /jails/web/usr/local/www/nginx-dist
root@practice:~ # head -c 300000 /dev/random > /jails/web/usr/local/www/nginx-dist/test.bin
root@practice:~ # ls -l /jails/web/usr/local/www/nginx-dist/test.bin
-rw-r--r--  1 root wheel 300000 Oct  6 14:35 /jails/web/usr/local/www/nginx-dist/test.bin
laptop:~ $ curl -s -o /dev/null -w "%{size_download} bytes in %{time_total} s (%{speed_download} bytes/s)\n" http://127.0.0.1:18080/test.bin
300000 bytes in 0.008031 s (37355248 bytes/s)

(The jail's web root is a link to a folder that does not exist, so we made that folder first: a leftover from the poudriere page, where we switched the sample pages off.) The download takes 0.008 seconds. Now we shape it:

root@practice:~ # kldload dummynet
root@practice:~ # ipfw pipe 1 config bw 400Kbit/s
root@practice:~ # kldstat | grep dummynet
12    1 0xffffffff830c1000    128a0 dummynet.ko
root@practice:~ # ipfw pipe 1 show
00001: 400.000 Kbit/s    0 ms burst 0 
q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
root@practice:~ # ipfw add 360 pipe 1 ip from 10.0.2.50 80 to any out via vtnet0
00360 pipe 1 ip from 10.0.2.50 80 to any out via vtnet0
laptop:~ $ curl -s -o /dev/null -w "%{size_download} bytes in %{time_total} s (%{speed_download} bytes/s)\n" http://127.0.0.1:18080/test.bin
300000 bytes in 0.006313 s (47520988 bytes/s)
root@practice:~ # ipfw -a list | grep "^00360"
00360    0      0 pipe 1 ip from 10.0.2.50 80 to any out via vtnet0

And nothing happened: the download took 0.006 seconds, and the counter of rule 360 says 0 packets. Why? Because of first match wins: rules 150, 200 and 250 already allowed those packets, long before rule 360. The pipe rule must come before them:

root@practice:~ # ipfw delete 360
root@practice:~ # ipfw add 120 pipe 1 ip from 10.0.2.50 80 to any out via vtnet0
00120 pipe 1 ip from 10.0.2.50 80 to any out via vtnet0
laptop:~ $ curl -s -o /dev/null -w "%{size_download} bytes in %{time_total} s (%{speed_download} bytes/s)\n" http://127.0.0.1:18080/test.bin
300000 bytes in 6.175936 s (48575 bytes/s)
root@practice:~ # ipfw -a list | grep "^00120"
00120  209 308622 pipe 1 ip from 10.0.2.50 80 to any out via vtnet0
root@practice:~ # ipfw pipe 1 show
00001: 400.000 Kbit/s    0 ms burst 0 
q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 1 active
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0      209   308622  0    0   0
root@practice:~ # ipfw delete 120
root@practice:~ # ipfw pipe 1 delete
laptop:~ $ curl -s -o /dev/null -w "%{size_download} bytes in %{time_total} s (%{speed_download} bytes/s)\n" http://127.0.0.1:18080/test.bin
300000 bytes in 0.007804 s (38441824 bytes/s)

We deleted rule 360 and added the same rule as number 120. Now the 300 KB download took 6.18 seconds, which is 48,575 bytes per second, exactly the 400 kilobits that the pipe allows. The counter shows 209 packets (308,622 bytes) went through rule 120, and ipfw pipe 1 show lists the traffic that went through the pipe. Deleting the rule and the pipe brings the speed back to full (0.008 seconds in the last line). That is how you test an application over a slow link, or protect a network from one greedy download. Dummynet can also add delay (delay 100ms) and random packet loss (plr 0.05) to a pipe, and share bandwidth fairly between flows with queues (from the documentation, not run here).

Making it permanent

Rules typed at the prompt vanish at the next reboot. Put them in a script and tell the system to run it at boot:

root@practice:~ # cat > /etc/ipfw.rules <<'EOF'
> #!/bin/sh
> # TechCommune lab ruleset: default deny, SSH and HTTP in
> cmd="ipfw -q add"
> ipfw -q -f flush
> $cmd 100 allow ip from any to any via lo0
> $cmd 150 check-state
> $cmd 200 allow tcp from any to any established
> $cmd 250 allow ip from me to any out via vtnet0 keep-state
> $cmd 300 allow tcp from any to me 22 in via vtnet0 setup keep-state
> $cmd 400 allow tcp from any to me 80 in via vtnet0 setup keep-state
> $cmd 65000 deny ip from any to any
> EOF
root@practice:~ # sh -n /etc/ipfw.rules && echo syntax-ok
syntax-ok
root@practice:~ # sysrc firewall_enable=YES firewall_script=/etc/ipfw.rules firewall_logging=YES
firewall_enable: NO -> YES
firewall_script: /etc/rc.firewall -> /etc/ipfw.rules
firewall_logging: NO -> YES
root@practice:~ # grep "^firewall" /etc/rc.conf
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_logging="YES"
root@practice:~ # service ipfw restart 2>&1 | tail -4
Firewall rules loaded.
Firewall logging enabled.
root@practice:~ # ipfw list
00100 allow ip from any to any via lo0
00150 check-state :default
00200 allow tcp from any to any established
00250 allow ip from me to any out via vtnet0 keep-state :default
00300 allow tcp from any to me 22 in via vtnet0 setup keep-state :default
00400 allow tcp from any to me 80 in via vtnet0 setup keep-state :default
65000 deny ip from any to any
65535 count ip from any to any not // orphaned dynamic states counter
65535 allow ip from any to any

Then the real test, a reboot, with the rules coming from the script:

root@practice:~ # uptime
 2:36PM  up 20 secs, 1 user, load averages: 0.28, 0.06, 0.02
root@practice:~ # kldstat | grep -E "ipfw|dummynet"
 9    1 0xffffffff83033000    26570 ipfw.ko
root@practice:~ # ipfw list
00100 allow ip from any to any via lo0
00150 check-state :default
00200 allow tcp from any to any established
00250 allow ip from me to any out via vtnet0 keep-state :default
00300 allow tcp from any to me 22 in via vtnet0 setup keep-state :default
00400 allow tcp from any to me 80 in via vtnet0 setup keep-state :default
65000 deny ip from any to any
65535 count ip from any to any not // orphaned dynamic states counter
65535 deny ip from any to any
root@practice:~ # sysctl net.inet.ip.fw.default_to_accept net.inet.ip.fw.verbose
net.inet.ip.fw.default_to_accept: 0
net.inet.ip.fw.verbose: 1
root@practice:~ # service ipfw status
ipfw is enabled

The machine was back and reachable over SSH about 15 seconds after the reboot. uptime says 20 seconds, ipfw.ko was loaded by the startup scripts, and the rules are all there. Look at the last line: this time the built-in rule 65535 says deny, and default_to_accept is 0. At boot the module is loaded with the kernel's own default, which is deny, and our script then adds the rules that let us in. Between those two moments, everything is blocked (from the documentation, not run here), which is why the script must let SSH in. (firewall_logging also set verbose to 1.)

laptop:~ $ curl -sI -m 5 http://127.0.0.1:18080 | head -1
HTTP/1.1 403 Forbidden
laptop:~ $ time curl -sI -m 5 http://127.0.0.1:18081; echo "exit code: $?"

real	0m5.019s
user	0m0.006s
sys	0m0.010s
exit code: 28

And from the outside: port 18080 answers (403), and the unlisted port 18081 times out (exit 28 after 5 seconds). Same behaviour as before the reboot.

What we did not do

All from the documentation, not run here:

ipfw or pf?

Whichever you choose, choose one, and learn to test a ruleset the way we did: from the outside, with a way back in.

⚠️ Never enable ipfw on a remote machine without a way back in. Set net.inet.ip.fw.default_to_accept=1 before loading the module, put an SSH rule in before the final deny, rule established in, check the script with sh -n, and keep a second SSH session or the console open when you change anything.

Why did the ipfw rule never get used?<br>
It came after the one that said "allow everything".<br>
It is very first-come, first-served in there.

🧪 Try it yourself

On a FreeBSD virtual machine (never first on a production server), with a service to test:

1. Turn pf off (pfctl -d), set net.inet.ip.fw.default_to_accept=1 with kenv, and load ipfw. What does ipfw list show? 2. Add a default-deny ruleset: loopback, check-state, established, outgoing with keep-state, SSH, your service, and a deny at 65000. 3. From another machine, test the service and a closed port. How long does each take to answer? 4. Put a deny before your allow, then after it. Which one works? Check ipfw -a list. 5. Change deny to reset. What changes for the machine that is blocked? 6. Make a table, add the other machine's address, and ban it from your service. Remove it again without touching the rules. 7. Load dummynet, make a pipe of 400 Kbit/s, and shape a download. Put the pipe rule first at a number that is too high, then a number that works. 8. Write the rules into a script, check it with sh -n, set the firewall_* lines in rc.conf, and reboot.

The cheat sheet

← vm-bhyve: Virtual Machines with Short Commands · ↑ FreeBSD Corner · Linux Binary Compatibility: Running Linux Programs on FreeBSD →