⚖️ OpenBSD relayd

table <web> { 127.0.0.2 127.0.0.3 }

relay "www" {
	listen on 127.0.0.1 port 8080
	forward to <web> port 80 mode roundrobin check http "/" code 200
}

relayd(8) is the relay daemon in the OpenBSD base system. It accepts connections and passes them on to a group of servers, notices when one of them is down, can rewrite HTTP headers and block requests, and can do the TLS for servers that only speak plain HTTP. This page builds a small setup with two httpd sites behind one relay, takes one away, adds a header and a block rule, adds HTTPS, and shows the mistakes to watch for.

Every example ran for real on OpenBSD 8.0 (amd64) in a virtual machine, with root commands run through doas. The two servers are two httpd sites on two loopback addresses, because there is one machine. The output you see is the output the machine gave.

OpenBSD has a relay daemon in the base system, relayd(8). It sits in front of one or more servers, accepts the connections of the clients, and passes them on. In between it can spread the load, notice that a server is down and stop using it, add or remove HTTP headers, block requests, and take over the TLS encryption so that the servers behind it do not have to. It needs no installation, and syspatch keeps it up to date.

Every example here ran for real on OpenBSD 8.0 (amd64) in a virtual machine, as an ordinary user, with doas in front of the commands that need root. There is one machine only, so the two servers behind the relay are two httpd sites that listen on two extra loopback addresses, 127.0.0.2 and 127.0.0.3. On your network they would be two machines. First give the loopback interface the two addresses (they last until reboot, which is enough for a test):

$ doas ifconfig lo0 alias 127.0.0.2/32
$ doas ifconfig lo0 alias 127.0.0.3/32
$ ifconfig lo0 | grep 'inet 127'
	inet 127.0.0.1 netmask 0xff000000
	inet 127.0.0.2 netmask 0xffffffff
	inet 127.0.0.3 netmask 0xffffffff

Now the two servers, each with one page that says which one answered. The httpd page of this site has the details of this file:

$ cat /tmp/httpd.conf
server "one" {
	listen on 127.0.0.2 port 80
	root "/htdocs/one"
}

server "two" {
	listen on 127.0.0.3 port 80
	root "/htdocs/two"
}
$ doas mkdir -p /var/www/htdocs/one /var/www/htdocs/two
$ echo "backend one" | doas tee /var/www/htdocs/one/index.html
backend one
$ echo "backend two" | doas tee /var/www/htdocs/two/index.html
backend two
$ doas cp /tmp/httpd.conf /etc/httpd.conf
$ doas rcctl enable httpd
$ doas rcctl start httpd
httpd(ok)
$ ftp -V -o - http://127.0.0.2/
backend one
$ ftp -V -o - http://127.0.0.3/
backend two

The relayd configuration is in /etc/relayd.conf, and there is no file in a new system, only an example in /etc/examples. A relay needs three things. A table, which is a list of servers. A protocol, which says what to do with the traffic, and here only to pass it. And the relay itself: where to listen, which protocol to use, and where to forward to, with a mode and a check. Round robin takes the servers in turn, and the check asks every server for the page / and expects the code 200, so a broken server is found:

$ cat /tmp/relayd.conf
table <web> { 127.0.0.2 127.0.0.3 }

http protocol "plain" {
	pass
}

relay "www" {
	listen on 127.0.0.1 port 8080
	protocol "plain"
	forward to <web> port 80 mode roundrobin check http "/" code 200
}
$ doas relayd -n -f /tmp/relayd.conf
configuration OK

The -n option only checks the file, as it does for httpd and pf. Then install it, switch relayd on and look at the state. The checks run every ten seconds, and a host is unknown until the first one is done, so give it a moment:

$ doas cp /tmp/relayd.conf /etc/relayd.conf
$ doas rcctl enable relayd
$ doas rcctl start relayd
relayd(ok)
$ sleep 6
$ doas relayctl show summary
Id   Type   Name                     Avlblty Checks   Status
1    relay  www                                       active
1    table  web:80                                    active
1    host   127.0.0.2                100.00% 1/1      up
2    host   127.0.0.3                100.00% 1/1      up

The relay is active, the table is active, and both hosts are up, with one check made. Now send ten requests to the relay and count which server answered:

$ for i in 1 2 3 4 5 6 7 8 9 10; do ftp -V -o - http://127.0.0.1:8080/; done | sort | uniq -c
   6 backend one
   4 backend two

Both servers answered. The shares are close but not always equal, because relayd runs several relay processes and each one takes its own turns. The client never knows that there are two servers behind the address.

A load balancer is only worth having if it stops sending requests to a server that is down. To take a server out on purpose, for maintenance, tell relayd with relayctl. The host is disabled, the relay keeps running, and all the requests go to the other server:

$ doas relayctl host disable 127.0.0.3
command succeeded
$ doas relayctl show hosts
Id   Type   Name                     Avlblty Checks   Status
1    host   127.0.0.2                100.00% 1/1      up
2    host   127.0.0.3                                 disabled
$ for i in 1 2 3 4 5 6; do ftp -V -o - http://127.0.0.1:8080/; done | sort | uniq -c
   6 backend one
$ doas relayctl host enable 127.0.0.3
command succeeded

Now a real failure. Take the second site out of the httpd configuration, so that nothing answers on 127.0.0.3, and wait a little more than the check interval:

$ sed '/server "two"/,$d' /tmp/httpd.conf > /tmp/httpd1.conf
$ doas cp /tmp/httpd1.conf /etc/httpd.conf
$ doas rcctl reload httpd
httpd(ok)
$ sleep 12
$ doas relayctl show hosts
Id   Type   Name                     Avlblty Checks   Status
1    host   127.0.0.2                100.00% 3/3      up
2    host   127.0.0.3                0.00%   0/2      down
$ for i in 1 2 3 4; do ftp -V -o - http://127.0.0.1:8080/; done | sort | uniq -c
   4 backend one

Without anyone telling it, relayd found the host down, and the requests keep working, served by the first server. Put the second site back, and wait for the next check. The host comes up again by itself:

$ doas cp /tmp/httpd.conf /etc/httpd.conf
$ doas rcctl reload httpd
httpd(ok)
$ sleep 12
$ doas relayctl show hosts
Id   Type   Name                     Avlblty Checks   Status
1    host   127.0.0.2                100.00% 4/4      up
2    host   127.0.0.3                33.33%  1/3      up

The availability column keeps the history, it is the share of checks that passed, and so a host that was down for one check shows less than a hundred percent for a while.

In http mode relayd understands the requests, and it can change them on the way. The protocol is where the rules go. A backend sees every request from the address of the relay, so the real address of the client is lost, unless the relay puts it in a header. The usual header is X-Forwarded-For. A response header can be set on the way back, and a rule can block a request:

$ cat /tmp/relayd.conf
table <web> { 127.0.0.2 127.0.0.3 }

http protocol "web" {
	match request header set "X-Forwarded-For" value "$REMOTE_ADDR"
	match response header set "X-Served-By" value "relayd"
	block request path "/admin*"
	pass
}

relay "www" {
	listen on 127.0.0.1 port 8080
	protocol "web"
	forward to <web> port 80 mode roundrobin check http "/" code 200
}
$ doas relayd -n -f /tmp/relayd.conf
configuration OK
$ doas cp /tmp/relayd.conf /etc/relayd.conf
$ doas rcctl reload relayd
relayd(ok)
$ sleep 6

The sleep after a reload is needed. The hosts are unknown for the first few seconds, until the first check has run, and a request in that time gets no answer. There is no curl in the base system, so the test sends the request by hand with nc, to see the headers of the answer:

$ printf 'GET / HTTP/1.0\r\nHost: x\r\n\r\n' | nc -w 2 127.0.0.1 8080 | grep -E 'HTTP|X-Served|backend'
HTTP/1.0 200 OK
X-Served-By: relayd
backend one

The header was added. To see the request header arrive on the other side, ask httpd to log it. The log style forwarded writes the X-Forwarded-For value at the end of every line of the access log:

$ cat /tmp/httpd.conf
server "one" {
	listen on 127.0.0.2 port 80
	log style forwarded
	root "/htdocs/one"
}

server "two" {
	listen on 127.0.0.3 port 80
	log style forwarded
	root "/htdocs/two"
}
$ doas cp /tmp/httpd.conf /etc/httpd.conf
$ doas rcctl reload httpd
httpd(ok)
$ ftp -V -o /dev/null http://127.0.0.1:8080/
$ doas tail -n 1 /var/www/logs/access.log
two 127.0.0.3 - - [07/Nov/2026:01:21:00 +0000] "GET / HTTP/1.1" 200 12 "" "OpenBSD ftp" 127.0.0.1 -

The address near the end of the line is the client, as the relay saw it, and the - after it is the port header, which relayd does not set. Now the rule. There is a page /admin behind the relay, and the rule says to block it. Look at what happens:

$ echo secret | doas tee /var/www/htdocs/one/admin
secret
$ doas cp /var/www/htdocs/one/admin /var/www/htdocs/two/admin
$ printf 'GET /admin HTTP/1.0\r\nHost: x\r\n\r\n' | nc -w 2 127.0.0.1 8080 | grep -E 'HTTP|secret'
HTTP/1.0 200 OK
secret

The page was served. The rules in a protocol work like the rules in pf: the last one that matches decides, and the pass at the end matches everything. A block that must win needs the word quick, which stops the search at that rule:

$ sed -i 's/block request path/block request quick path/' /tmp/relayd.conf
$ grep block /tmp/relayd.conf
	block request quick path "/admin*"
$ doas relayd -n -f /tmp/relayd.conf
configuration OK
$ doas cp /tmp/relayd.conf /etc/relayd.conf
$ doas rcctl reload relayd
relayd(ok)
$ sleep 6
$ printf 'GET /admin HTTP/1.0\r\nHost: x\r\n\r\n' | nc -w 2 127.0.0.1 8080 | grep -E 'HTTP|secret'; echo "answer lines: $(printf 'GET /admin HTTP/1.0\r\nHost: x\r\n\r\n' | nc -w 2 127.0.0.1 8080 | wc -l | tr -d ' ')"
answer lines: 0
$ ftp -V -o - http://127.0.0.1:8080/
backend one

The blocked request gets no answer at all, the connection is closed, and the home page still works.

The servers behind the relay can speak plain HTTP, while the clients use HTTPS. The relay does the encryption. It looks for the certificate and the key by the address it listens on, /etc/ssl/ADDRESS.crt and /etc/ssl/private/ADDRESS.key, or by a name with the keypair option. For a test, make a self-signed one, as on the httpd page:

$ doas openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=127.0.0.1" -addext "subjectAltName=IP:127.0.0.1" -keyout /etc/ssl/private/127.0.0.1.key -out /etc/ssl/127.0.0.1.crt 2>/dev/null
$ doas chmod 600 /etc/ssl/private/127.0.0.1.key
$ doas ls -l /etc/ssl/127.0.0.1.crt /etc/ssl/private/127.0.0.1.key
-rw-r--r--  1 root  wheel  1013 Nov  7 01:21 /etc/ssl/127.0.0.1.crt
-rw-------  1 root  wheel  1708 Nov  7 01:21 /etc/ssl/private/127.0.0.1.key

A second relay is added to the file, with the same protocol and the same table, listening on another port with the word tls:

$ cat /tmp/relay2.conf

relay "secure" {
	listen on 127.0.0.1 port 8443 tls
	protocol "web"
	forward to <web> port 80 mode roundrobin check http "/" code 200
}
$ cat /tmp/relay2.conf >> /tmp/relayd.conf
$ doas relayd -n -f /tmp/relayd.conf
configuration OK
$ doas cp /tmp/relayd.conf /etc/relayd.conf
$ doas rcctl reload relayd
relayd(ok)
$ sleep 6
$ ftp -V -S cafile=/etc/ssl/127.0.0.1.crt -o - https://127.0.0.1:8443/
backend one
$ doas relayctl show relays
Id   Type   Name                     Avlblty Checks   Status
1    relay  www                                       active
2    relay  secure                                    active

The test trusts the certificate that was just made. The second relay answers with the same pages, from the same servers, and the servers know nothing about TLS. The rules of the protocol apply to both relays. On a real site the certificate would come from a certificate authority, and acme-client(1) in the base system can fetch it.

relayd -n finds mistakes before they reach the running daemon, and says which line. An address that is not on the machine:

$ cat /tmp/e3.conf
table <web> { 127.0.0.2 }
relay "www" {
	listen on 127.0.0.9 port 8080 tls
	forward to <web> port 80
}
$ doas relayd -n -f /tmp/e3.conf; echo "exit status $?"
/tmp/e3.conf:3: no local listen ip: 127.0.0.9
no actions, nothing to do
exit status 1

A protocol that nothing uses is only a warning. The file is still accepted:

$ cat /tmp/e1.conf
table <web> { 127.0.0.2 }
relay "www" {
	listen on 127.0.0.1 port 8080
	forward to <web> port 80 mode roundrobin
}
http protocol "x" {
	pass
}
$ doas relayd -n -f /tmp/e1.conf; echo "exit status $?"
unused protocol: x
configuration OK
exit status 0

A relay with a TLS port and no certificate also passes this test, because the certificates are only read when the daemon starts. So after a change, reload, and look at relayctl, not only at relayd -n:

$ doas relayctl show summary
Id   Type   Name                     Avlblty Checks   Status
1    relay  www                                       active
1    table  web:80                                    active
1    host   127.0.0.2                100.00% 1/1      up
2    host   127.0.0.3                100.00% 1/1      up
2    relay  secure                                    active
2    table  web:80                                    active
3    host   127.0.0.2                100.00% 1/1      up
4    host   127.0.0.3                100.00% 1/1      up

If a relay is missing from the list, or the hosts are down when they should be up, look at the log of the daemon, where relayd writes why it could not start the relay:

$ doas grep relayd /var/log/daemon | tail -n 3
Nov  7 01:21:12 obsd relayd[15653]: adding 2 hosts from table web:80
Nov  7 01:21:12 obsd relayd[79495]: adding 2 hosts from table web:80
Nov  7 01:21:12 obsd relayd[57773]: adding 2 hosts from table web:80

And when a rule seems to be ignored, remember the order: the last matching rule wins, and a block needs quick.