Installing a NetBSD based router on Compact Flash

There are several low power boards on the market which qualify for a Unix-based router. For example the Soekris Netxxxx series, the WRAP or ALIX boards from PC Engines, or most of the VIA EPIA products, when adding a second ethernet adapter. Also the Efika 5k2 from bPlan is a very interesting candidate. There are certainly a lot more.

Why not just buy a standard router with a proprietary OS? Those will be perfect for most of the people of course. But a BSD or Linux or OpenWRT based router gives you the possibility to control everything, when you are willed to invest the time. They can act as a full-grown server and allow you to log-in into your home LAN from whereever you are, for example.

In this example I will install NetBSD onto a Soekris Net4501. A Compact Flash card of 128 MB is sufficient for a base system. But the more the better, in case you want to install additional programs.

First of all, a Compact Flash card reader is highly recommended. You could theoretically install via PXEBoot/NFS, but I won't discuss that here. Also I assume a second NetBSD system, of any kind, to plug in the card reader and to connect to the Soekris console via nullmodem cable. It should be switched to 9600 bps, which is standard for the NetBSD boot loader (Net4501 defaults to 19200 bps). After that you can connect to the board as simply as typing tip dty0. The card reader will show up in kernel messages as:

umass0: Generic Card Reader, rev 2.00/93.21, addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, 3 luns per target
sd0 at scsibus0 target 0 lun 0:  disk removable
sd0: fabricating a geometry
sd0: 122 MB, 122 cyl, 64 head, 32 sec, 512 bytes/sect x 250880 sectors

This means there are 250880 sectors of 512 bytes on the card. We will create an image, filled with zeroes, of exactly that size.

# dd if=/dev/zero of=/tmp/arwen.img bs=512 count=250880

The image is attached to a vnode disk device, which allows us to treat it virtually as a real disk device. Using an image instead of writing to the card directly will give us some speed advantage during the installation. Writing to CF is very slow.

# vnconfig vnd0 /tmp/arwen.img

Install an active MBR partition, which extends over the whole disk.

# fdisk -ua0 vnd0
fdisk: primary partition table invalid, no magic in sector 0
Disk: /dev/rvnd0d
NetBSD disklabel disk geometry:
cylinders: 122, heads: 64, sectors/track: 32 (2048 sectors/cylinder)
total sectors: 250880

BIOS disk geometry:
cylinders: 122, heads: 64, sectors/track: 32 (2048 sectors/cylinder)
total sectors: 250880

Do you want to change our idea of what BIOS thinks? [n] 

Partition 0:

The data for partition 0 is:

sysid: [0..255 default: 169] 
start: [0..123cyl default: 32, 0cyl, 0MB] 
size: [0..122cyl default: 250848, 122cyl, 122MB] 
bootmenu: [] 
Do you want to change the active partition? [n] y
Choosing 4 will make no partition active.
active partition: [0..4 default: 0] 0
Are you happy with this choice? [n] y

We haven't written the MBR back to disk yet.  This is your last chance.
Partition table:
0: NetBSD (sysid 169)
    start 32, size 250848 (122 MB, Cyls 0-122/32/1), Active
1: 
2: 
3: 
Bootselector disabled.
Should we write new partition table? [n] y

Install a NetBSD diskabel and create an 'a' partition to hold the root file system. A 'b' partition for swap space is not recommended (who wants to swap to CF?). Partition 'd' will contain the whole disk, as usual for the i386 architecture (this is 'c' for others). Remember that the partition table is edited in vi mode.

# disklabel -e -I vnd0

# /dev/rvnd0d:
type: vnd
disk: vnd
label: fictitious
flags:
bytes/sector: 512
sectors/track: 32
tracks/cylinder: 64
sectors/cylinder: 2048
cylinders: 122
total sectors: 250880
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

5 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:    250848        32     4.2BSD      0     0     0  # (Cyl.      0*-    122*)
 c:    250848        32     unused      0     0        # (Cyl.      0*-    122*)
 d:    250880         0     unused      0     0        # (Cyl.      0 -    122*)

Now we can create the file system in the root partition, /dev/vnd0a. Option -B sets the byte-order to little-endian for Soekris, in case you make the image on a big-endian system. -m 0 sets the amount of reserved disk space to 0% and the -o option instructs the filesystem to minimize space fragmentation. You can omit the last two options when dealing with larger CF cards in the GB range.

# newfs -B le -m 0 -o space vnd0a

/dev/rvnd0a: 122.5MB (250848 sectors) block size 8192, fragment size 1024
        using 4 cylinder groups of 30.62MB, 3920 blks, 7552 inodes.
super-block backups (for fsck_ffs -b #) at:
     32,  62752, 125472, 188192,

Mount the file system and extract the base and etc sets of the selected NetBSD release. For larger cards you can add man and text.

# mount /dev/vnd0a /mnt
# cd /mnt
# tar xzfp /home/frank/i386sets_3.1/base.tgz
# tar xzfp /home/frank/i386sets_3.1/etc.tgz

For the Soekris, you would want to make a cusomized kernel, which has support for the special features of the board and is as small as possible to avoid wasting any byte of the valuable 64 MB RAM. There is already a NET4501 config file in src/sys/arch/i386/conf, which you may use as a start. NetBSD 4.x or higher is recommended to get access to the board's GPIO and watchdog timers.

[...]
#      link  ARWEN/netbsd
ld -T ../../../../arch/i386/conf/kern.ldscript -Ttext c0100000 -e start -X -o netbsd ${SYSTEM_OBJ} ${EXTRA_OBJ} vers.o
   text    data     bss     dec     hex filename
1310906   27684  181844 1520434  173332 netbsd

# cp /home/frank/netbsd/3.1/src/sys/arch/i386/compile/ARWEN/netbsd /mnt/

Copy the second level i386 boot loader to the image file system.

# cp /usr/mdec/boot /mnt/

Install the first level boot loader, with console set to serial at 9600 bps. Note that since NetBSD 4 you have to patch the binary with installboot -e -o console=com0,speed=9600 mybootxx_ffsv1 first, before installing it. /usr/mdec is only valid when working on an i386 system with the same release. Otherwise you have to take the boot loaders from src/obj/destdir.i386/usr/mdec in your source tree.

# installboot -v -o console=com0,speed=9600 /dev/rvnd0a /usr/mdec/bootxx_ffsv1
File system:         /dev/rvnd0a
File system type:    ffs (blocksize 8192, needswap 0)
Primary bootstrap:   /usr/mdec/bootxx_ffsv1
Preserving 51 (0x33) bytes of the BPB

Create the /etc/fstab. The root file system wd0a (the CF appears as standard IDE on the Soekris) has got the noatime attribute to prevent constant updates of the access-time in the file system. softdep for improved performance. Additionally we create a memory file system with a maximum of 32 MB, which holds all the dynamic data under /var.

/dev/wd0a       /       ffs     rw,noatime,softdep              1 1
varfs           /var    mfs     rw,nodev,nosuid,-s=32m          0 0

You might consider to make your root file system read-only, but I decided against it, because I still want to be able to transfer files to my router and to install new software. I did my best to limit write access to the CF medium to a minimum, and I'm confident that a modern card will survive that for a very long time.

An archive of the initial /var file system is stored in /etc and extracted on system startup. After having done that everything below /var can be deleted. Only the mount-point should persist.

# cd /mnt/var
# tar czf /mnt/etc/var.tar.gz .
# rm -rf *

Therefore we have to modify the /etc/rc.d/mountcritlocal start script to fill the memory file system after it has been mounted. The following diff file shows the differences to apply.

--- /tmp/mountcritlocal 2008-06-28 21:41:31.000000000 +0200
+++ /mnt/etc/rc.d/mountcritlocal        2008-06-28 21:42:23.000000000 +0200
@@ -19,6 +19,8 @@
        #       This usually includes /var.
        #
        mount_critical_filesystems local
+       echo "Constructing /var."
+       (cd /var && tar xzpf /etc/var.tar.gz)
 
        #       clean up left-over files.
        #       this could include the cleanup of lock files and /var/run, etc.

Tune some kernel settings in /etc/sysctl.conf to improve router's networking.

[...]
# Consider interface's MTU when calculating MSS
net.inet.tcp.mss_ifmtu=1

# Connect to hosts with bad setup
net.inet.tcp.rfc1323=0

It is recommended to configure /etc/namedb.conf and /etc/namedb/ to run the DNS service on your router. This allows all the hosts in your LAN to just use the router's IP as default gateway and DNS entry. Read the DNS chapter of the NetBSD guide for more information how to set up your name service.

/etc/resolv.conf defines my own domain, and a familiar one I want to automatically search as well. The nameserver is set to the IP of the router.

domain owl.de
search owl.de hasenbraten.de
nameserver 192.168.0.250

Don't forget your host name in /etc/hosts.

192.168.0.250           arwen.owl.de arwen

Enable IPFilter, IPNAT, name server, ifwatchd and SSH in /etc/rc.conf. Declare that we don't want swap space or kernel core dumps after a reboot. The first ethernet interface, sip0, is set up for the LAN. The second is pppoe0, which is configured via /etc/ifconfig.pppoe0.

[...]
hostname="arwen.owl.de"
auto_ifconfig=NO
net_interfaces="sip0 pppoe0"
ifconfig_sip0="inet 192.168.0.250 media auto up"
ipfilter=YES
ipnat=YES
ifwatchd=YES
named=YES
sshd=YES
no_swap=YES
savecore=NO

Create /etc/ifconfig.pppoe0 to configure your DSL connection via PPPOE. In the case of the Soekris Net4501 the second ethernet interface, called sip1, is attached to the pppoe0 pseudo device. sip0 was reserved for the LAN connection. You have to supply pppoectl with the authentication protocol (pap or chap), login-name and password.

create
! /sbin/ifconfig sip1 up
! /sbin/pppoectl -e sip1 $int
! /sbin/pppoectl $int myauthproto=pap myauthname=xxxxxxxxxxxxxxxxx#@t-online.de myauthsecret=xxxxxx hisauthproto=none
0.0.0.0 0.0.0.1 netmask 255.255.255.255 up

I'm also adding the following lines to /etc/daily.local to make sure that I disconnect at a defined time during the night. In Germany most providers force a disconnect after 24h at the latest. With this method I am sure that the disconnect won't happen during the day. Also useful to synchronize your clock with a time server (ntpdate).

#!/bin/sh
/usr/sbin/ntpdate -s ntp1.ptb.de ntp2.ptb.de
/bin/sleep 3
/sbin/ifconfig pppoe0 down
/bin/sleep 300
/sbin/pppoectl pppoe0 myauthproto=pap myauthname=xxxxxxxxxxxxxxxxx#@t-online.de myauthsecret=xxxxxx hisauthproto=none
/bin/sleep 3
/sbin/ifconfig pppoe0 0.0.0.0 0.0.0.1 netmask 255.255.255.255 up

Create ip-up and ip-down scripts in /etc/ppp (also create the directory when missing), which are called by the ifwatchd daemon whenever the PPP connection goes down or up. In the up case we can set the new default route and delete it again in the down case. Optionally ip-up may be used to refresh the IP for your host registered at DynDNS or similar services. I'm using InaDyn, which is, for example, available at Source Forge.

#!/bin/sh
/sbin/route add default $5
/usr/local/bin/inadyn-advanced -u username -p password -a myhost.dyndns.org --iterations 1

#!/bin/sh
/sbin/route delete default $5

For the IPFilter I use a restrictive configuration which blocks everything from extern except icmp and the tcp ports 22 (ssh), 80 (http) and 113 (ident). You may need others. Note that sip0 is the name of the ethernet interface connected to your LAN, in case you want to use the rules on another system. Copy the following to /etc/ipf.conf:

# block corrupt or dangerous packets
block in log quick from any to any with ipopts
block in log quick proto tcp from any to any with short
block in log quick from any to any with frag
block in log quick from any to any with opt lsrr
block in log quick from any to any with opt ssrr

# allow loopback
pass out quick on lo0 from any to any
pass in  quick on lo0 from any to any

# allow LAN
pass out quick on sip0 from any to any
pass in  quick on sip0 from any to any

# block packets from reserved addresses
block in log body quick on pppoe0 from 192.168.0.0/16 to any
block in log body quick on pppoe0 from 172.16.0.0/12 to any
block in log body quick on pppoe0 from 10.0.0.0/8 to any
block in log body quick on pppoe0 from 127.0.0.0/8 to any
block in log body quick on pppoe0 from 0.0.0.0/8 to any
block in log body quick on pppoe0 from 169.254.0.0/16 to any
block in log body quick on pppoe0 from 192.0.2.0/24 to any
block in log body quick on pppoe0 from 204.152.64.0/23 to any
block in log body quick on pppoe0 from 224.0.0.0/3 to any
block out log body quick on pppoe0 from any to 192.168.0.0/16
block out log body quick on pppoe0 from any to 172.16.0.0/12
block out log body quick on pppoe0 from any to 10.0.0.0/8
block out log body quick on pppoe0 from any to 127.0.0.0/8
block out log body quick on pppoe0 from any to 0.0.0.0/8
block out log body quick on pppoe0 from any to 169.254.0.0/16
block out log body quick on pppoe0 from any to 192.0.2.0/24
block out log body quick on pppoe0 from any to 204.152.64.0/23
block out log body quick on pppoe0 from any to 224.0.0.0/3

# allow ping, ident, http and ssh from extern
pass in log quick on pppoe0 proto icmp from any to any icmp-type echo keep state
pass in log quick on pppoe0 proto tcp from any to any port = 113 flags S keep state keep frags
pass in log quick on pppoe0 proto tcp from any to any port = 80 flags S keep state keep frags
pass in log quick on pppoe0 proto tcp from any to any port = 22 flags S keep state keep frags

# keep state of all connection from internal to external
pass out quick on pppoe0 proto tcp from any to any flags S keep state keep frags
pass out quick on pppoe0 proto udp from any to any keep state keep frags
pass out quick on pppoe0 proto icmp from any to any keep state keep frags

# block the rest
block in log quick from any to any
block out log quick from any to any

Create /etc/ipnat.conf for Network Address Translation, needed to allow multiple systems in your LAN to access the internet via the router. My LAN has the address 192.168.0, which you might want to change.

map pppoe0 192.168.0.0/24 -> 0/32 proxy port ftp ftp/tcp mssclamp 1412
map pppoe0 192.168.0.0/24 -> 0/32 portmap tcp/udp auto mssclamp 1412
map pppoe0 192.168.0.0/24 -> 0/32 mssclamp 1412

Set the time zone for your location.

# rm /mnt/etc/localtime
# ln -s /usr/share/zoneinfo/Europe/Berlin /mnt/etc/localtime

The /dev file system is automatically mounted as a memory file system by the init process, when it is empty. /dev/MAKEDEV init will be called to create the device nodes. This is exactly what we need, but unfortunately some important devices for IPNat and IPFilter are missing in the init case. So we have to modify the script. Additionally I like to have the pci* devices and some more pseudo terminals to log in.

--- /dev/MAKEDEV        2006-11-08 18:11:56.000000000 +0100
+++ /mnt/dev/MAKEDEV    2008-06-28 00:57:33.000000000 +0200
@@ -218,8 +218,9 @@
 case $i in
 
 # As of 2005-03-15, the "init" case must not create more than 1024 entries.
+# Modified for Soekris.
 init)
-       makedev std wscons wt0 fd0 fd1 wd0 wd1 wd2 wd3 sd0 sd1 sd2 sd3 sd4
+       makedev std wscons wt0 fd0 fd1 wd0 wd1 sd0 sd1
        makedev tty0 tty1
        makedev st0 st1 ch0 cd0 cd1 mcd0 vnd0
        makedev bpf
@@ -232,6 +233,8 @@
        makedev xbd0 xbd1 xencons
        makedev usbs
        makedev ipty
+       makedev ipl pf crypto systrace
+       makedev pci0 pci1
        makedev local
        ;;
 
@@ -960,8 +963,12 @@
 ipty)
        mkdev ttyp0 c 5 0 666
        mkdev ttyp1 c 5 1 666
+       mkdev ttyp2 c 5 2 666
+       mkdev ttyp3 c 5 3 666
        mkdev ptyp0 c 6 0 666
        mkdev ptyp1 c 6 1 666
+       mkdev ptyp2 c 6 2 666
+       mkdev ptyp3 c 6 3 666
        ;;
 
 ptm)

For NetBSD 4 we would also like to include gpio0 and sysmon for the watchdog timers.

--- /tmp/MAKEDEV        2008-06-29 12:21:59.000000000 +0200
+++ dev/MAKEDEV 2008-06-29 12:23:46.000000000 +0200
@@ -457,8 +457,7 @@
 # As of 2005-03-15, the "init" case must not create more than 1024 entries.
 init)
        makedev std wscons wt0 fd0 fd1
-       makedev wd0 wd1 wd2 wd3 wd4 wd5 wd6 wd7
-       makedev sd0 sd1 sd2 sd3 sd4
+       makedev wd0 wd1 sd0 sd1
        makedev tty0 tty1
        makedev st0 st1 ch0 cd0 cd1 mcd0 vnd0
        makedev bpf
@@ -471,6 +470,7 @@
        makedev xbd0 xbd1 xencons
        makedev usbs
        makedev ipty
+       makedev ipl pf crypto sysmon pci0 pci1 gpio0
        makedev local
        ;;
 
@@ -1289,8 +1289,12 @@
 ipty)
        mkdev ttyp0 c 5 0 666
        mkdev ttyp1 c 5 1 666
+       mkdev ttyp2 c 5 2 666
+       mkdev ttyp3 c 5 3 666
        mkdev ptyp0 c 6 0 666
        mkdev ptyp1 c 6 1 666
+       mkdev ptyp2 c 6 2 666
+       mkdev ptyp3 c 6 3 666
        ;;
 
 ptm)

We are done working on the image. Unmount the file system and detach our virtual disk.

# umount /mnt
# vnconfig -u vnd0

Write the image onto the Compact Flash card. This can take a while, especially with bigger cards. Remember to use /dev/sd0c when running on non-i386 hosts.

# dd if=arwen.img of=/dev/sd0d bs=4k

Now we can try to boot up our freshly installed NetBSD router for the first time and perform some final configurations directly on the live system. First action would be to set your root password with passwd and create a user:

# useradd -m username
# passwd username
# vipw

When installing packages from pkgsrc the default package database path of /var/db/pkg is a problem, because the contents of /var is volatile now. So we set an environment variable in the rc-script for all shells we want to use. In .shrc it would be setenv PKG_DBDIR /usr/pkg/db and in .bashrc is looks like export PKG_DBDIR=/usr/pkg/db. This makes sure that new packages are registered under /usr/pkg/db.

export PKG_DBDIR=/usr/pkg/db

To avoid writing of a history when using the more (or less) command, we have to add this to .bashrc:

export LESSHISTFILE=-

The same problem exists with the command line history of the Bash. So we also add:

HISTFILE=/dev/null
HISTFILESIZE=0
HISTSIZE=100

When the DNS service starts up, it will probably complain about a missing rndc.key. Go to /etc and create it with rndc-confgen -a -r keyboard. The last option is required because you have to provide the entropy by typing on your keyboard. Otherwise you can wait for a very long time for the command to finish.

# cd /etc
# rndc-confgen -a -r keyboard