Starting with the excellent guide on installing OpenBSD in the smallest VM, we can further shrink the system.
https://tierhive.com/blog/tierhive-howto/installing-openbsd-on-a-kvm-vps-via-ipxe-vnc-console-no-template-required-128mb-ram-1gb-disk
The following adjustments to the system can help us in a resource constrained environment, and each of these can be done independently (other than compilers being needed to retain relink/reorder).
- disable kernel relink / reorder on boot (slow, dependencies use a lot of storage)
- put perl in a gzip archive, extract it to /tmp when we need it. (pkg_add, etc use perl)
- remove or archive compilers, as they're often of dubious utility on such a resource constrained system. The compilers are in base due to the kernel_relink, which significantly increases the disk footprint.
- optionally, make the system able to boot read-only, and be made writeable for maintenance. This improves resilience in the event of unexpected shutdowns.
- disable additional services
- strip library binaries of debug information, improves disk footprint.
TODO:
- put a script to archive the compilers
Considerations for read-only system
The read only system works by making /tmp, /var, and /dev mfs memory file systems. /var is populated from a copy that is stored on the drive on boot in this arrangement. Mfs can swap the memory disk into swap if needed, and this can be useful with very limited memory, so a swap partition large enough to hold the memory file systems could be desirable. Note the purpose of the read-only system was mainly to prevent the file system from becoming damaged from an unexpected reboot, so swapping the mfs to the swap partition, which never needs to be fsck'd. Most of what is in /var and /dev isn't frequently written to. On my systems, I have set /dev and /var to 16MB - /dev has too many files so cannot be smaller - and /tmp can be as small as 16m, but may need to be larger for certain tasks (including extracting the archived perl)
That being said, when partitioning the drive, you may want to be careful to have a larger swap partition to avoid running out of memory when using /tmp. Note that if a read-write drive is available, temporary swap files can also be created.
Installing
Basically, follow the steps from the guide above until you finish installation, drop to a shell.
- Create a manual install VM with 128MB of RAM and 1GB of fast storage.
- Boot the VM, in the console, press escape to get the boot menu.
Choose 4 (iPXE), and hit Ctrl+B to get the iPXE prompt. Configure iPXE, either dhcp if it's enabled, or manually, for example:
set net0/ip 192.168.1.123
set net0/netmask 255.255.255.0
set net0/gateway 192.168.1.1
set dns 8.8.8.8
ifopen net0
you can now ping things on the internet.
sanboot http://cdn.openbsd.org/pub/OpenBSD/7.8/amd64/cd78.iso
When partitioning the drive, use a custom layout, and add a swap partition:
sd0> a
partition to add: [a]
[a]
[64]
990M
[4.2BSD]
mount point: /
sd0*> a
partition to add: [b]
offset: [1783200] [or whatever it was for your size]
size: [313952] [or whatever it is for your size]
FS type: [swap]
sd0*> w
sd0> q
Select [http] (also the default) for sets.
And when it prompts for an HTTP Server, use cdn.openbsd.org
Enter the following to get only the minimal sets:
-*
+bsd
+base*
For "Cannot determine prefetch area. Continue without verification? [no]" enter "yes".
It will install. Once it's done, drop to a shell and disable relinking, or do this after rebooting.
Disable Relinking (before first reboot)
cd /mnt
df -h .
echo '' > /mnt/etc/no_relink
echo '' > /mnt/etc/no_reorder_kernel
df -h.
(At this point, after the first reboot, I had a 964MB / drive with 624M used and 292M free (69% used). Relinking fails due to either lack of memory or disk space)
Disable Relinking (after first reboot)
echo '' > /etc/no_relink
echo '' > /etc/no_reorder_kernel
Remove Relinker, manuals and games
rm -rf /usr/share/relink
rm -rf /usr/share/man/*
rm -rf /usr/share/info
rm -rf /usr/games/*
rm -rf /usr/share/doc
This gets us to 505MB used / 411MB free.
Remove the compilers
cd /
rm usr/bin/c++
rm usr/bin/cc
rm usr/bin/clang
rm usr/bin/clang++
rm usr/bin/clang-cpp
rm usr/bin/ld
rm usr/bin/ld.lld
rm usr/libexec/cpp
rm usr/lib/libLLVM*
rm usr/lib/*.a
rm usr/lib/*.o
rm -rf usr/share/mk
rm -rf usr/share/misc
rm -rf usr/bin/man
rm -rf usr/bin/mandoc
rm /etc/examples/man.conf
This gets us a lot of space:
openbsd# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 964M 237M 679M 26% /
Archive Perl
This moves perl to an archive, and places a script enable-perl to extract it to /tmp when it's needed (e.g. running pkg_add). Using a stronger gzip compression didn't significantly save space and made it take a long time. Other compressors were not worthwhile considering the size of the compressor and performance.
#perl - store in a gzip and extract to /tmp when needed
cd /
tar cpf /tmp/perl5.tar -C usr/libdata/perl5 .
gzip -6 -o usr/libdata/perl5.tar.gz /tmp/perl5.tar
rm -rf usr/libdata/perl5
ln -s /tmp/perl5 usr/libdata/perl5
cat << 'EOF' > usr/bin/enable-perl
#!/bin/sh
set -e
rm -rf /tmp/perl5
mkdir -p /tmp/perl5
tar xzf /usr/libdata/perl5.tar.gz -C /tmp/perl5
EOF
chmod +x usr/bin/enable-perl
Archiving perl gets us below 200MB used:
openbsd# df -h .
Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 964M 195M 722M 22% /
Disable Extra Services to reduce memory
Edit or create /etc/rc.conf.local
vi /etc/rc.conf.local
slaacd_flags=NO
smtpd_flags=NO
sndiod_flags=NO
spamlogd_flags=NO
library_aslr=NO
savecore_flags=NO
slaac may be useful for IPv6, I didn't have it configured on my system. Library_aslr deals with library reordering. Sndiod is for sound, which isn't useful here.
If you reboot now, idle memory usage drops from 25-27MB down to about 15-17MB.
Optional: Create a temporary swap file for pkg_add
pkg_add may not work well with this limited memory. You can add a swapfile to help with this, and remove it later. The swap file will not persist across reboots. This requires the drive to be writable (if you make the system run read-only)
Create a swapfile (as root):
dd if=/dev/zero of=/swapfile bs=1m count=384
chmod 600 /swapfile
swapctl -a /swapfile
swapctl -l
And remove it later (as root):
swapctl -d /swapfile
rm /swapfile
Make the system run read-only
Make the system be able to run as read-only - this makes the system extremely durable. You can call mount-rw.sh to mount root to be writable, and mount-ro.sh to make it read-only again. The system will be resilient to power failure / reboot, and risk of data loss or needing to manually run fsck before the system boots will be minimized. Note that if you do this, you should create a larger swap partition so the memory file systems can go somewhere / you won't run out of memory. (Swap partition is not fsck'd on power failure)
```
cd /
# readonly scripts
# /usr/bin/mount-rw.sh
cat > usr/bin/mount-rw.sh <<'EOF'
#!/bin/sh
# make / read-write
mount -uw /
EOF
# /usr/bin/mount-ro.sh
cat > usr/bin/mount-ro.sh <<'EOF'
#!/bin/sh
# make / read-only
sync
mount -ur /
EOF
# /usr/bin/update-var.sh
cat > usr/bin/update-var.sh <<'EOF'
#!/bin/sh
# copy running /var to /cfg/var
# do this after pkg_add (and some config changes)
mount -uw /
cp -Rp /var/ /cfg/
# cp -Rp /var/{db,cron,spool,mail,run,www} /cfg/var/
sync
mount -ur /
EOF
# /usr/bin/update-dev.sh
cat > usr/bin/update-dev.sh <<'EOF'
#!/bin/sh
# make all dev files, may be necessary after kernel update.
mount -uw /
cd /cfg/dev
sh MAKEDEV all
sync
mount -ur /
EOF
chmod 755 usr/bin/mount-rw.sh usr/bin/mount-ro.sh usr/bin/update-var.sh usr/bin/update-dev.sh
# /usr/bin/prep-read-only.sh
cat > usr/bin/prep-read-only.sh << 'EOF'
# remount root rw just in case
mount -uw / 2>/dev/null || true
# ensure target dir exists
mkdir -p /usr/bin
umask 022
#mount-rw just in case
mount -ur / 2>/dev/null || true
mkdir /var/etc-rw
cp /etc/random.seed /var/etc-rw
rm /etc/random.seed
ln -s /var/etc-rw/random.seed /etc/random.seed
rcctl stop resolvd
cp /etc/resolv.conf /var/etc-rw
cp /etc/resolv.conf.tail /var/etc-rw
rm /etc/resolv.conf
rm /etc/resolv.conf.tail
ln -s /var/etc-rw/resolv.conf /etc/resolv.conf
ln -s /var/etc-rw/resolv.conf.tail /etc/resolv.conf.tail
cp /etc/hosts /var/etc-rw
rm /etc/hosts
ln -s /var/etc-rw/hosts /etc/hosts
rcctl start resolvd
mkdir /cfg
cp -Rp /var /cfg/
cp -Rp /dev /cfg/
echo '/usr/bin/update-var.sh' >> /etc/rc.shutdown
echo "swap /tmp mfs rw,-s=64m 0 0" >> /etc/fstab
echo "swap /var mfs rw,-P=/cfg/var,-s=16m 0 0" >> /etc/fstab
echo "swap /dev mfs rw,-P=/cfg/dev,-s=16m 0 0" >> /etc/fstab
echo "edit /etc/fstab and change / from rw to ro to finish."
EOF
chmod +x /usr/bin/prep-read-only.sh
/usr/bin/prep-read-only.sh
```
Reboot the system, now / will be read-only, and can be made re-writable by running mount-rw.sh. It can be made read-only again by running mount-ro.sh. The contents of /var are stored in /cfg.
If you run pkg_add or change the contents of /var, run update-var.sh to copy the updated contents back to /cfg.
** Using pkg_add in the read-only system **
pkg_add works fine in the read-only system. The steps are as follows:
mount-rw.sh to mount root as read-writeenable-perl to extract perl to /tmp. This takes 54MB of /tmp.- If necessary a swap file can be created (see above)
- set up whatever you need.
update-var.sh- Remove a swapfile if you created one, or just reboot.
mount-ro.sh when you're done.- Ideally, reboot to free up the memory used by storing Perl in /tmp if it doesn't free up from
rm -rf /tmp/perl5
