Excellent suggestion, thank you. I will look into this idea, it might be just what I am looking for.
@Strit Maybe you can guide me. I am looking into using overlayfs on boot, it appears the systemd-volatile-root.service is not included in the initramfs after switching udev to systemd in the HOOKS.
Do I add the missing systemd service file and binary in the mkinitcpio.conf? Or maybe I am missing some additional mkinitcpio hooks package?
Edit: I think this bug was addressed here
Attempting to examine the error is a bit tough since I can not get to a prompt.
However, if I attempt to run the unit file (which is probably not valid after booting) I see the following:
$ systemctl start /usr/lib/systemd.systemd-volatile-root
Failed to start usr-lib-systemd-systemd\x2dvolatile\x2droot.mount: Unit usr-lib-systemd-systemd\x2dvolatile\x2droot.mount not found
This file does not exist on my system.
Edit: This seems to be irrelevant, ignore it.
This works! 
MODULES=(overlay)
Edit: Now to backtrack and undo a few things that I tried to find what is required.
This is what I think is needed to get systemd to do the overlay work:
Install:
$ pacman -Syu mkinitcpio-systemd-tool
Then edit /etc/mkinitcpio.conf and modify your settings to include:
MODULES=(overlay)
BINARIES=(/usr/lib/systemd/systemd-volatile-root)
FILES=(/etc/os-release /usr/lib/os-release /usr/lib/systemd/system/systemd-volatile-root.service)
Edit HOOKS, by removing “udev” and replacing it with “systemd”:
HOOKS=(base systemd autodetect modconf block filesystems keyboard fsck)
Then change directory to /boot and run:
$ mkinitcpio -P
Finally modify your /boot/cmdline.txt by adding:
systemd.volatile=overlay
Now reboot and you should see something like the following for the root filesystem:
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 925600 71388 854212 8% /
I am currently unsure if changing “rw” to “ro” in the cmdline.txt file makes any difference.
Warning: Play with this on a new SD image. Easy to get stuck in a spot with a broken image.
3 Likes
Appreciate you posting the results of your testing here for others to use as a guide.
1 Like
For those of you keeping score at home… I am now attempting to combine the nfsroot and overlayfs… however, it seems there is an issue with systemd performing switch-root to an nfs mount. Working on it.
@Strit @Darksky Any thoughts on when systemd-247 will land in arm-stable?
Edit: Eh, never mind. I will just move forward with arm-testing.
Well, after spending far too much time and effort to get systemd to work with nfsroot, much to my surprise, it simply does not support it. Oh well, I learned stuff in the attempt.
I used pamac-manager to install raspberry-overlayroot-git from the AUR.
Reverted the systemd/udev change in /etc/mkinitcpio.conf and added overlayroot to HOOKS= and to the /boot/cmdline.txt file.
Rebuilt the initramfs and rebooted…
And it works, overlayfs on top of a read only nfsroot.
1 Like
Note: systemd really ought to be able to boot via nfs, but it must still be bugged. This has been buggy in the past and I assume it is once again.
I believe what currently happens is the Switch Root fails, presumably due to a bug in either systemd itself, or the systemd HOOK. It would seem systemd fails to mount the nfsroot as /sysroot.
See this for a “fix” for an issue with HOOKS=(net) which is required for nfsroot and the “missing” wifi device.
After sleeping several nights, the thought comes to mind that maybe why systemd is not performing the nfsroot mount, is because all of the required systemd files (rootfs mount unit) are not made a part of the initramfs. I may make another attempt to get this to work properly this weekend.
I have not forgotten about this project. In fact I have done more testing but without success. Well, I just stumbled across this concerning moving from mkinitcpio to dracut.
I believe dracut has implemented booting from nfs with systemd init.
So… @Darksky and/or @Strit is it reasonably possible for me to switch to dracut? Anyone try it? I would think it possible from a system perspective, even have them co-exist. But is there a some RPi4 or Manjaro config that makes this undesirable or incompatible?
I have no clue. I have not messed with it.
Well then, seems I have a new project.
That turned out to be surprisingly easy. I have not yet done all that will be required to make this a long term configuration change. I used the Arch wiki as a guide. I was not sure, so I made /etc/dracut.conf.d/cmdline.conf and added the contents of /boot/cmdline.txt. That is all I did before making the initramfs… and it booted.
There is much to investigate before adopting this configuration, but it is off to a good start.
Edit: The /etc/dracut.conf.d/cmdline.conf is not required. I removed it and no change, which is good, /boot/cmdline.txt continues to function as expected.
Also I was able to configure an overlayfs-root by simply creating /etc/module-load.d/overlay.conf and changing /boot/cmdline.txt. Looking good so far.
1 Like
Egads, this has been an interesting adventure. But finally, I have dracut creating an initramfs which will successfully mount a nfsroot, and then have systemd create and mount an overlayfs for the root filesystem.
Edit: This is what it looks like:
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.7G 0 1.7G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 749M 26M 724M 4% /run
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
overlay 936M 94M 843M 10% /
tmpfs 1.9G 40M 1.8G 3% /tmp
nfs.mydomain.com:/home 1.8T 1.4T 391G 78% /home
tmpfs 375M 52K 375M 1% /run/user/0
- Note: /boot is intentionally left unmounted and /home is a writable network mount.
No actual writing can take place on the root file system, but it is pseudo-writable. It appears as if it is writable, so applications run fine. But all files written to the root file system remain in memory and will not persist after a reboot. This keeps the root filesystem in a pristine state, which allows for multiple RPi to boot and share this single network file system.
So what is it good for?
Think of classrooms, offices, raspberry pi clusters, home networks, etc. All with a just single install to maintain.
The hack
To “disable” mkinitcpio and “activate” dracut, I chose to override the hooks which are installed during kernel updates. And then to utilize the linux-rpi4*.preset file to direct dracut, as it directs mkinitcpio.
Since I change kernels often, and have used linux-rpi4, linux-rpi4-mainline, and linux-rpi4-rc versions, I have three corresponding files placed in /etc/pacman.d/hooks/
$ cat /etc/pacman.d/hooks/90-linux-rpi4-mainline.hook
[Trigger]
Type = File
Operation = Install
Operation = Upgrade
Target = boot/kernel.img
Target = boot/kernel7.img
Target = boot/kernel8.img
Target = usr/lib/dracut/*
[Action]
Description = Updating linux-rpi4-mainline dracut...
When = PostTransaction
Exec = /usr/local/bin/dracut-install linux-rpi4-mainline
These files will override a very similar file installed with the kernel. Replacing calls to mkinitcpio with calls to the script below, which will call dracut.
$ cat /usr/local/bin/dracut-install
#!/usr/bin/bash
_file="/etc/mkinitcpio.d/${1}.preset"
if [[ -f ${_file} ]]; then
source ${_file}
if [[ -z "${ALL_kver}" ]] || [[ -z "${default_image}" ]]; then
echo "Error: ${0} failed to source dracut parameters, aborting"
exit 1
fi
else
echo " Skipping ${1} dracut, kernel not installed"
exit 0
fi
/usr/bin/dracut --quiet --force --no-hostonly --no-hostonly-cmdline --no-early-microcode --kver="${ALL_kver}" "${default_image}"
exit $?
So while it is a hack, simply deleting or renaming these same files from /etc/pacman.d/hooks/ “deactivates” dracut and “re-activates” mkinitcpio. This allows for both mkinitcpio and dracut to co-exist, while dracut will be sourcing the .preset file.
This produces an initramfs that is much larger (x5) than our current configuration of mkinitcpio produces. I am sure the size can be reduced, so that is now on my todo list.
It would appear that the minimal functional size will be approximately 31MB. The network and nfs modules accounting for most of the size increase. The maximum size appears to be around 48MB.
1 Like
How exactly did you do this?
I tried to explain how, guess I did not do so well… even after as it stands now, 17 edits of the posting.
By placing files in /etc/pacman.d/hooks/ with the same name as the mkinitcpio hooks that are placed in /usr/share/libalpm/hooks/ when a new kernel is installed. Since I switch kernel branches, I need three files, other folks might only need one.