Appreciate you posting the results of your testing here for others to use as a guide.
1 Like
For those of you keeping score at home… I am now attempting to combine the nfsroot and overlayfs… however, it seems there is an issue with systemd performing switch-root to an nfs mount. Working on it.
@Strit @Darksky Any thoughts on when systemd-247 will land in arm-stable?
Edit: Eh, never mind. I will just move forward with arm-testing.
Well, after spending far too much time and effort to get systemd to work with nfsroot, much to my surprise, it simply does not support it. Oh well, I learned stuff in the attempt.
I used pamac-manager to install raspberry-overlayroot-git from the AUR.
Reverted the systemd/udev change in /etc/mkinitcpio.conf and added overlayroot to HOOKS= and to the /boot/cmdline.txt file.
Rebuilt the initramfs and rebooted…
And it works, overlayfs on top of a read only nfsroot.
1 Like
Note: systemd really ought to be able to boot via nfs, but it must still be bugged. This has been buggy in the past and I assume it is once again.
I believe what currently happens is the Switch Root fails, presumably due to a bug in either systemd itself, or the systemd HOOK. It would seem systemd fails to mount the nfsroot as /sysroot.
See this for a “fix” for an issue with HOOKS=(net) which is required for nfsroot and the “missing” wifi device.
After sleeping several nights, the thought comes to mind that maybe why systemd is not performing the nfsroot mount, is because all of the required systemd files (rootfs mount unit) are not made a part of the initramfs. I may make another attempt to get this to work properly this weekend.
I have not forgotten about this project. In fact I have done more testing but without success. Well, I just stumbled across this concerning moving from mkinitcpio to dracut.
I believe dracut has implemented booting from nfs with systemd init.
So… @Darksky and/or @Strit is it reasonably possible for me to switch to dracut? Anyone try it? I would think it possible from a system perspective, even have them co-exist. But is there a some RPi4 or Manjaro config that makes this undesirable or incompatible?
I have no clue. I have not messed with it.
Well then, seems I have a new project.
That turned out to be surprisingly easy. I have not yet done all that will be required to make this a long term configuration change. I used the Arch wiki as a guide. I was not sure, so I made /etc/dracut.conf.d/cmdline.conf and added the contents of /boot/cmdline.txt. That is all I did before making the initramfs… and it booted.
There is much to investigate before adopting this configuration, but it is off to a good start.
Edit: The /etc/dracut.conf.d/cmdline.conf is not required. I removed it and no change, which is good, /boot/cmdline.txt continues to function as expected.
Also I was able to configure an overlayfs-root by simply creating /etc/module-load.d/overlay.conf and changing /boot/cmdline.txt. Looking good so far.
1 Like
Egads, this has been an interesting adventure. But finally, I have dracut creating an initramfs which will successfully mount a nfsroot, and then have systemd create and mount an overlayfs for the root filesystem.
Edit: This is what it looks like:
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.7G 0 1.7G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 749M 26M 724M 4% /run
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
overlay 936M 94M 843M 10% /
tmpfs 1.9G 40M 1.8G 3% /tmp
nfs.mydomain.com:/home 1.8T 1.4T 391G 78% /home
tmpfs 375M 52K 375M 1% /run/user/0
- Note: /boot is intentionally left unmounted and /home is a writable network mount.
No actual writing can take place on the root file system, but it is pseudo-writable. It appears as if it is writable, so applications run fine. But all files written to the root file system remain in memory and will not persist after a reboot. This keeps the root filesystem in a pristine state, which allows for multiple RPi to boot and share this single network file system.
So what is it good for?
Think of classrooms, offices, raspberry pi clusters, home networks, etc. All with a just single install to maintain.
The hack
To “disable” mkinitcpio and “activate” dracut, I chose to override the hooks which are installed during kernel updates. And then to utilize the linux-rpi4*.preset file to direct dracut, as it directs mkinitcpio.
Since I change kernels often, and have used linux-rpi4, linux-rpi4-mainline, and linux-rpi4-rc versions, I have three corresponding files placed in /etc/pacman.d/hooks/
$ cat /etc/pacman.d/hooks/90-linux-rpi4-mainline.hook
[Trigger]
Type = File
Operation = Install
Operation = Upgrade
Target = boot/kernel.img
Target = boot/kernel7.img
Target = boot/kernel8.img
Target = usr/lib/dracut/*
[Action]
Description = Updating linux-rpi4-mainline dracut...
When = PostTransaction
Exec = /usr/local/bin/dracut-install linux-rpi4-mainline
These files will override a very similar file installed with the kernel. Replacing calls to mkinitcpio with calls to the script below, which will call dracut.
$ cat /usr/local/bin/dracut-install
#!/usr/bin/bash
_file="/etc/mkinitcpio.d/${1}.preset"
if [[ -f ${_file} ]]; then
source ${_file}
if [[ -z "${ALL_kver}" ]] || [[ -z "${default_image}" ]]; then
echo "Error: ${0} failed to source dracut parameters, aborting"
exit 1
fi
else
echo " Skipping ${1} dracut, kernel not installed"
exit 0
fi
/usr/bin/dracut --quiet --force --no-hostonly --no-hostonly-cmdline --no-early-microcode --kver="${ALL_kver}" "${default_image}"
exit $?
So while it is a hack, simply deleting or renaming these same files from /etc/pacman.d/hooks/ “deactivates” dracut and “re-activates” mkinitcpio. This allows for both mkinitcpio and dracut to co-exist, while dracut will be sourcing the .preset file.
This produces an initramfs that is much larger (x5) than our current configuration of mkinitcpio produces. I am sure the size can be reduced, so that is now on my todo list.
It would appear that the minimal functional size will be approximately 31MB. The network and nfs modules accounting for most of the size increase. The maximum size appears to be around 48MB.
1 Like
How exactly did you do this?
I tried to explain how, guess I did not do so well… even after as it stands now, 17 edits of the posting. 
By placing files in /etc/pacman.d/hooks/ with the same name as the mkinitcpio hooks that are placed in /usr/share/libalpm/hooks/ when a new kernel is installed. Since I switch kernel branches, I need three files, other folks might only need one.
Without knowing totally what you have configured there on the outside looking in it appears the original hooks and .preset files placed by a new kernel install are still present. There are some instances that all hooks get run which means the last .img created in /boot/ created wins if the .img is created by mkinitcpio and dracut has the same name. You have the image created by dracut as “default” and as far as I can see from the wiki it might be the same name generated as mkinitcpio does.
If that is the case there and you run into this when some package upgrade calls for mkinitcpio to generate a new initramfs image on post install I would suggest changing the .img generated by dracut to something else in /boot and modify /boot/config.txt to point to the dracut .img. Say for example generate linux-dracut.img and in /boot/config.txt change this line to:
initramfs initramfs-dracut.img followkernelAh, yes. I did not know of a way to simulate this situation, so I left the issue to be resolved when it next occurred.
Excellent suggestion to make an alternately named initramfs… back to the drawing board.
In considering this more, I still have not made up my mind how I prefer this to work.
To address the the situation where the 90-mkinitcpio-install.hook is triggered, I could override that hook too. I am not aware of another hook that calls mkinitcpio. So that is a possibility, making this more or less and “replacement” initramfs setup.
Or I could do as you suggest and rather than a “replacement” setup, make a “supplement” setup. Instead of the override hooks, make a 95-dracut-install.hook that triggers on any change to initramfs, which would then build the alternate initramfs. The downside to this, is that I will endure initramfs builds, which I won’t use.
Edit: Nope, the 95- idea triggering on initramfs updates does not work. Evidently the trigger detection only occurs before the hooks are executed.
In looking at the 90-mkinitcpio-install.hook, it can trigger, but should it? I think the intent is for only the 90-linux-rpi4*.hook to trigger. The 90-mkinitcpio.hook will only trigger on a RPi upon an update to files in /usr/lib/initcpio/* and that is also a trigger for the 90-linux-rpi4*.hook.
I have wondered why some updates the initramfs build is triggered twice. Maybe this why?
Edit: To test this, I have made a override hook that performs nothing except for a bit of output, just to know if it was triggered.
Edit 2: I have confirmed, at least to my satisfaction, that the 90-mkinitcpio-install.hook does indeed run and it is redundant on an RPi4. Which then causing the initramfs to be built twice during some updates.
According to the arch alpm-hooks man page, the proper way to override this would be to create a symbolic link in /etc/pacman.d/hooks/90-mkinitcpio-install.hook → /dev/null
@Darksky do you concur with this assessment and resolution?
I think trying to maintain dracut and mkinitcpio at the same time can be problematic so what ever you get worked out that can work for you then your good.
Well, at least you did not advise that overriding the 90-mkinitcpio-install.hook would be a mistake. That it is required for some reason that I am not aware of. I will run with it and see if it bites me.
Edit: The use of dracut is only for the nfsroot image, so it can use the systemd overlay. It is superior for my needs to what can be done with busybox. I plan to stick with mkinitcpio for my other installs. I would always prefer to stick with the officially supported packages. I make enough trouble for myself without going an looking for it.