How to enforce boot time file system check?

openminded · 29 September 2020 09:12

This service is auto-run (enabled-runtime).

See? Check boot options:
cat /proc/cmdline
I bet you’ll see rw instead of ro there.

Termy · 29 September 2020 09:21

oh, my bad, i somehow didn’t read the first sentence of your post - sorry for that ^^
of course you are right.
When setting the boot options to ro - is it set back to rw after check or do you need to reboot again with rw?

openminded · 29 September 2020 10:03

No worries, that’s also my fault for speaking bad English Android text auto-correction also makes my sentences hard to read lol
I’m almost sure it’s re-mounted read-write to boot properly (if set so in fstab via defaults or explicit rw options). Usually cmdline’s ro only affects initrd stage where fsck is run.
Actually I’ve never seen my root filesystem mounted read-only.

dumpe2fs seems to not indicate a proper state of the last check when the filesystem is created without special flags, so you can do tune2fs -i 10 /dev/your/device to “force” a check every 10 days.

ananthp · 30 September 2020 19:01

@stargazer Thanks for the detailed research. Reading the archived forum discussions helped me understand my options better but didn’t provide a conclusive way. I was about to add fsck hook to mkinitcpio, because mounting / as read-only didn’t make sense initially.

It took me a while to realize that, even if I set ro for / in grub, it will be eventually mounted as writable. Thanks @openminded for pointing this out.

I tested this by editing the grub entry during boot, and mounting / with ro. Disk check happened and it can be verified by dumpe2fs output and systemd-fsck-root.service status.`

Being happy with the experiment, I decided to change the grub config permanently. This is proving trickier than I thought. I inspected my /etc/default/grub, looking for the rw kernel parameter to change to ro. GRUB_CMDLINE_LINUX_DEFAULT did not have that flag. It is hard-coded in /etc/grub.d/10_linux line 152, before suffixing other kernel params from default grub config. I suppose editing this file isn’t a good idea as it will be overwritten when updating grub.

I think this is a manjaro bug. As the distro enables systemd-fsck-root.service by default (it does, doesn’t it?), it makes no sense to fail the service by mounting root as rw.

openminded · 1 October 2020 01:16

If you add ro to GRUB_CMDLINE_LINUX_DEFAULT in grub config it will override rw set in etc/grub.d/10_linux I suppose. I am not using grub though that’s why I am not 100% sure.

ananthp · 1 October 2020 09:32

No, it doesn’t. GRUB_CMDLINE_LINUX_DEFAULT entries are appended after rw, so it reads as rw ro quiet .... I tried.

This has to be fixed in the distro itself. I’ll try to report it to the development team.

openminded · 1 October 2020 11:41

OK but when I was using grub, it helped me, at least I remember systemd-fsck-root.service reporting about successful checks… About a year ago. So I am writing based on my experience in the past.

Termy · 2 October 2020 09:17

So doing it at boottime (holding shift to get GRUB, then changing rw to ro) works
It didn’t really seem to delay boot by a noticable amount though, so it would be a viable option to default to that.
Bummer that it seems to be a “bug” in manjaro - did you already get it to the team @ananthp ?

ananthp · 2 October 2020 17:28

I had some trouble figuring out how manjaro’s development is organized. I couldn’t locate the repo/bug-tracker. See my query about that here.

Finally, filed an issue at manjaro’s grub repo. Let’s see how it goes.

ananthp · 21 October 2020 17:03

Ok. While this bug is still open, here’s a temporary fix for the problem:

Edit /etc/grub.d/10_linux. Change the following line (was line 152 in my setup)

	linux	${rel_dirname}/${basename} root=${linux_root_device_thisversion} rw ${args}

to

	linux	${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}

Optional: Disable quiet mode in grub. This will print log messages while booting, rather than showing blank screen. This is useful to see if scheduled disk checks happen while booting. Edit /etc/default/grub remove quiet from GRUB_CMDLINE_LINUX_DEFAULT line.
Regenerate grub config with

sudo grub-mkconfig -o /boot/grub/grub.cfg

Ensure systemd-fsck-root.service is enabled/active

systemctl status systemd-fsck-root.service

if not, start with sudo systemctl enable systemd-fsck-root.service.
5. Restart.

Keep an eye on sudo dumpe2fs -h <your-disk> to see if disk checks are happening periodically. Inspect Mount Count, Maximum Mount Count, Last Checked.

Note: If there’s an update to grub, all these changes may be overwritten. You’ll need to do these steps again.

Yochanan · 21 October 2020 18:10

That would be systemctl enable systemd-fsck-root.service.

If you start a service without enabling it and reboot, the service will not start on it’s own.

ananthp · 22 October 2020 17:42

@Yochanan Thanks. Updated the post.

sapo · 9 December 2020 11:57

Thank you, this worked for me!

Ste74 · 12 December 2020 10:49

Let me check… thank you to provide it

DAC324 · 11 January 2021 15:54

Looks like this is interesting as well: Systemd-fsck's contradiction: should I be worried?

With the configuration information provided here How to enforce boot time file system check?, I get my root filesystem checked. However, there is another disk in the system that does not appear to be checked.

Here is my /etc/fstab:

UUID=345b76c0-f339-4ca7-8582-440f229512a1 /              ext4    defaults,noatime,nodiratime 0 1
UUID=3acbe656-9d3d-4de3-926c-a839bb0e94b2 /home          ext4    defaults,noatime,nodiratime 0 2
UUID=a9353f2a-2cf9-4d4e-ada9-f72d8602045e none           swap    defaults 0 0

This is the output of journalctl -u systemd-fsck*:

-- Boot 7181084dd29542b8807cf15cbfc04b06 --
Jan 11 17:01:04 manjaro systemd[1]: Starting File System Check on /dev/disk/by-uuid/345b76c0-f339-4ca7-8582-440f229512a1...
Jan 11 17:01:04 manjaro systemd-fsck[178]: /dev/sda1: clean, 1325356/28745728 files, 29920520/114959872 blocks
Jan 11 17:01:04 manjaro systemd[1]: Finished File System Check on /dev/disk/by-uuid/345b76c0-f339-4ca7-8582-440f229512a1.

So it looks like the second disk is never checked.

openminded · 11 January 2021 18:37

man systemd-fsck@.service

DAC324 · 12 January 2021 08:04

Does not seem to help. As you can see from my /etc/fstab, passno is set to greater than zero as described in the man page:

UUID=345b76c0-f339-4ca7-8582-440f229512a1 /              ext4    defaults,noatime,nodiratime 0 1
UUID=3acbe656-9d3d-4de3-926c-a839bb0e94b2 /home          ext4    defaults,noatime,nodiratime 0 2
UUID=a9353f2a-2cf9-4d4e-ada9-f72d8602045e none           swap    defaults 0 0

Nevertheless, only the root filesystem on 345b76c0-f339-4ca7-8582-440f229512a1 is checked.

Even with fsck.mode=force set in the kernel options, I only get the root filesystem checked. Is there no way to have other attached filesystems checked as well?
Did I overlook something? Please let me know.

There is something else I noticed:

This renders the mount options for the root filesystem to appear twice in the kernel command line:

cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.10-x86_64 root=UUID=345b76c0-f339-4ca7-8582-440f229512a1 ro quiet acpi_enforce_resources=lax "acpi_osi=Windows 2009" loglevel=3 root=UUID=345b76c0-f339-4ca7-8582-440f229512a1 ro resume=UUID=a9353f2a-2cf9-4d4e-ada9-f72d8602045e vga=off nvidia-drm.modeset=1

DAC324 · 12 January 2021 18:48

This looks good as far as I can tell:

systemctl status systemd-fsck@* ● systemd-fsck@dev-disk-by\x2duuid-3acbe656\x2d9d3d\x2d4de3\x2d926c\x2da839bb0e94b2.service - File System Check on /dev/disk/by-uu> Loaded: loaded (/etc/systemd/system/systemd-fsck@.service; static) Active: active (exited) since Tue 2021-01-12 16:52:10 CET; 2h 46min ago Docs: man:systemd-fsck@.service(8) Process: 237 ExecStart=/usr/lib/systemd/systemd-fsck /dev/disk/by-uuid/3acbe656-9d3d-4de3-926c-a839bb0e94b2 (code=exited, stat> Main PID: 237 (code=exited, status=0/SUCCESS)

I checked, it’s there.

Here, only my root filesystem on /dev/sda1 appears:

-- Boot 9014e54e0e4d466d93a1430ae19df252 -- Jan 12 15:37:51 manjaro systemd-fsck[181]: /dev/sda1: clean, 1325399/28745728 files, 29948663/114959872 blocks -- Boot 132050fd8efe4cb793541f5e7cf9e82d -- Jan 12 16:52:09 manjaro systemd-fsck[181]: /dev/sda1: clean, 1325397/28745728 files, 29948665/114959872 blocks - that’s just why I was asking.
UPDATE: I just saw that Maximum mount count was set to -1 meaning that this filesystem is never checked. I now set it to be checked at least after a certain interval using tune2fs /dev/sdb1 -i 1m.

DAC324 · 3 February 2021 13:42

anon77807029:

[Make sure that] /etc/default/grub is sane and contains:

# Uncomment to ensure that the root filesystem is mounted read-only so that
# systemd-fsck can run the check automatically
GRUB_ROOT_FS_RO=true

Interestingly, in my /etc/default/grub, there is the following comment:

# Ensure that the root filesystem is mounted read-only so that systemd-fsck
# can run the check. This requires that /etc/grub.d/11_linux_root_fs is
# installed and enabled. 
GRUB_ROOT_FS_RO=true

There is no file /etc/grub.d/11_linux_root_fs on my system. Does this comment refer to that previous hack from How to enforce boot time file system check? - #24 by 10101000 ?

linux-aarhus · 9 February 2022 18:06