How to enforce boot time file system check?

My root file system doesn’t seem to be checked periodically. dumpe2fs shows this:

Mount count:              252
Maximum mount count:      20
Last checked:             Sat Feb 22 15:44:13 2020
Check interval:           604800 (1 week)
Next check after:         Sat Feb 29 15:44:13 2020

Last checked is several months ago. I’ve used tune2fs recently to enforce checks at every 20 mounts and once in every week. Even after a couple of reboots the file system doesn’t seem to be checked. The last column in /etc/fstab entry for this root file system is 1, so it doesn’t prevent the checks.

How do I enforce periodic fsck checks on this filesystem?

Wait … it isnt happening already?
And I suppose you mean a relatively ‘stock’ system without having made relevant changes?

I mean … I force my system to mount read-only and only check if there was a quick reboot or failure, etc.
But I configured that.

https://wiki.archlinux.org/index.php/Fsck

This is a great question and something someone should understand before a problem occurs :slight_smile: Systemd changed the game.

According to systemd/poettering, “Traditionally on Linux the root fs is fsck’ed while mounted. Only with systemd we introduced for the first time that the root fs is checked from the initrd before we mounted it instead of from the host after doing so. That is, as long as there’s an initrd at all. If there is not we continue to do the fsck after the transition, but only if the rootfs is actually read-only, see systemd-fsck-root.service about that, specifically the line ConditionPathIsReadWrite=!/ which conditions this check on / being mounted read-only.”
https://github.com/systemd/systemd/issues/14452#issuecomment-570163904

@cscs does archlinux, thus manjaro, have the fsck hook (#1), which does allow a fsck on the root file system and obeys the tune2fs options, or is fsck.mode=force still required and filesystem options are ignored?

Summary

Relevant Commands
journalctl -u systemd-fsck*
journalctl -t systemd-fsck
systemctl status systemd-fsck-root.service

Documentation
https://wiki.archlinux.org/index.php/Fsck#Mechanism
systemd-fsck@.service(8) - Linux manual page
systemd-fsck@.service(8) - Linux manual page

@cscs Yes, with relatively stock system without any changes, I assumed my file systems would be checked automatically from time to time. I understand there’s a fstab/kernel param to mount a filesystem in read-only mode in case of failure. But shouldn’t there be periodic checks to prevent such failures in first place?

@stargazer Thanks for the insights. I tried starting systemd-fsck-root.service, but it fails.

  Condition: start condition failed at Mon 2020-09-28 19:41:42 IST; 24min ago
             └─ ConditionPathIsReadWrite=!/ was not met

Normally, the root fs is mounted as read-write. I don’t understand when it will be mounted read-only for this condition to succeed for the check to happen. So should I manually tweek fstab or grub entry to mount it as read-only to enforce the check?

Adding fsck.mode=force temporarily to grub didn’t help either.

journalctl -u systemd-fsck is empty, with -t I see my other partition (a FAT partition left for UEFI?) seems to be checked at every boot.

I think usually you have two standard options:

  1. Change the fstab <options> entry for / to “ro” (“default” causes rw) and set <pass> to 1
  2. Keep “defaults” and therefore “rw” for / and set the “fsck” hook in /etc/mkinitcpio.conf

Both will ensure your / partition is checked periodically.

Unfortunately, the standard install ends up in “defaults” and missing “fsck” hook… (At least mine did.)

https://wiki.archlinux.org/index.php/Mkinitcpio

@ananthp I’m going to start with the ending first :slight_smile: Bottom-line: a fsck is not done automatically on the root file system on a Manjaro distribution running systemd.

I did a search in the archived forum and found some helpful information. Copy and paste the link/URL into your browser’s address bar to view the topic. You might want to look at the search results for more topics.

  • Search mkinitcpio hook fsck:
    https://archived.forum.manjaro.org/search?q=mkinitcpio%20hook%20fsck

  • Changes made to Grub-Quiet:
    https://archived.forum.manjaro.org/t/changes-made-to-grub-quiet/59925/1

  • Why is “fsck” hook not enabled by default?:
    https://archived.forum.manjaro.org/t/why-is-fsck-hook-not-enabled-by-default/148753

I think, in a systemd world, what will happen with a defective disk is that the system will boot, if it can, to the system target emergency or rescue, where you can shutdown and boot from a live disk or USB to examine, save what you can (hopefully you have backups), possibly “repair”.

Failures don’t normally happen all at once. You’ll notice hangs and strange behavior. You also might want to check smartctl in the Official Repositories.
https://www.smartmontools.org/


The output from systemctl status systemd-fsck-root.service is saying the service was not run because the root filesystem is not mounted as ro (!=Not /=root ro=ReadOnly). See below.


Following the archwiki mentioned by @cscs and @Wollie, option #2 is the default for Manjaro, but action must be taken to modify grub menu at boot (temporary) or the grub file (permanent).

Temporary
Edit the grub menu at boot time. Be very careful. When the menu appears type e.

  • Keys
    – E = edit
    – F10 = save/boot
    – ESC = discard
    – End key or Ctrl-E goes to the end of the line
    – arrow keys work

Scroll to find the line. The line may look like:

/boot/vmlinuxz-5.4-86_64 root=UUID=.... rw

Go to the end of the line and arrow back to change the rw to ro.

You might be thinking, where is this grub menu at boot :slight_smile: I believe, by default, the grub menu is not displayed (see link above). You could try the ESC key right away at boot to get the menu to appear.

One of the first things I changed in Manjaro (or any Linux distribution) is all the quiet boot stuff, because of the nature of my environment I need to see it, plus a couple seconds at boot verses not seeing it is well worth it.

Permanent
To run every time, edit the grub menu file. The links below will help (notice the file name and the update-grub command). I’m sure there are more details in a search at https://archived.forum.manjaro.org.

  • GRUB Menu Not Showing On Boot, Boots into Default Kernel Instead
    https://forum.manjaro.org/t/grub-menu-not-showing-on-boot-boots-into-default-kernel-instead/13410/2

  • Grub Tips and Tricks
    https://wiki.archlinux.org/index.php/GRUB/Tips_and_tricks


As far as Option #1 goes, I played with it on Archlinux first. #1 is the default on Archlinux. A user would need to run mkinitcpio which is a highly technical thing to do to get the hook on Manjaro, and it would have to be done for each kernel. It creates initramfs which grub loads from the initrd parameter in the grub file.

The sequence of events at boot: grub >initramfs >linux_kernel >systemd (init replacment)

  • See what hooks are already installed:
    less -g --pattern='^HOOKS=' /etc/mkinitcpio.conf

  • List available hooks:
    mkinitcpio --listhooks

  • List help for hooks:
    mkinitcpio --hookhelp fsck

At boot, I added fsck.mode=force rd.log=all. If I wanted this permanent I’d add it to the boot file.

3 Likes

So what do you advise? Adding the fsck hook or edit the Grub menu?

I checked and the last time systemd ran the fsck.service was when the filesystem was created.

I’m still reading and testing what works and how it works. There’s a lot of systemd doc and sometimes you have to read between the lines, visit multiple man pages, systemd git, and additional searches. Getting most of my info from reading archlinux and the archived Manjaro forum.

Start Here:

  • Why is “fsck” hook not enabled by default?
    `https://archived.forum.manjaro.org/t/why-is-fsck-hook-not-enabled-by-default/148753/5`

    • How to fsck the root filesystem? fsck.mode=force ?
      `https://archived.forum.manjaro.org/t/how-to-fsck-the-root-filesystem-fsck-mode-force/149466/10`

@Yochanan, Manjaro Team - 17 Jun

The fsck mkinitcpio hook was removed in favor of systemd-fsck-silent.

@mbod - 23 Jun

Yes you are right. It is checked every boot anyways.
But in out of the box Manjaro it is not checked at all because fsck and systemd hook are missing. They took out fsck because it is throwing messages during boot which breaks the silent boot concept. But they did not add systemd instead.

I am not a fan of “silent”, black boxes, or less info in these cases. I’ll try to go an informed user route myself :slight_smile: , that includes smartctl (been using it for years). A fast boot isn’t a priority for me on my desktop. Although I like the ArchLinux way better, Option #2 seems the less painful Manjaro way.

The second thread was created because of the first thread, but I included it separately above for everyone’s ease of use.

1 Like

To make fsck check root periodically one need to edit boot options passed to kernel, specifically change rw to ro. This should be enough for systemd to start systemd-fsck-root.service during boot, assuming there’s 1 in fstab for /
No fsck initcpio hook is needed.
I wrote this a number of times on old forum when we were discussing a transition to quiet boot, but it seems that for some reason the default is still rw.

The output of these commands on my laptop proves me right:

┬─[openm@reiwa:~]─[18:56:53]
╰─>$ journalctl -u systemd-fsck-root.service 
-- Logs begin at Sun 2020-09-27 21:28:31 +10, end at Tue 2020-09-29 18:56:58 +10. --
Sep 28 01:30:45 reiwa systemd[1]: systemd-fsck-root.service: Succeeded.
Sep 28 01:30:45 reiwa systemd[1]: Stopped File System Check on Root Device.
-- Reboot --
Sep 28 01:31:14 reiwa systemd-fsck[574]: MROOT: clean, 734795/3276800 files, 11078334/13107200 blocks
Sep 28 22:36:51 reiwa systemd[1]: systemd-fsck-root.service: Succeeded.
Sep 28 22:36:51 reiwa systemd[1]: Stopped File System Check on Root Device.
-- Reboot --
Sep 28 22:43:19 reiwa systemd-fsck[574]: MROOT: clean, 736156/3276800 files, 11070531/13107200 blocks
Sep 28 23:06:57 reiwa systemd[1]: systemd-fsck-root.service: Succeeded.
Sep 28 23:06:57 reiwa systemd[1]: Stopped File System Check on Root Device.
-- Reboot --
Sep 29 01:33:39 reiwa systemd[1]: Starting File System Check on /dev/reiwa/root...
Sep 29 01:33:39 reiwa systemd-fsck[1323]: MROOT: clean, 736193/3276800 files, 11111241/13107200 blocks
Sep 29 01:33:39 reiwa systemd[1]: Finished File System Check on /dev/reiwa/root.
┬─[openm@reiwa:~]─[18:56:58]
╰─>$ systemctl status systemd-fsck-root.service
● systemd-fsck-root.service - File System Check on Root Device
     Loaded: loaded (/usr/lib/systemd/system/systemd-fsck-root.service; enabled-runtime; vendor preset: disabled)
     Active: active (exited) since Tue 2020-09-29 01:33:39 +10; 17h ago
       Docs: man:systemd-fsck-root.service(8)
   Main PID: 1321 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 18942)
     Memory: 0B
     CGroup: /system.slice/systemd-fsck-root.service

Sep 29 01:33:39 reiwa systemd[1]: Starting File System Check on /dev/reiwa/root...
Sep 29 01:33:39 reiwa systemd-fsck[1323]: MROOT: clean, 736193/3276800 files, 11111241/13107200 blocks
Sep 29 01:33:39 reiwa systemd[1]: Finished File System Check on /dev/reiwa/root.
1 Like

if got 1 for / in fstab, but journalctl -t systemd-fsck shows me that my root is not checked and systemctl yields:

     Loaded: loaded (/usr/lib/systemd/system/systemd-fsck-root.service; enabled-runtime; vendor preset: disabled)
     Active: inactive (dead)
  Condition: start condition failed at Tue 2020-09-29 09:20:33 CEST; 1h 45min ago
       Docs: man:systemd-fsck-root.service(8)

Sep 29 09:20:32 Manjaro-ZZ systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
Sep 29 09:20:33 Manjaro-ZZ systemd[1]: Condition check resulted in File System Check on Root Device being skipped.

i’m not sure why or what to make of it? Can it be caused by having full disk encryption?

But as i’m pretty sure i didn’t manually enable this service, it should be on by default (at least a few years ago ^^)

This service is auto-run (enabled-runtime).

See? Check boot options:
cat /proc/cmdline
I bet you’ll see rw instead of ro there.

1 Like

oh, my bad, i somehow didn’t read the first sentence of your post - sorry for that ^^
of course you are right.
When setting the boot options to ro - is it set back to rw after check or do you need to reboot again with rw? :thinking:

1 Like

No worries, that’s also my fault for speaking bad English :sweat_smile: Android text auto-correction also makes my sentences hard to read lol
I’m almost sure it’s re-mounted read-write to boot properly (if set so in fstab via defaults or explicit rw options). Usually cmdline’s ro only affects initrd stage where fsck is run.
Actually I’ve never seen my root filesystem mounted read-only.

dumpe2fs seems to not indicate a proper state of the last check when the filesystem is created without special flags, so you can do tune2fs -i 10 /dev/your/device to “force” a check every 10 days.

1 Like

@stargazer Thanks for the detailed research. Reading the archived forum discussions helped me understand my options better but didn’t provide a conclusive way. I was about to add fsck hook to mkinitcpio, because mounting / as read-only didn’t make sense initially.

It took me a while to realize that, even if I set ro for / in grub, it will be eventually mounted as writable. Thanks @openminded for pointing this out.

I tested this by editing the grub entry during boot, and mounting / with ro. Disk check happened and it can be verified by dumpe2fs output and systemd-fsck-root.service status.`


Being happy with the experiment, I decided to change the grub config permanently. This is proving trickier than I thought. I inspected my /etc/default/grub, looking for the rw kernel parameter to change to ro. GRUB_CMDLINE_LINUX_DEFAULT did not have that flag. It is hard-coded in /etc/grub.d/10_linux line 152, before suffixing other kernel params from default grub config. I suppose editing this file isn’t a good idea as it will be overwritten when updating grub.

I think this is a manjaro bug. As the distro enables systemd-fsck-root.service by default (it does, doesn’t it?), it makes no sense to fail the service by mounting root as rw.

If you add ro to GRUB_CMDLINE_LINUX_DEFAULT in grub config it will override rw set in etc/grub.d/10_linux I suppose. I am not using grub though that’s why I am not 100% sure.

No, it doesn’t. GRUB_CMDLINE_LINUX_DEFAULT entries are appended after rw, so it reads as rw ro quiet .... I tried.

This has to be fixed in the distro itself. I’ll try to report it to the development team.

OK but when I was using grub, it helped me, at least I remember systemd-fsck-root.service reporting about successful checks… About a year ago. So I am writing based on my experience in the past.

So doing it at boottime (holding shift to get GRUB, then changing rw to ro) works :+1:
It didn’t really seem to delay boot by a noticable amount though, so it would be a viable option to default to that.
Bummer that it seems to be a “bug” in manjaro - did you already get it to the team @ananthp ?

I had some trouble figuring out how manjaro’s development is organized. I couldn’t locate the repo/bug-tracker. See my query about that here.

Finally, filed an issue at manjaro’s grub repo. Let’s see how it goes.

1 Like

Ok. While this bug is still open, here’s a temporary fix for the problem:

  1. Edit /etc/grub.d/10_linux. Change the following line (was line 152 in my setup)
	linux	${rel_dirname}/${basename} root=${linux_root_device_thisversion} rw ${args}

to

	linux	${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}
  1. Optional: Disable quiet mode in grub. This will print log messages while booting, rather than showing blank screen. This is useful to see if scheduled disk checks happen while booting. Edit /etc/default/grub remove quiet from GRUB_CMDLINE_LINUX_DEFAULT line.
  2. Regenerate grub config with
sudo grub-mkconfig -o /boot/grub/grub.cfg
  1. Ensure systemd-fsck-root.service is enabled/active
systemctl status systemd-fsck-root.service

if not, start with sudo systemctl enable systemd-fsck-root.service.
5. Restart.

Keep an eye on sudo dumpe2fs -h <your-disk> to see if disk checks are happening periodically. Inspect Mount Count, Maximum Mount Count, Last Checked.

Note: If there’s an update to grub, all these changes may be overwritten. You’ll need to do these steps again.

1 Like