How to enforce boot time file system check?

That would be systemctl enable systemd-fsck-root.service. :wink:

If you start a service without enabling it and reboot, the service will not start on it’s own.

@Yochanan Thanks. Updated the post.

Unfortunately the ticket created by @ananthp has been closed by @schinfo without resolution. I was hoping for a better solution (perhaps user configurable?). Manual edit of /etc/grub.d/10_linux after every update to the grub package has become exhausting. I’ve had to resort to this ugly hack and figured I’d post it since there doesn’t seem to be an official resolution in sight.

EDIT: the original version posted can prevent subsequent grub.d execution like 30_os-prober & 60_memtest86+. It must be run last. I’ve updated the filename to prefix 99.

One caveat is that it only works when using update-grub as we have to guess the output file since grub-mkconfig doesn’t export ${grub_cfg}.

  1. Create /etc/grub.d/99_linux_root_fs and add the following:
#! /bin/sh
set -e

. /etc/default/grub

FILE_GRUB_CFG="`awk '/^.*grub-mkconfig\s+-o\s+(.+)$/{ print $3 }' /usr/bin/update-grub`.new"

if [ "x${GRUB_ROOT_FS_RO}" = "xtrue" ]; then
  if test -n "${FILE_GRUB_CFG}" && test -e "${FILE_GRUB_CFG}"; then
    sed -e 's/^\(\s*linux.\+root\s*=.\+\s\)rw\(\s\+.*\)$/\1ro\2/i' -i ${FILE_GRUB_CFG}
  fi
fi

echo "### END ${0} ###" >> ${FILE_GRUB_CFG}
  1. Ensure that /etc/grub.d/99_linux_root_fs is executable:
chmod +x /etc/grub.d/99_linux_root_fs
  1. Add this to /etc/default/grub:
# Ensure that the root filesystem is mounted read-only so that systemd-fsck
# can run the check. This requires that /etc/grub.d/11_linux_root_fs is
# installed and enabled. 
GRUB_ROOT_FS_RO=true
  1. Run update-grub

Good luck.

1 Like

Thank you, this worked for me!

1 Like

I’m glad it worked for you. I’ve spent a bit of time to update the package to support user control of the root mount mode. Personally I’d prefer to keep the mkinitcpio fsck hook disabled in favor of this method for a couple of reasons:

  1. There doesn’t seem to be any logging of fsck using the hook (unless I’m missing something?), this is a problem if you miss the boot output.
  2. I’m not concerned with the minor increase in time required to remount as rw.

It would be nice if we could create merge/pull requests…

@Ste74 Would you be willing to review this and possibly implement? I’m able to build the package and GRUB_ROOT_FS_RO funcitons as expected on my system. Here is the git diff from my local branch:

diff --git a/0001-grub-maybe_quiet.patch b/0001-grub-maybe_quiet.patch
index 6a9fbb3..05c558e 100644
--- a/0001-grub-maybe_quiet.patch
+++ b/0001-grub-maybe_quiet.patch
@@ -353,7 +353,7 @@ diff --git a/util/grub.d/10_linux.in b/util/grub.d/10_linux.in
 +EOF
 +  fi
 +    sed "s/^/$submenu_indentation/" << EOF
-       linux   ${rel_dirname}/${basename} root=${linux_root_device_thisversion} rw ${args}
+       linux   ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ${grub_root_fs_mode} ${args}
  EOF
    if test -n "${initrd}" ; then
 @@ -153,8 +158,12 @@ EOF
diff --git a/PKGBUILD b/PKGBUILD
index 710c917..828f9a6 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -28,7 +28,7 @@ _build_platforms="i386-pc ${_target_arch}-efi"
 
 pkgname="grub"
 pkgver=2.04
-pkgrel=12
+pkgrel=13
 pkgdesc="GNU GRand Unified Bootloader (2)"
 arch=('x86_64' 'i686')
 url="https://www.gnu.org/software/grub/"
@@ -75,13 +75,13 @@ sha256sums=('SKIP'
             'SKIP'
             '63c611189a60d68c6ae094f2ced91ac576b3921b7fd2e75a551c2dc6baefc35e'
             'a5198267ceb04dceb6d2ea7800281a42b3f91fd02da55d2cc9ea20d47273ca29'
-            'cf00c96aee37e0a73c1ab6ed6ccfe74fa2b2859f55cd315a4caa6c880ce7aeba'
+            '2f4f0715dceddffbed55c57c51b6f8d91c59b6b313dc495f165d4847f77cbd74'
             '20b2b6e7f501596b5cce6ffa05906980427f760c03d308d0e045cf2ecf47bb0e'
-            '9a0ef2efe572f3e206d8f145cb9a00098f44d41eaf396110810f6f79885bd5de'
+            '4a0a90ae29c97b395c0610f6d78f3d39d57a7a4b41647ace9bcce4b864e19497'
             '39d7843dfe1e10ead912a81be370813b8621794a7967b3cc5e4d4188b5bf7264'
             'd222ea6e676268f44abdc2c92e9e4a6265646525ace108b828f4eb01cf20f8dd'
             '01264c247283b7bbdef65d7646541c022440ddaf54f8eaf5aeb3a02eb98b4dd8'
-            'f1b226fe671665305e21e06dc1744b94f734e397697e2e9915bd95fbcd0c6ef5'
+            '4bb14ddf6c41e70b34b03f2d3e10a01842679a7ed18900f639b10ff7a82816ce'
             '7fc95d49c0febe98a76e56b606a280565cb736580adecf163bc6b5aca8e7cbd8'
             '467b0101154076fee99d9574a5fb6b772a3923cc200a1f4ca08fe17be8d68111'
             '2eb199f510340cf8d190ba2fa80d5bdcf1e2e7ca53e8011af2ee62ea3b8dd03b'
diff --git a/grub-manjaro-modifications.patch b/grub-manjaro-modifications.patch
index 39ccb7f..36313a8 100644
--- a/grub-manjaro-modifications.patch
+++ b/grub-manjaro-modifications.patch
@@ -2,17 +2,35 @@ From 009ecc71a0b397faed8fafb3c4d62e96d0d86a8a Mon Sep 17 00:00:00 2001
 From: Keshav Padram Amburay <address@hidden>
 Modified for Manjaro by: Philip Müller <address@hidden>
 Date: Wed, 6 Nov 2013 21:49:40 +0530
+
+Additional modification for GRUB_ROOT_FS_MODE by: 10101000 <address@hidden>
+Date: Fri, 11 Dec 2020 14:03:57 -0700
+
 Subject: [PATCH] Add Arch Linux specific grub-mkconfig fixes
 
 Patch modified based on ideas from Felix aka fstirlitz,
 given at https://bugs.archlinux.org/task/37904?getfile=11257
 ---
 
+diff --git a/util/grub-mkconfig.in b/util/grub-mkconfig.in
+index d3e879b8e..4a56a5958 100644
+--- a/util/grub-mkconfig.in
++++ b/util/grub-mkconfig.in
+@@ -248,7 +248,8 @@ export GRUB_DEFAULT \
+   GRUB_ENABLE_CRYPTODISK \
+   GRUB_BADRAM \
+   GRUB_OS_PROBER_SKIP_LIST \
+-  GRUB_DISABLE_SUBMENU
++  GRUB_DISABLE_SUBMENU \
++  GRUB_ROOT_FS_RO
+ 
+ if test "x${grub_cfg}" != "x"; then
+   rm -f "${grub_cfg}.new"
 diff --git a/util/grub-mkconfig_lib.in b/util/grub-mkconfig_lib.in
-index a9cf7fc..7eb8950 100644
+index 301d1ac22..803dddb4e 100644
 --- a/util/grub-mkconfig_lib.in
 +++ b/util/grub-mkconfig_lib.in
-@@ -248,6 +248,9 @@ version_test_gt ()
+@@ -253,6 +253,9 @@ version_test_gt ()
      *.old:*.old) ;;
      *.old:*) version_test_gt_a="`echo "$version_test_gt_a" | sed -e 's/\.old$//'`" ; version_test_gt_cmp=gt ;;
      *:*.old) version_test_gt_b="`echo "$version_test_gt_b" | sed -e 's/\.old$//'`" ; version_test_gt_cmp=ge ;;
@@ -23,7 +41,7 @@ index a9cf7fc..7eb8950 100644
    version_test_numeric "$version_test_gt_a" "$version_test_gt_cmp" "$version_test_gt_b"
    return "$?"
 diff --git a/util/grub.d/10_linux.in b/util/grub.d/10_linux.in
-index 00d1931..f403585 100644
+index e8b01c0d0..c3dd34a59 100644
 --- a/util/grub.d/10_linux.in
 +++ b/util/grub.d/10_linux.in
 @@ -29,9 +29,9 @@ export TEXTDOMAINDIR="@localedir@"
@@ -38,7 +56,7 @@ index 00d1931..f403585 100644
    CLASS="--class $(echo ${GRUB_DISTRIBUTOR} | tr 'A-Z' 'a-z' | cut -d' ' -f1|LC_ALL=C sed 's,[^[:alnum:]_],_,g') ${CLASS}"
  fi
  
-@@ -82,9 +82,11 @@ linux_entry ()
+@@ -92,9 +92,11 @@ linux_entry ()
    if [ x$type != xsimple ] ; then
        case $type in
          recovery)
@@ -52,7 +70,7 @@ index 00d1931..f403585 100644
        esac
        if [ x"$title" = x"$GRUB_ACTUAL_DEFAULT" ] || [ x"Previous Linux versions>$title" = x"$GRUB_ACTUAL_DEFAULT" ]; then
          replacement_title="$(echo "Advanced options for ${OS}" | sed 's,>,>>,g')>$(echo "$title" | sed 's,>,>>,g')"
-@@ -96,7 +98,7 @@ linux_entry ()
+@@ -106,7 +108,7 @@ linux_entry ()
    else
        echo "menuentry '$(echo "$os" | grub_quote)' ${CLASS} \$menuentry_id_option 'gnulinux-simple-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
    fi      
@@ -61,21 +79,27 @@ index 00d1931..f403585 100644
        save_default_entry | grub_add_tab
    fi
  
-@@ -128,10 +130,11 @@ linux_entry ()
+@@ -138,10 +140,17 @@ linux_entry ()
      fi
      printf '%s\n' "${prepare_boot_cache}" | sed "s/^/$submenu_indentation/"
    fi
 -  message="$(gettext_printf "Loading Linux %s ..." ${version})"
 +
++  if [ "x${GRUB_ROOT_FS_RO}" = "xtrue" ]; then
++    grub_root_fs_mode=ro
++  else
++    grub_root_fs_mode=rw
++  fi
++
 +  message="$(gettext_printf "Loading Linux %s ..." "${version}")"
    sed "s/^/$submenu_indentation/" << EOF
        echo    '$(echo "$message" | grub_quote)'
 -      linux   ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}
-+      linux   ${rel_dirname}/${basename} root=${linux_root_device_thisversion} rw ${args}
++      linux   ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ${grub_root_fs_mode} ${args}
  EOF
    if test -n "${initrd}" ; then
      # TRANSLATORS: ramdisk isn't identifier. Should be translated.
-@@ -192,6 +195,25 @@ while [ "x$list" != "x" ] ; do
+@@ -202,6 +211,25 @@ while [ "x$list" != "x" ] ; do
    alt_version=`echo $version | sed -e "s,\.old$,,g"`
    linux_root_device_thisversion="${LINUX_ROOT_DEVICE}"
  
@@ -101,7 +125,7 @@ index 00d1931..f403585 100644
    initrd_early=
    for i in ${GRUB_EARLY_INITRD_LINUX_STOCK} \
           ${GRUB_EARLY_INITRD_LINUX_CUSTOM}; do
-@@ -201,6 +223,7 @@ while [ "x$list" != "x" ] ; do
+@@ -211,6 +239,7 @@ while [ "x$list" != "x" ] ; do
    done
  
    initrd_real=
@@ -109,7 +133,7 @@ index 00d1931..f403585 100644
    for i in "initrd.img-${version}" "initrd-${version}.img" "initrd-${version}.gz" \
           "initrd-${version}" "initramfs-${version}.img" \
           "initrd.img-${alt_version}" "initrd-${alt_version}.img" \
-@@ -208,7 +231,8 @@ while [ "x$list" != "x" ] ; do
+@@ -218,7 +247,8 @@ while [ "x$list" != "x" ] ; do
           "initramfs-genkernel-${version}" \
           "initramfs-genkernel-${alt_version}" \
           "initramfs-genkernel-${GENKERNEL_ARCH}-${version}" \
@@ -119,7 +143,7 @@ index 00d1931..f403585 100644
      if test -e "${dirname}/${i}" ; then
        initrd_real="${i}"
        break
-@@ -261,6 +285,17 @@ while [ "x$list" != "x" ] ; do
+@@ -285,6 +315,17 @@ while [ "x$list" != "x" ] ; do
  
    linux_entry "${OS}" "${version}" advanced \
                "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"
@@ -137,8 +161,10 @@ index 00d1931..f403585 100644
    if [ "x${GRUB_DISABLE_RECOVERY}" != "xtrue" ]; then
      linux_entry "${OS}" "${version}" recovery \
                  "single ${GRUB_CMDLINE_LINUX}"
---- a/util/grub.d/30_os-prober.in      2018-03-18 13:11:56.882154575 +0100
-+++ b/util/grub.d/30_os-prober.in      2018-03-18 13:13:23.162726852 +0100
+diff --git a/util/grub.d/30_os-prober.in b/util/grub.d/30_os-prober.in
+index 1b91c102f..1ecb80226 100644
+--- a/util/grub.d/30_os-prober.in
++++ b/util/grub.d/30_os-prober.in
 @@ -207,7 +207,7 @@ EOF
          LBOOT="`echo ${LINUX} | cut -d ':' -f 2`"
          LLABEL="`echo ${LINUX} | cut -d ':' -f 3 | tr '^' ' '`"
diff --git a/grub.default b/grub.default
index 29797c7..60dc117 100755
--- a/grub.default
+++ b/grub.default
@@ -47,3 +47,7 @@ GRUB_COLOR_HIGHLIGHT="green/black"
 
 # Uncomment to get a beep at GRUB start
 #GRUB_INIT_TUNE="480 440 1"
+
+# Uncomment to ensure that the root filesystem is mounted read-only so that
+# systemd-fsck can run the check automatically
+#GRUB_ROOT_FS_RO=true

Let me check… thank you to provide it

1 Like

Thank you very much, I see the changes in gitlab and the package update in the unstable repo. Looks good, works as expected.

Looks like this is interesting as well: Systemd-fsck's contradiction: should I be worried?

With the configuration information provided here How to enforce boot time file system check?, I get my root filesystem checked. However, there is another disk in the system that does not appear to be checked.

Here is my /etc/fstab:

UUID=345b76c0-f339-4ca7-8582-440f229512a1 /              ext4    defaults,noatime,nodiratime 0 1
UUID=3acbe656-9d3d-4de3-926c-a839bb0e94b2 /home          ext4    defaults,noatime,nodiratime 0 2
UUID=a9353f2a-2cf9-4d4e-ada9-f72d8602045e none           swap    defaults 0 0 

This is the output of journalctl -u systemd-fsck*:

-- Boot 7181084dd29542b8807cf15cbfc04b06 --
Jan 11 17:01:04 manjaro systemd[1]: Starting File System Check on /dev/disk/by-uuid/345b76c0-f339-4ca7-8582-440f229512a1...
Jan 11 17:01:04 manjaro systemd-fsck[178]: /dev/sda1: clean, 1325356/28745728 files, 29920520/114959872 blocks
Jan 11 17:01:04 manjaro systemd[1]: Finished File System Check on /dev/disk/by-uuid/345b76c0-f339-4ca7-8582-440f229512a1.

So it looks like the second disk is never checked.

man systemd-fsck@.service

Does not seem to help. As you can see from my /etc/fstab, passno is set to greater than zero as described in the man page:

UUID=345b76c0-f339-4ca7-8582-440f229512a1 /              ext4    defaults,noatime,nodiratime 0 1
UUID=3acbe656-9d3d-4de3-926c-a839bb0e94b2 /home          ext4    defaults,noatime,nodiratime 0 2
UUID=a9353f2a-2cf9-4d4e-ada9-f72d8602045e none           swap    defaults 0 0 

Nevertheless, only the root filesystem on 345b76c0-f339-4ca7-8582-440f229512a1 is checked.

Even with fsck.mode=force set in the kernel options, I only get the root filesystem checked. Is there no way to have other attached filesystems checked as well?
Did I overlook something? Please let me know.

There is something else I noticed:

This renders the mount options for the root filesystem to appear twice in the kernel command line:

cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.10-x86_64 root=UUID=345b76c0-f339-4ca7-8582-440f229512a1 ro quiet acpi_enforce_resources=lax "acpi_osi=Windows 2009" loglevel=3 root=UUID=345b76c0-f339-4ca7-8582-440f229512a1 ro resume=UUID=a9353f2a-2cf9-4d4e-ada9-f72d8602045e vga=off nvidia-drm.modeset=1

Did you read the entire thread? According to Manjaro - Branch Compare the latest version of grub in all branches is 2.04-13. If you’re on this version, there is no need for the ugly /etc/grub.d/99_linux_root_fs workaround as my changes described in this comment have been implemented. You need to disable all third party hacks and ensure that the systemd-fsck services are enabled for each desired filesystem (systemctl status systemd-fsck@*), also that /etc/default/grub is sane and contains:

# Uncomment to ensure that the root filesystem is mounted read-only so that
# systemd-fsck can run the check automatically
GRUB_ROOT_FS_RO=true

Validate with journalctl -t systemd-fsck. I’ve multiple partitions that are all checked as expected and verified by the journal log.

This looks good as far as I can tell:

systemctl status systemd-fsck@* ● systemd-fsck@dev-disk-by\x2duuid-3acbe656\x2d9d3d\x2d4de3\x2d926c\x2da839bb0e94b2.service - File System Check on /dev/disk/by-uu> Loaded: loaded (/etc/systemd/system/systemd-fsck@.service; static) Active: active (exited) since Tue 2021-01-12 16:52:10 CET; 2h 46min ago Docs: man:systemd-fsck@.service(8) Process: 237 ExecStart=/usr/lib/systemd/systemd-fsck /dev/disk/by-uuid/3acbe656-9d3d-4de3-926c-a839bb0e94b2 (code=exited, stat> Main PID: 237 (code=exited, status=0/SUCCESS)

I checked, it’s there.

Here, only my root filesystem on /dev/sda1 appears:

-- Boot 9014e54e0e4d466d93a1430ae19df252 -- Jan 12 15:37:51 manjaro systemd-fsck[181]: /dev/sda1: clean, 1325399/28745728 files, 29948663/114959872 blocks -- Boot 132050fd8efe4cb793541f5e7cf9e82d -- Jan 12 16:52:09 manjaro systemd-fsck[181]: /dev/sda1: clean, 1325397/28745728 files, 29948665/114959872 blocks - that’s just why I was asking.
UPDATE: I just saw that Maximum mount count was set to -1 meaning that this filesystem is never checked. I now set it to be checked at least after a certain interval using tune2fs /dev/sdb1 -i 1m.