"linux-modules-cleanup.service" is not automatically called on Kernel updates

As a follow-up to my earlier post How do I properly clean out a rogue kernel and/or modules? … I have to manually execute linux-modules-cleanup.service each kernel update in order to clean up the /lib/modules folder.

And it’s important to note that a few updates ago I’d had removed kernel-alive and replaced it with kernel-modules-hook… after reading Please use kernel-modules-hook instead of kernel-alive! - #87 by andreas85

After today’s stable update, I ran the following command to see how many “rogue” folders were still in the /lib/modules folder:

$ ls -la /lib/modules
total 292
drwxr-xr-x  14 root root   4096 Feb  5 10:31 .
drwxr-xr-x 239 root root 241664 Feb  5 10:31 ..
drwxr-xr-x   3 root root   4096 Feb  5 10:31 5.10.96-1-MANJARO
drwxr-xr-x   3 root root   4096 Nov 29 21:01 5.13.19-2-MANJARO
drwxr-xr-x   3 root root   4096 Dec 10 14:52 5.14.21-2-MANJARO
drwxr-xr-x   3 root root   4096 Jan 23 08:46 5.15.12-1-MANJARO
drwxr-xr-x   3 root root   4096 Feb  5 10:31 5.15.16-1-MANJARO
drwxr-xr-x   3 root root   4096 Feb  5 10:31 5.15.19-1-MANJARO
drwxr-xr-x   3 root root   4096 Jan  2 10:32 5.15.7-1-MANJARO
drwxr-xr-x   2 root root   4096 Feb  5 10:31 extramodules-5.10-MANJARO
drwxr-xr-x   2 root root   4096 Nov 29 21:01 extramodules-5.13-MANJARO
drwxr-xr-x   2 root root   4096 Dec 10 14:52 extramodules-5.14-MANJARO
drwxr-xr-x   2 root root   4096 Feb  5 10:31 extramodules-5.15-MANJARO
drwxr-xr-x   2 root root   4096 Dec 20 09:47 .old

Today’s updates migrated me to kernel 5.15.19-1, so the “rogue” 5.15.x paths were:

  • 5.15.12-1-MANJARO
  • 5.15.16-1-MANJARO
  • 5.15.7-1-MANJARO

I can clean up those folders “manually” by executing… $ systemctl start linux-modules-cleanup.service, and confirm the “rogue” folders have been (re)moved:

$ ls -la /lib/modules                           
total 280
drwxr-xr-x  11 root root   4096 Feb  5 11:10 .
drwxr-xr-x 239 root root 241664 Feb  5 10:31 ..
drwxr-xr-x   3 root root   4096 Feb  5 10:31 5.10.96-1-MANJARO
drwxr-xr-x   3 root root   4096 Nov 29 21:01 5.13.19-2-MANJARO
drwxr-xr-x   3 root root   4096 Dec 10 14:52 5.14.21-2-MANJARO
drwxr-xr-x   3 root root   4096 Feb  5 10:31 5.15.19-1-MANJARO
drwxr-xr-x   2 root root   4096 Feb  5 10:31 extramodules-5.10-MANJARO
drwxr-xr-x   2 root root   4096 Nov 29 21:01 extramodules-5.13-MANJARO
drwxr-xr-x   2 root root   4096 Dec 10 14:52 extramodules-5.14-MANJARO
drwxr-xr-x   2 root root   4096 Feb  5 10:31 extramodules-5.15-MANJARO
drwxr-xr-x   5 root root   4096 Feb  5 11:10 .old

kernel-modules-hook logic (called at some point by the above systemctl command) actually moves the “rogue” folders into the .old folder (and then after a few days purge them), and we can see the (re)moved folders here:

$ ls -la /lib/modules/.old
total 20
drwxr-xr-x  5 root root 4096 Feb  5 11:10 .
drwxr-xr-x 11 root root 4096 Feb  5 11:10 ..
drwxr-xr-x  3 root root 4096 Jan 23 08:46 5.15.12-1-MANJARO
drwxr-xr-x  3 root root 4096 Feb  5 10:31 5.15.16-1-MANJARO
drwxr-xr-x  3 root root 4096 Jan  2 10:32 5.15.7-1-MANJARO

Knowing that others are successfully using kernel-modules-hook, I’d appreciate some help digging deeper into the root of my issue and returning the kernel update process to automatically take care of cleaning out the “rogue” module folders?

Okay, it looks like the root cause was already suggested in my earlier post, but I thought I’d proved it wasn’t the case in error.

I see now that I must have confused/mixed-up systemd-tmpfiles-clean.service for linux-modules-cleanup.service… because the service definitely is not listed in my timers…

$ systemctl list-timers -all
NEXT                        LEFT                LAST                        PASSED     UNIT                          ACTIVATES                      
Sun 2022-02-06 02:40:54 CST 2h 25min left       Sat 2022-02-05 00:25:11 CST 23h ago    updatedb.timer                updatedb.service
Sun 2022-02-06 11:04:17 CST 10h left            Sat 2022-02-05 11:04:17 CST 13h ago    systemd-tmpfiles-clean.timer  systemd-tmpfiles-clean.service
Mon 2022-02-07 00:00:00 CST 23h left            Sun 2022-02-06 00:00:17 CST 15min ago  logrotate.timer               logrotate.service
Mon 2022-02-07 00:00:00 CST 23h left            Sun 2022-02-06 00:00:17 CST 15min ago  man-db.timer                  man-db.service
Mon 2022-02-07 00:00:00 CST 23h left            Sun 2022-02-06 00:00:17 CST 15min ago  pkgfile-update.timer          pkgfile-update.service
Mon 2022-02-07 00:00:00 CST 23h left            Sun 2022-02-06 00:00:17 CST 15min ago  shadow.timer                  shadow.service
Mon 2022-02-07 01:38:07 CST 1 day 1h left       Mon 2022-01-31 00:24:31 CST 5 days ago fstrim.timer                  fstrim.service
Thu 2022-02-10 14:42:37 CST 4 days left         Thu 2022-02-03 20:47:54 CST 2 days ago pamac-mirrorlist.timer        pamac-mirrorlist.service
Sat 2022-03-05 15:00:00 CST 3 weeks 6 days left Sat 2022-02-05 15:00:00 CST 9h ago     pamac-cleancache.timer        pamac-cleancache.service
n/a                         n/a                 n/a                         n/a        mdadm-last-resort@md127.timer mdadm-last-resort@md127.service

10 timers listed.

Memory told me I could fix this by “enabling” the service, so I ran…

$ systemctl enable linux-modules-cleanup.service
Created symlink /etc/systemd/system/basic.target.wants/linux-modules-cleanup.service → /usr/lib/systemd/system/linux-modules-cleanup.service.

But when I re-request the list, the service is not listed.

Should I have run the service enable as sudo? Or perhaps miss/botch a step or command?


EDIT: Oh my, reading is a skill. systemd-tmpfiles-clean.service was the recommended service to ensure was running, assumption being that it would take care of running linux-modules-cleanup.service

So have I further complicated things by creating another “unnecessary” symlink? (even though it didn’t seem to register a running service?)

$ systemctl enable linux-modules-cleanup.service
Created symlink /etc/systemd/system/basic.target.wants/linux-modules-cleanup.service → /usr/lib/systemd/system/linux-modules-cleanup.service.

Assuming I can clean that up by removing the symlink… does that leave my revised issue/question as… why is my systemd-tmpfiles-clean.service not running the linux-modules-cleanup.service as expected?


EDIT 2: Thinking I’d explore removing the symlink I’d created, I made an interesting discovery. Whereas the link I created was “plural” (linux-modules-cleanup.service), there appears to be a broken (white text with blinking read highlight) “singular” (linux-module-cleanup.service)symlink in the same folder…

$ ls -la /etc/systemd/system/basic.target.wants/
total 8
drwxr-xr-x  2 root root 4096 Feb  6 00:17 .
drwxr-xr-x 11 root root 4096 Nov 19 10:51 ..
lrwxrwxrwx  1 root root   48 Jun 14  2021 linux-module-cleanup.service -> /etc/systemd/system/linux-module-cleanup.service
lrwxrwxrwx  1 root root   53 Feb  6 00:17 linux-modules-cleanup.service -> /usr/lib/systemd/system/linux-modules-cleanup.service

Here is a screenshot for clarity…

What do you see in your /etc/systemd/system/basic.target.wants/ folder? What steps should I follow to clean up and correct my links?

In case it's helpful, here are the contents of the "broken" links folder
$ ls -la /etc/systemd/system/
total 48
drwxr-xr-x 11 root root 4096 Nov 19 10:51 .
drwxr-xr-x  7 root root 4096 Feb  5 10:31 ..
drwxr-xr-x  2 root root 4096 Feb  6 00:17 basic.target.wants
drwxr-xr-x  2 root root 4096 Jul 13  2021 bluetooth.target.wants
lrwxrwxrwx  1 root root   41 Jul 13  2021 dbus-org.bluez.service -> /usr/lib/systemd/system/bluetooth.service
lrwxrwxrwx  1 root root   44 Jul 13  2021 dbus-org.freedesktop.Avahi.service -> /usr/lib/systemd/system/avahi-daemon.service
lrwxrwxrwx  1 root root   44 Jul 13  2021 dbus-org.freedesktop.ModemManager1.service -> /usr/lib/systemd/system/ModemManager.service
lrwxrwxrwx  1 root root   57 Jul 13  2021 dbus-org.freedesktop.nm-dispatcher.service -> /usr/lib/systemd/system/NetworkManager-dispatcher.service
lrwxrwxrwx  1 root root   49 Jul 13  2021 dbus-org.freedesktop.timesync1.service -> /usr/lib/systemd/system/systemd-timesyncd.service
lrwxrwxrwx  1 root root   36 Jul 13  2021 display-manager.service -> /usr/lib/systemd/system/sddm.service
drwxr-xr-x  2 root root 4096 Jun 14  2021 getty.target.wants
-rw-r--r--  1 root root  298 Oct 13 10:40 grub-btrfs.path
drwxr-xr-x  2 root root 4096 Jan 28 22:19 multi-user.target.wants
drwxr-xr-x  2 root root 4096 Jul 13  2021 network-online.target.wants
drwxr-xr-x  2 root root 4096 Jul 13  2021 printer.target.wants
drwxr-xr-x  2 root root 4096 Jul 13  2021 sockets.target.wants
drwxr-xr-x  2 root root 4096 Jul 13  2021 sysinit.target.wants
lrwxrwxrwx  1 root root   56 Oct 21  2018 systemd-fsck-silent-root.service -> /usr/lib/systemd/system/systemd-fsck-silent-root.service
lrwxrwxrwx  1 root root   52 Oct 21  2018 systemd-fsck-silent@.service -> /usr/lib/systemd/system/systemd-fsck-silent@.service
drwxr-xr-x  2 root root 4096 Jul 28  2021 timers.target.wants
In case it's helpful, here are the contents of the folder for the link I might have made incorrectly
$ ls -la /usr/lib/systemd/system/
total 1676
drwxr-xr-x 16 root root 20480 Feb  5 10:31  .
drwxr-xr-x 18 root root  4096 Feb  5 10:31  ..
-rw-r--r--  1 root root  1939 Jan 28 18:50  accounts-daemon.service
-rw-r--r--  1 root root   161 Sep 18 18:49  acpid.service
-rw-r--r--  1 root root   410 Dec  6 12:44  alsa-restore.service
-rw-r--r--  1 root root   387 Dec  6 12:44  alsa-state.service
-rw-r--r--  1 root root   880 Feb  3 18:54  apparmor.service
-rw-r--r--  1 root root  1690 Jan 29 16:02  auditd.service
-rw-r--r--  1 root root   706 Jan 23 13:45  auth-rpcgss-module.service
lrwxrwxrwx  1 root root    14 Jan 30 10:33  autovt@.service -> getty@.service
-rw-r--r--  1 root root  1042 Dec  1 14:38  avahi-daemon.service
-rw-r--r--  1 root root   870 Dec  1 14:38  avahi-daemon.socket
-rw-r--r--  1 root root   950 Dec  1 14:38  avahi-dnsconfd.service
-rw-r--r--  1 root root   927 Jan 30 10:33  basic.target
-r--r--r--  1 root root   407 Nov 29 07:43  blk-availability.service
-rw-r--r--  1 root root   449 Jan 30 10:33  blockdev@.target
-rw-r--r--  1 root root   348 Jan  6 14:23  bluetooth-mesh.service
-rw-r--r--  1 root root   420 Jan  6 14:23  bluetooth.service
-rw-r--r--  1 root root   435 Jan 30 10:33  bluetooth.target
-rw-r--r--  1 root root   463 Jan 30 10:33  boot-complete.target
-rw-r--r--  1 root root   190 Jan 12 13:32  btrfs-scrub@.service
-rw-r--r--  1 root root   160 Jan 12 13:32  btrfs-scrub@.timer
-rw-r--r--  1 root root   491 Jun 23  2021  canberra-system-bootup.service
-rw-r--r--  1 root root   509 Jun 23  2021  canberra-system-shutdown-reboot.service
-rw-r--r--  1 root root   466 Jun 23  2021  canberra-system-shutdown.service
-rw-r--r--  1 root root   659 Jan 13 09:37  clamav-clamonacc.service
-rw-r--r--  1 root root   496 Jan 13 09:37  clamav-daemon.service
-rw-r--r--  1 root root   434 Jan 13 09:37  clamav-daemon.socket
-rw-r--r--  1 root root   446 Jan 13 09:37  clamav-freshclam.service
-rw-r--r--  1 root root   291 Jun  6  2021  colord.service
-rw-r--r--  1 root root   154 Dec  2 17:25  configure-printer@.service
-rw-r--r--  1 root root  1064 Jan 30 10:33  console-getty.service
-rw-r--r--  1 root root  1246 Jan 30 10:33  container-getty@.service
-rw-r--r--  1 root root   247 Jan 10 05:12  cpupower.service
-rw-r--r--  1 root root   194 Mar 29  2021  cronie.service
-rw-r--r--  1 root root   473 Jan 30 10:33  cryptsetup-pre.target
-rw-r--r--  1 root root   420 Jan 30 10:33  cryptsetup.target
lrwxrwxrwx  1 root root    13 Jan 30 10:33  ctrl-alt-del.target -> reboot.target
-rw-r--r--  1 root root   277 Jan 16 04:27  cups-browsed.service
-r--r--r--  1 root root   148 Jan 27 10:58  cups-lpd@.service
-r--r--r--  1 root root   139 Jan 27 10:58  cups-lpd.socket
-r--r--r--  1 root root   142 Jan 27 10:58  cups.path
-r--r--r--  1 root root   291 Jan 27 10:58  cups.service
-r--r--r--  1 root root   132 Jan 27 10:58  cups.socket
-rw-r--r--  1 root root   221 Jan 13 16:16  daxdev-reconfigure@.service
lrwxrwxrwx  1 root root    25 Jan 30 10:33  dbus-org.freedesktop.hostname1.service -> systemd-hostnamed.service
lrwxrwxrwx  1 root root    23 Jan 30 10:33  dbus-org.freedesktop.import1.service -> systemd-importd.service
lrwxrwxrwx  1 root root    23 Jan 30 10:33  dbus-org.freedesktop.locale1.service -> systemd-localed.service
lrwxrwxrwx  1 root root    22 Jan 30 10:33  dbus-org.freedesktop.login1.service -> systemd-logind.service
lrwxrwxrwx  1 root root    24 Jan 30 10:33  dbus-org.freedesktop.machine1.service -> systemd-machined.service
lrwxrwxrwx  1 root root    20 Jan 30 10:33  dbus-org.freedesktop.oom1.service -> systemd-oomd.service
lrwxrwxrwx  1 root root    25 Jan 30 10:33  dbus-org.freedesktop.portable1.service -> systemd-portabled.service
lrwxrwxrwx  1 root root    25 Jan 30 10:33  dbus-org.freedesktop.timedate1.service -> systemd-timedated.service
-rw-r--r--  1 root root   380 Jul  2  2020  dbus.service
-rw-r--r--  1 root root   102 Jul  2  2020  dbus.socket
-rw-r--r--  1 root root  1063 Jan 30 10:33  debug-shell.service
lrwxrwxrwx  1 root root    16 Jan 30 10:33  default.target -> graphical.target
-rw-r--r--  1 root root   758 Jan 30 10:33  dev-hugepages.mount
-rw-r--r--  1 root root   701 Jan 30 10:33  dev-mqueue.mount
-rw-r--r--  1 root root   255 Dec 26 13:01  dhclient@.service
-rw-r--r--  1 root root   233 Dec  6 06:27  dhcpcd.service
-rw-r--r--  1 root root   316 Dec  6 06:27  dhcpcd@.service
-rw-r--r--  1 root root   387 Dec 20 16:49  dhtcluster.service
-rw-r--r--  1 root root   923 Dec 20 16:49  dhtnode.service
-r--r--r--  1 root root   344 Nov 29 07:43  dm-event.service
-r--r--r--  1 root root   214 Nov 29 07:43  dm-event.socket
-rw-r--r--  1 root root   336 Feb  2  2021  dmraid.service
-rw-r--r--  1 root root   485 Sep  9 00:59  dnsmasq.service
-rw-r--r--  1 root root   300 Feb  3 05:18  e2scrub_all.service
-rw-r--r--  1 root root   251 Feb  3 05:18  e2scrub_all.timer
-rw-r--r--  1 root root   228 Feb  3 05:18  e2scrub_fail@.service
-rw-r--r--  1 root root   556 Feb  3 05:18  e2scrub_reap.service
-rw-r--r--  1 root root   441 Feb  3 05:18  e2scrub@.service
-rw-r--r--  1 root root   813 Jan 30 10:33  emergency.service
-rw-r--r--  1 root root   479 Jan 30 10:33  emergency.target
-rw-r--r--  1 root root   134 Jan  9 10:05  etc-pacman.d-gnupg.mount
-rw-r--r--  1 root root   549 Jan 30 10:33  exit.target
-rw-r--r--  1 root root   457 Jan 30 10:33  factory-reset.target
-rw-r--r--  1 root root   235 Feb 18  2021  fancontrol.service
-rw-r--r--  1 root root   500 Jan 30 10:33  final.target
-rw-r--r--  1 root root   461 Jan 30 10:33  first-boot-complete.target
-rw-r--r--  1 root root   259 Jan 19 13:48  flatpak-system-helper.service
-rw-r--r--  1 root root   480 Jan 31 08:16  fstrim.service
-rw-r--r--  1 root root   270 Jan 31 08:16  fstrim.timer
-rw-r--r--  1 root root   205 Sep  8 16:13  ftpd.service
-rw-r--r--  1 root root   402 Jan 13 17:04  fwupd-offline-update.service
-rw-r--r--  1 root root   404 Jan 13 17:04  fwupd-refresh.service
-rw-r--r--  1 root root   159 Jan 13 17:04  fwupd-refresh.timer
-rw-r--r--  1 root root   611 Jan 13 17:04  fwupd.service
-rw-r--r--  1 root root   464 Sep  4 17:24  geoclue.service
-rw-r--r--  1 root root   517 Jan 30 10:33  getty-pre.target
-rw-r--r--  1 root root  1979 Jan 30 10:33  getty@.service
-rw-r--r--  1 root root   508 Jan 30 10:33  getty.target
-rw-r--r--  1 root root   324 Jan 29 03:19  git-daemon@.service
-rw-r--r--  1 root root   112 Jan 29 03:19  git-daemon.socket
-rw-r--r--  1 root root    83 Aug 31 14:06  gpm.path
-rw-r--r--  1 root root   271 Aug 31 14:06  gpm.service
-rw-r--r--  1 root root   606 Jan 30 10:33  graphical.target
-rw-r--r--  1 root root   400 Jan 14  2021  gssproxy.service
-rw-r--r--  1 root root   542 Jan 30 10:33  halt.target
-rw-r--r--  1 root root   881 Jan  8 14:23  haveged-once.service
-rw-r--r--  1 root root  1049 Jan  8 14:23  haveged.service
-rw-r--r--  1 root root   199 Feb 18  2021  healthd.service
-rw-r--r--  1 root root   526 Jan 30 10:33  hibernate.target
-rw-r--r--  1 root root   124 Dec 19 04:29  hplip-printer@.service
-rw-r--r--  1 root root   538 Jan 30 10:33  hybrid-sleep.target
-rw-r--r--  1 root root   670 Jan 30 10:33  initrd-cleanup.service
-rw-r--r--  1 root root   598 Jan 30 10:33  initrd-fs.target
-rw-r--r--  1 root root   820 Jan 30 10:33  initrd-parse-etc.service
-rw-r--r--  1 root root   566 Jan 30 10:33  initrd-root-device.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  initrd-root-device.target.wants
-rw-r--r--  1 root root   571 Jan 30 10:33  initrd-root-fs.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  initrd-root-fs.target.wants
-rw-r--r--  1 root root   589 Jan 30 10:33  initrd-switch-root.service
-rw-r--r--  1 root root   779 Jan 30 10:33  initrd-switch-root.target
-rw-r--r--  1 root root   810 Jan 30 10:33  initrd.target
-rw-r--r--  1 root root   823 Jan 30 10:33  initrd-udevadm-cleanup-db.service
-rw-r--r--  1 root root   571 Jan 30 10:33  initrd-usr-fs.target
-rw-r--r--  1 root root   487 Jan 30 10:33  integritysetup-pre.target
-rw-r--r--  1 root root   430 Jan 30 10:33  integritysetup.target
-rw-r--r--  1 root root   393 Jan 20  2021  ip6tables.service
-rw-r--r--  1 root root   364 Jan 20  2021  iptables.service
-rw-r--r--  1 root root   631 Dec  2 13:23  irexec.service
-rw-r--r--  1 root root   549 Jan 30 10:33  kexec.target
-rw-r--r--  1 root root   705 Jan 30 10:33  kmod-static-nodes.service
-rw-r--r--  1 root root   136 Nov 13 12:07  krb5-kadmind.service
-rw-r--r--  1 root root   128 Nov 13 12:07  krb5-kdc.service
-rw-r--r--  1 root root   127 Nov 13 12:07  krb5-kpropd.service
-rw-r--r--  1 root root   158 Nov 13 12:07  krb5-kpropd@.service
-rw-r--r--  1 root root   122 Nov 13 12:07  krb5-kpropd.socket
-rw-r--r--  1 root root   695 Jan 30 10:33  ldconfig.service
-rw-r--r--  1 root root   310 Aug 21 05:41  linux-modules-cleanup.service
-rw-r--r--  1 root root   632 Dec  2 13:23  lircd.service
-rw-r--r--  1 root root   182 Dec  2 13:23  lircd-setup.service
-rw-r--r--  1 root root    93 Dec  2 13:23  lircd.socket
-rw-r--r--  1 root root   489 Dec  2 13:23  lircd-uinput.service
-rw-r--r--  1 root root   507 Dec  2 13:23  lircmd.service
-rw-r--r--  1 root root   328 Feb 18  2021  lm_sensors.service
-rw-r--r--  1 root root   453 Jan 30 10:33  local-fs-pre.target
-rw-r--r--  1 root root   555 Jan 30 10:33  local-fs.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  local-fs.target.wants
-rw-r--r--  1 root root   870 Jan  8 02:06  logrotate.service
-rw-r--r--  1 root root   191 Jan  8 02:06  logrotate.timer
-r--r--r--  1 root root   326 Nov 29 07:43  lvm2-lvmpolld.service
-r--r--r--  1 root root   205 Nov 29 07:43  lvm2-lvmpolld.socket
-r--r--r--  1 root root   539 Nov 29 07:43  lvm2-monitor.service
-rw-r--r--  1 root root   453 Jan 30 10:33  machine.slice
-rw-r--r--  1 root root   470 Jan 30 10:33  machines.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  machines.target.wants
-rw-r--r--  1 root root   485 Jun 22  2021  man-db.service
-rw-r--r--  1 root root   164 Jun 22  2021  man-db.timer
-rw-r--r--  1 root root   481 Jan 10 10:12  mdadm-grow-continue@.service
-rw-r--r--  1 root root   210 Jan 10 10:12  mdadm-last-resort@.service
-rw-r--r--  1 root root   179 Jan 10 10:12  mdadm-last-resort@.timer
-rw-r--r--  1 root root   613 Jan 10 10:12  mdcheck_continue.service
-rw-r--r--  1 root root   400 Jan 10 10:12  mdcheck_continue.timer
-rw-r--r--  1 root root   563 Jan 10 10:12  mdcheck_start.service
-rw-r--r--  1 root root   426 Jan 10 10:12  mdcheck_start.timer
-rw-r--r--  1 root root   521 Jan 10 10:12  mdmonitor-oneshot.service
-rw-r--r--  1 root root   398 Jan 10 10:12  mdmonitor-oneshot.timer
-rw-r--r--  1 root root   521 Jan 10 10:12  mdmonitor.service
-rw-r--r--  1 root root  1034 Jan 10 10:12  mdmon@.service
-rw-r--r--  1 root root   345 Dec  3 09:06  mkinitcpio-generate-shutdown-ramfs.service
-rw-r--r--  1 root root   479 Nov 26 12:06  ModemManager.service
-rw-r--r--  1 root root   549 Jan 30 10:33  modprobe@.service
-rw-r--r--  1 root root   903 Dec  1 15:36  mpd.service
-rw-r--r--  1 root root   135 Dec  1 15:36  mpd.socket
-rw-r--r--  1 root root   540 Jan 30 10:33  multi-user.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  multi-user.target.wants
-rw-r--r--  1 root root   134 Jan 13 16:16  ndctl-monitor.service
-rw-r--r--  1 root root   637 Jan 13 09:58  NetworkManager-dispatcher.service
-rw-r--r--  1 root root  1335 Jan 13 09:58  NetworkManager.service
drwxr-xr-x  2 root root  4096 Jan 23 08:45  NetworkManager.service.d
-rw-r--r--  1 root root  1148 Jan 13 09:58  NetworkManager-wait-online.service
-rw-r--r--  1 root root   513 Jan 30 10:33  network-online.target
-rw-r--r--  1 root root   520 Jan 30 10:33  network-pre.target
-rw-r--r--  1 root root   529 Jan 30 10:33  network.target
-rw-r--r--  1 root root   291 Jan 23 13:45  nfs-blkmap.service
-rw-r--r--  1 root root   433 Jan 23 13:45  nfs-client.target
-rw-r--r--  1 root root   236 Jan 23 13:45  nfsdcld.service
-rw-r--r--  1 root root   222 Jan 23 13:45  nfs-idmapd.service
-rw-r--r--  1 root root   287 Jan 23 13:45  nfs-mountd.service
-rw-r--r--  1 root root   895 Jan 23 13:45  nfs-server.service
-rw-r--r--  1 root root   567 Jan 23 13:45  nfs-utils.service
-rw-r--r--  1 root root   273 Jan 23 13:45  nfsv4-exportd.service
-rw-r--r--  1 root root   779 Jan 23 13:45  nfsv4-server.service
-rw-r--r--  1 root root   573 Dec 22 04:10  ninfod.service
-rw-r--r--  1 root root   404 Jan 31 10:42  nmb.service
-rw-r--r--  1 root root  2177 Jan 13 09:58  nm-priv-helper.service
-rw-r--r--  1 root root   424 May 12  2021  nscd.service
-rw-r--r--  1 root root   562 Jan 30 10:33  nss-lookup.target
-rw-r--r--  1 root root   521 Jan 30 10:33  nss-user-lookup.target
-rw-r--r--  1 root root   230 Jul  1  2020  ntpdate.service
-rw-r--r--  1 root root   246 Jul  1  2020  ntpd.service
-rw-r--r--  1 root root   841 Dec 15 05:52  openvpn-client@.service
-rw-r--r--  1 root root  1000 Dec 15 05:52  openvpn-server@.service
-rw-r--r--  1 root root   981 Jan  6 16:31  ostree-finalize-staged.path
-rw-r--r--  1 root root  1870 Jan  6 16:31  ostree-finalize-staged.service
-rw-r--r--  1 root root  1147 Jan  6 16:31  ostree-prepare-root.service
-rw-r--r--  1 root root  1481 Jan  6 16:31  ostree-remount.service
-rw-r--r--  1 root root   109 Jan  9 10:05  paccache.service
-rw-r--r--  1 root root   142 Jan  9 10:05  paccache.timer
-rw-r--r--  1 root root   332 Jan  9 10:05  pacman-init.service
-rw-r--r--  1 root root   108 Jan 29 15:51  pamac-cleancache.service
-rw-r--r--  1 root root   140 Jan 29 15:51  pamac-cleancache.timer
-rw-r--r--  1 root root   118 Jan 29 15:51  pamac-daemon.service
-rw-r--r--  1 root root   157 Jan 29 15:51  pamac-mirrorlist.service
-rw-r--r--  1 root root   157 Jan 29 15:51  pamac-mirrorlist.timer
-rw-r--r--  1 root root   330 Sep  8 16:18  pam_namespace.service
-rw-r--r--  1 root root   407 Jan 30 10:33  paths.target
-rw-r--r--  1 root root   266 Dec  4 11:46  pcscd.service
-rw-r--r--  1 root root   157 Dec  4 11:46  pcscd.socket
-rw-r--r--  1 root root   319 May 27  2020  pkgfile-update.service
-rw-r--r--  1 root root   143 May 27  2020  pkgfile-update.timer
-rw-r--r--  1 root root   172 Jan 25 18:09  polkit.service
-rw-r--r--  1 root root   607 Jan 30 10:33  poweroff.target
-rw-r--r--  1 root root   153 May  2  2021  ppp@.service
-rw-r--r--  1 root root   433 Jan 30 10:33  printer.target
-rw-r--r--  1 root root    98 Jan 23 13:45  proc-fs-nfsd.mount
-rw-r--r--  1 root root   779 Jan 30 10:33  proc-sys-fs-binfmt_misc.automount
-rw-r--r--  1 root root   726 Jan 30 10:33  proc-sys-fs-binfmt_misc.mount
-rw-r--r--  1 root root   623 Jan 30 10:33  quotaon.service
-rw-r--r--  1 root root   674 Dec 22 04:10  rarpd@.service
-rw-r--r--  1 root root   741 Dec 22 04:10  rdisc.service
-rw-r--r--  1 root root   598 Jan 30 10:33  reboot.target
-rw-r--r--  1 root root   557 Jan 30 10:33  remote-cryptsetup.target
-rw-r--r--  1 root root   454 Jan 30 10:33  remote-fs-pre.target
-rw-r--r--  1 root root   530 Jan 30 10:33  remote-fs.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  remote-fs.target.wants
-rw-r--r--  1 root root   565 Jan 30 10:33  remote-veritysetup.target
-rw-r--r--  1 root root   804 Jan 30 10:33  rescue.service
-rw-r--r--  1 root root   500 Jan 30 10:33  rescue.target
-rw-r--r--  1 root root   165 Jan 31 08:16  rfkill-block@.service
-rw-r--r--  1 root root   167 Jan 31 08:16  rfkill-unblock@.service
-rw-r--r--  1 root root   132 Sep  8 16:13  rlogin@.service
-rw-r--r--  1 root root   135 Sep  8 16:13  rlogin.socket
-rw-r--r--  1 root root   480 Dec  2 13:58  rpcbind.service
-rw-r--r--  1 root root   368 Dec  2 13:58  rpcbind.socket
-rw-r--r--  1 root root   548 Jan 30 10:33  rpcbind.target
-rw-r--r--  1 root root   281 Jan 23 13:45  rpc-gssd.service
-rw-r--r--  1 root root    80 Jan 23 13:45  rpc_pipefs.target
-rw-r--r--  1 root root   387 Jan 23 13:45  rpc-statd-notify.service
-rw-r--r--  1 root root   429 Jan 23 13:45  rpc-statd.service
-rw-r--r--  1 root root   129 Sep  8 16:13  rsh@.service
-rw-r--r--  1 root root   135 Sep  8 16:13  rsh.socket
-rw-r--r--  1 root root  1011 Jul 28  2021  rsyncd.service
-rw-r--r--  1 root root   933 Jul 28  2021  rsyncd@.service
-rw-r--r--  1 root root   165 Jul 28  2021  rsyncd.socket
-rw-r--r--  1 root root  1020 Apr  4  2020  rtkit-daemon.service
-rw-r--r--  1 root root   406 Jan 31 10:42  samba.service
-rw-r--r--  1 root root   431 Jan 19 12:13  saned@.service
-rw-r--r--  1 root root   132 Jan 19 12:13  saned.socket
-rw-r--r--  1 root root   357 Feb  3 15:26  sddm.service
-rw-r--r--  1 root root   264 Feb 18  2021  sensord.service
-rw-r--r--  1 root root  1496 Jan 30 10:33  serial-getty@.service
-rw-r--r--  1 root root   319 Jan 27 07:47  shadow.service
-rw-r--r--  1 root root   124 Jan 27 07:47  shadow.timer
-rw-r--r--  1 root root   457 Jan 30 10:33  shutdown.target
drwxr-xr-x  2 root root  4096 Dec 10 14:52  shutdown.target.wants
-rw-r--r--  1 root root   410 Jan 30 10:33  sigpwr.target
-rw-r--r--  1 root root   468 Jan 30 10:33  sleep.target
-rw-r--r--  1 root root   462 Jan 30 10:33  slices.target
-rw-r--r--  1 root root   428 Jan 30 10:33  smartcard.target
-rw-r--r--  1 root root   323 Dec 31  2020  smartd.service
-rw-r--r--  1 root root   450 Jan 31 10:42  smb.service
-rw-r--r--  1 root root   880 Jan 12 12:43  snapd.apparmor.service
-rw-r--r--  1 root root   151 Jan 12 12:43  snapd.failure.service
-rw-r--r--  1 root root   318 Jan 12 12:43  snapd.seeded.service
-rw-r--r--  1 root root   477 Jan 12 12:43  snapd.service
-rw-r--r--  1 root root   281 Jan 12 12:43  snapd.socket
-rw-r--r--  1 root root   223 Aug  3  2021  sndiod.service
-rw-r--r--  1 root root   266 Dec  1 15:45  snmpd.service
-rw-r--r--  1 root root   283 Dec  1 15:45  snmptrapd.service
-rw-r--r--  1 root root   409 Jan 30 10:33  sockets.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  sockets.target.wants
-rw-r--r--  1 root root   428 Jan 30 10:33  sound.target
drwxr-xr-x  2 root root  4096 Dec 16 07:50  sound.target.wants
-rw-r--r--  1 root root   527 Sep 26 13:41  sshdgenkeys.service
-rw-r--r--  1 root root   250 Sep 26 13:41  sshd.service
-rw-r--r--  1 root root   306 Feb  2 02:02  sudo_logsrvd.service
-rw-r--r--  1 root root   511 Jan 30 10:33  suspend.target
-rw-r--r--  1 root root   585 Jan 30 10:33  suspend-then-hibernate.target
-rw-r--r--  1 root root   402 Jan 30 10:33  swap.target
-rw-r--r--  1 root root  1162 Jan 30 10:33  sys-fs-fuse-connections.mount
-rw-r--r--  1 root root   566 Jan 30 10:33  sysinit.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  sysinit.target.wants
-rw-r--r--  1 root root  1136 Jan 30 10:33  sys-kernel-config.mount
-rw-r--r--  1 root root   746 Jan 30 10:33  sys-kernel-debug.mount
-rw-r--r--  1 root root   772 Jan 30 10:33  sys-kernel-tracing.mount
-rw-r--r--  1 root root  1415 Jan 30 10:33  syslog.socket
-rw-r--r--  1 root root   727 Jan 30 10:33  systemd-ask-password-console.path
-rw-r--r--  1 root root   745 Jan 30 10:33  systemd-ask-password-console.service
-rw-r--r--  1 root root   652 Jan 30 10:33  systemd-ask-password-wall.path
-rw-r--r--  1 root root   747 Jan 30 10:33  systemd-ask-password-wall.service
-rw-r--r--  1 root root   735 Jan 30 10:33  systemd-backlight@.service
-rw-r--r--  1 root root  1213 Jan 30 10:33  systemd-binfmt.service
-rw-r--r--  1 root root   690 Jan 30 10:33  systemd-bless-boot.service
-rw-r--r--  1 root root   730 Jan 30 10:33  systemd-boot-check-no-failures.service
-rw-r--r--  1 root root  1379 Jan 30 10:33  systemd-boot-system-token.service
-rw-r--r--  1 root root   684 Jan 30 10:33  systemd-boot-update.service
-rw-r--r--  1 root root  1149 Jan 30 10:33  systemd-coredump@.service
-rw-r--r--  1 root root   594 Jan 30 10:33  systemd-coredump.socket
-rw-r--r--  1 root root   564 Jan 30 10:33  systemd-exit.service
-rw-r--r--  1 root root  1317 Jan 30 10:33  systemd-firstboot.service
-rw-r--r--  1 root root   717 Jan 30 10:33  systemd-fsck-root.service
-rw-r--r--  1 root root   705 Jan 30 10:33  systemd-fsck@.service
-rw-r--r--  1 root root   668 Oct 21  2018  systemd-fsck-silent-root.service
-rw-r--r--  1 root root   725 Oct 21  2018  systemd-fsck-silent@.service
-rw-r--r--  1 root root   594 Jan 30 10:33  systemd-halt.service
-rw-r--r--  1 root root   680 Jan 30 10:33  systemd-hibernate-resume@.service
-rw-r--r--  1 root root   555 Jan 30 10:33  systemd-hibernate.service
-rw-r--r--  1 root root   645 Jan 30 10:33  systemd-homed-activate.service
-rw-r--r--  1 root root  1362 Jan 30 10:33  systemd-homed.service
-rw-r--r--  1 root root  1205 Jan 30 10:33  systemd-hostnamed.service
-rw-r--r--  1 root root   825 Jan 30 10:33  systemd-hwdb-update.service
-rw-r--r--  1 root root   576 Jan 30 10:33  systemd-hybrid-sleep.service
-rw-r--r--  1 root root  1032 Jan 30 10:33  systemd-importd.service
-rw-r--r--  1 root root   741 Jan 30 10:33  systemd-journal-catalog-update.service
-rw-r--r--  1 root root   655 Jan 30 10:33  systemd-journald-audit.socket
-rw-r--r--  1 root root  1154 Jan 30 10:33  systemd-journald-dev-log.socket
-rw-r--r--  1 root root  1816 Jan 30 10:33  systemd-journald.service
-rw-r--r--  1 root root  1458 Jan 30 10:33  systemd-journald@.service
-rw-r--r--  1 root root   906 Jan 30 10:33  systemd-journald.socket
-rw-r--r--  1 root root   746 Jan 30 10:33  systemd-journald@.socket
-rw-r--r--  1 root root   605 Jan 30 10:33  systemd-journald-varlink@.socket
-rw-r--r--  1 root root   788 Jan 30 10:33  systemd-journal-flush.service
-rw-r--r--  1 root root  1145 Jan 30 10:33  systemd-journal-gatewayd.service
-rw-r--r--  1 root root   500 Jan 30 10:33  systemd-journal-gatewayd.socket
-rw-r--r--  1 root root  1307 Jan 30 10:33  systemd-journal-remote.service
-rw-r--r--  1 root root   450 Jan 30 10:33  systemd-journal-remote.socket
-rw-r--r--  1 root root  1200 Jan 30 10:33  systemd-journal-upload.service
-rw-r--r--  1 root root   601 Jan 30 10:33  systemd-kexec.service
-rw-r--r--  1 root root  1213 Jan 30 10:33  systemd-localed.service
-rw-r--r--  1 root root  2018 Jan 30 10:33  systemd-logind.service
-rw-r--r--  1 root root  1306 Jan 30 10:33  systemd-machined.service
-rw-r--r--  1 root root   748 Jan 30 10:33  systemd-machine-id-commit.service
-rw-r--r--  1 root root  1019 Jan 30 10:33  systemd-modules-load.service
-rw-r--r--  1 root root  2325 Jan 30 10:33  systemd-networkd.service
-rw-r--r--  1 root root   682 Jan 30 10:33  systemd-networkd.socket
-rw-r--r--  1 root root   752 Jan 30 10:33  systemd-networkd-wait-online.service
-rw-r--r--  1 root root   690 Jan 30 10:33  systemd-network-generator.service
-rw-r--r--  1 root root  1596 Jan 30 10:33  systemd-nspawn@.service
-rw-r--r--  1 root root  1710 Jan 30 10:33  systemd-oomd.service
-rw-r--r--  1 root root   602 Jan 30 10:33  systemd-oomd.socket
-rw-r--r--  1 root root  1027 Jan 30 10:33  systemd-portabled.service
-rw-r--r--  1 root root   575 Jan 30 10:33  systemd-poweroff.service
-rw-r--r--  1 root root   747 Jan 30 10:33  systemd-pstore.service
-rw-r--r--  1 root root   670 Jan 30 10:33  systemd-quotacheck.service
-rw-r--r--  1 root root  1175 Jan 30 10:33  systemd-random-seed.service
-rw-r--r--  1 root root   568 Jan 30 10:33  systemd-reboot.service
-rw-r--r--  1 root root   779 Jan 30 10:33  systemd-remount-fs.service
-rw-r--r--  1 root root  1217 Jan 30 10:33  systemd-repart.service
-rw-r--r--  1 root root  1789 Jan 30 10:33  systemd-resolved.service
-rw-r--r--  1 root root   729 Jan 30 10:33  systemd-rfkill.service
-rw-r--r--  1 root root   734 Jan 30 10:33  systemd-rfkill.socket
-rw-r--r--  1 root root   556 Jan 30 10:33  systemd-suspend.service
-rw-r--r--  1 root root   623 Jan 30 10:33  systemd-suspend-then-hibernate.service
-rw-r--r--  1 root root   705 Jan 30 10:33  systemd-sysctl.service
-rw-r--r--  1 root root  1002 Jan 30 10:33  systemd-sysext.service
-rw-r--r--  1 root root  1024 Jan 30 10:33  systemd-sysusers.service
-rw-r--r--  1 root root  1162 Jan 30 10:33  systemd-timedated.service
-rw-r--r--  1 root root  1768 Jan 30 10:33  systemd-timesyncd.service
-rw-r--r--  1 root root  1208 Jan 30 10:33  systemd-time-wait-sync.service
-rw-r--r--  1 root root   693 Jan 30 10:33  systemd-tmpfiles-clean.service
-rw-r--r--  1 root root   539 Jan 30 10:33  systemd-tmpfiles-clean.timer
-rw-r--r--  1 root root   747 Jan 30 10:33  systemd-tmpfiles-setup-dev.service
-rw-r--r--  1 root root   814 Jan 30 10:33  systemd-tmpfiles-setup.service
-rw-r--r--  1 root root   650 Jan 30 10:33  systemd-udevd-control.socket
-rw-r--r--  1 root root   624 Jan 30 10:33  systemd-udevd-kernel.socket
-rw-r--r--  1 root root  1317 Jan 30 10:33  systemd-udevd.service
-rw-r--r--  1 root root   863 Jan 30 10:33  systemd-udev-settle.service
-rw-r--r--  1 root root   763 Jan 30 10:33  systemd-udev-trigger.service
-rw-r--r--  1 root root   682 Jan 30 10:33  systemd-update-done.service
-rw-r--r--  1 root root   807 Jan 30 10:33  systemd-update-utmp.service
-rw-r--r--  1 root root  1187 Jan 30 10:33  systemd-userdbd.service
-rw-r--r--  1 root root   691 Jan 30 10:33  systemd-userdbd.socket
-rw-r--r--  1 root root   655 Jan 30 10:33  systemd-user-sessions.service
-rw-r--r--  1 root root   650 Jan 30 10:33  systemd-vconsole-setup.service
-rw-r--r--  1 root root   743 Jan 30 10:33  systemd-volatile-root.service
-rw-r--r--  1 root root   454 Jan 30 10:33 'system-systemd\x2dcryptsetup.slice'
-rw-r--r--  1 root root  1423 Jan 30 10:33  system-update-cleanup.service
-rw-r--r--  1 root root   551 Jan 30 10:33  system-update-pre.target
-rw-r--r--  1 root root   625 Jan 30 10:33  system-update.target
drwxr-xr-x  2 root root  4096 Jan 23 08:45  system-update.target.wants
-rw-r--r--  1 root root   177 Sep  8 16:13  talk.service
-rw-r--r--  1 root root   160 Sep  8 16:13  talk.socket
-rw-r--r--  1 root root   244 Dec  2 13:18  teamd@.service
-rw-r--r--  1 root root   381 Jan 27 22:14  teamviewerd.service
-rw-r--r--  1 root root   115 Sep  8 16:13  telnet@.service
-rw-r--r--  1 root root   153 Sep  8 16:13  telnet.socket
-rw-r--r--  1 root root   458 Jan 30 10:33  timers.target
drwxr-xr-x  2 root root  4096 Feb  5 10:31  timers.target.wants
-rw-r--r--  1 root root   434 Jan 30 10:33  time-set.target
-rw-r--r--  1 root root   487 Jan 30 10:33  time-sync.target
-rw-r--r--  1 root root   797 Jan 30 10:33  tmp.mount
-rw-r--r--  1 root root   203 Oct  1 09:27  udisks2.service
-rw-r--r--  1 root root   608 Oct  1 09:27  udisks2-zram-setup@.service
-rw-r--r--  1 root root   351 Dec 11 12:49  ufw.service
-rw-r--r--  1 root root   465 Jan 30 10:33  umount.target
-rw-r--r--  1 root root   224 Apr 21  2021  updatedb.service
-rw-r--r--  1 root root   113 Apr 21  2021  updatedb.timer
-rw-r--r--  1 root root   990 Aug 26 16:34  upower.service
-rw-r--r--  1 root root   426 Jan 30 10:33  usb-gadget.target
-rw-r--r--  1 root root   200 Jul 19  2020  usb_modeswitch@.service
-rw-r--r--  1 root root   199 Jun 17  2020  usbmuxd.service
-rw-r--r--  1 root root   704 Jan 30 10:33  user-runtime-dir@.service
-rw-r--r--  1 root root   779 Jan 30 10:33  user@.service
-rw-r--r--  1 root root   440 Jan 30 10:33  user.slice
drwxr-xr-x  2 root root  4096 Feb  5 10:31  user-.slice.d
-rw-r--r--  1 root root   538 Jan 31 08:16  uuidd.service
-rw-r--r--  1 root root   149 Jan 31 08:16  uuidd.socket
-rw-r--r--  1 root root   807 Jan 30 10:33  var-lib-machines.mount
-rw-r--r--  1 root root   191 Jan 23 13:45  var-lib-nfs-rpc_pipefs.mount
-rw-r--r--  1 root root   221 Jan 18 14:32  vboxweb.service
-rw-r--r--  1 root root   481 Jan 30 10:33  veritysetup-pre.target
-rw-r--r--  1 root root   427 Jan 30 10:33  veritysetup.target
-rw-r--r--  1 root root   234 Jul 20  2021  vpnc@.service
-rw-r--r--  1 root root   385 Jan 31 10:42  winbind.service
-rw-r--r--  1 root root   452 Jan 23 03:35  wpa_supplicant-nl80211@.service
-rw-r--r--  1 root root   310 Jan 23 03:35  wpa_supplicant.service
-rw-r--r--  1 root root   414 Jan 23 03:35  wpa_supplicant@.service
-rw-r--r--  1 root root   446 Jan 23 03:35  wpa_supplicant-wired@.service
-rw-r--r--  1 root root   375 Dec  7 02:35  xfs_scrub_all.service
-rw-r--r--  1 root root   250 Dec  7 02:35  xfs_scrub_all.timer
-rw-r--r--  1 root root   272 Dec  7 02:35  xfs_scrub_fail@.service
-rw-r--r--  1 root root   540 Dec  7 02:35  xfs_scrub@.service
And here are the contents of the file the symlink I created points to
$ cat /usr/lib/systemd/system/linux-modules-cleanup.service
[Unit]
Description=Clean up modules from old kernels

[Service]
Type=oneshot
ExecStart=/bin/bash -exc 'for i in /usr/lib/modules/[0-9]*; do if [[ $${i##*/} = \'%v\' ]] || pacman -Qo "$${i}"; then continue; fi; rsync -AHXal "$${i}" /usr/lib/modules/.old/; rm -rf "$${i}"; done'

[Install]
WantedBy=basic.target

Hmm, I’m starting to think I might have been on the right track (that many files inside /usr/lib/systemd/system/ must be meaningful ), and that the only thing left to do is drop the broken link :thinking:

Hopefully someone can confirm my thoughts.

ls -l /etc/systemd/system/basic.target.wants                                                                                                     
insgesamt 4
lrwxrwxrwx 1 root root 48 19. Jan 2020  linux-module-cleanup.service -> /etc/systemd/system/linux-module-cleanup.service

BUT this link is broken in my system :frowning:

because /etc/systemd/system/linux-module-cleanup.service does not exist !

This is a leftover from kernel-alive !

ls -l /lib/modules                                                                                                                               
insgesamt 0
drwxr-xr-x 1 root root 468 25. Jan 19:02 4.19.225-1-MANJARO
drwxr-xr-x 1 root root 514 20. Nov 08:56 5.10.70-1-MANJARO
drwxr-xr-x 1 root root 514 20. Nov 08:56 5.10.79-1-MANJARO
drwxr-xr-x 1 root root 514  5. Jan 13:08 5.10.84-1-MANJARO
drwxr-xr-x 1 root root 514 25. Jan 19:03 5.10.89-1-MANJARO
drwxr-xr-x 1 root root 514 25. Jan 19:03 5.10.93-1-MANJARO
drwxr-xr-x 1 root root 514 12. Dez 12:04 5.14.18-1-MANJARO
drwxr-xr-x 1 root root 514 25. Jan 19:03 5.15.16-1-MANJARO
drwxr-xr-x 1 root root 104 25. Jan 19:00 extramodules-4.19-MANJARO
drwxr-xr-x 1 root root 104 25. Jan 19:00 extramodules-5.10-MANJARO
drwxr-xr-x 1 root root 104 25. Jan 19:01 extramodules-5.15-MANJARO


mhwd-kernel -li                                                                                                                                  
Currently running: 5.10.93-1-MANJARO (linux510)
The following kernels are installed in your system:
   * linux419
   * linux510
   * linux515

And the modules are not cleaned up

I did not enable/disable any service of kernel-modules !

I’m glad I’m not alone in my experience, and I also added a few more details to my post before this one to try keep my thoughts/findings together.

Hopefully someone can confirm/correct them and provide the next step(s).

Be carefull !

It seems

kernel-alive

contains a service linux-module-cleanup.service

kernel-modules-hook

contains a service linux-modules-cleanup.service

:rofl:

1 Like

Updates are not to be automated or blindly applied and the packages creates a false sense of security as you have seen.

In my opinion both the kernel-alive and kernel-modules-hook has very little real life application as it can create issues which should not have been in the first place if the system was rebooted after a kernel sync.

The problem with kernel-modules-hook is that it targets Archlinux - and Arch kernel naming is different than Manjaro - which will cause problems on Manjaro when you have more than one kernel installed.

Hmm, if that’s correct, then (while using kernel-modules-hook) it seems to support my evolving thinking that running $ systemctl enable linux-modules-cleanup.service was important for it’s symlink to be created… Created symlink /etc/systemd/system/basic.target.wants/linux-modules-cleanup.service → /usr/lib/systemd/system/linux-modules-cleanup.service.

And since kernel-alive was uninstalled (in my case), deleting that broken link isn’t going to create any problems… in fact, systemd might appreciate not chasing broken links.

This is right, BUT the service in kernel-modules-hook does fit manjaro naming. (/usr/lib/modules/[0-9]*)

cat ./linux-modules-cleanup.service
[Unit]
Description=Clean up modules from old kernels

[Service]
Type=oneshot
ExecStart=/bin/bash -exc 'for i in /usr/lib/modules/[0-9]*; do if [[ $${i##*/} = \'%v\' ]] || pacman -Qo "$${i}"; then continue; fi; rsync -AHXal "$${i}" /usr/lib/modules/.old/; rm -rf "$${i}"; done'

[Install]
WantedBy=basic.target

BUT it seems disabled by default :frowning:

systemctl status linux-modules-cleanup.service                                                                                                   
○ linux-modules-cleanup.service - Clean up modules from old kernels
     Loaded: loaded (/usr/lib/systemd/system/linux-modules-cleanup.service; disabled; vendor preset: disabled)
     Active: inactive (dead)
enable linux-modules-cleanup.service                                                                                                
Created symlink /etc/systemd/system/basic.target.wants/linux-modules-cleanup.service → /usr/lib/systemd/system/linux-modules-cleanup.service.
start linux-modules-cleanup.service
status linux-modules-cleanup.service                                                                                                   
○ linux-modules-cleanup.service - Clean up modules from old kernels
     Loaded: loaded (/usr/lib/systemd/system/linux-modules-cleanup.service; enabled; vendor preset: disabled)
     Active: inactive (dead) since Sun 2022-02-06 08:46:07 CET; 5s ago
    Process: 2915704 ExecStart=/bin/bash -exc for i in /usr/lib/modules/[0-9]*; do if [[ $${i##*/} = '5.10.93-1-MANJARO' ]] || pacman -Qo "$${i}"; then cont>
   Main PID: 2915704 (code=exited, status=0/SUCCESS)
        CPU: 13.498s

Feb 06 08:46:05 manjaro18 bash[2915704]: + rsync -AHXal /usr/lib/modules/5.14.18-1-MANJARO /usr/lib/modules/.old/
Feb 06 08:46:06 manjaro18 bash[2915704]: + rm -rf /usr/lib/modules/5.14.18-1-MANJARO
Feb 06 08:46:07 manjaro18 bash[2915704]: + for i in /usr/lib/modules/[0-9]*
Feb 06 08:46:07 manjaro18 bash[2915704]: + [[ 5.15.16-1-MANJARO = \5\.\1\0\.\9\3\-\1\-\M\A\N\J\A\R\O ]]
Feb 06 08:46:07 manjaro18 bash[2915704]: + pacman -Qo /usr/lib/modules/5.15.16-1-MANJARO
Feb 06 08:46:07 manjaro18 bash[2916298]: /usr/lib/modules/5.15.16-1-MANJARO/ ist in linux515 5.15.16-1 enthalten
Feb 06 08:46:07 manjaro18 bash[2915704]: + continue
Feb 06 08:46:07 manjaro18 systemd[1]: linux-modules-cleanup.service: Deactivated successfully.
Feb 06 08:46:07 manjaro18 systemd[1]: Finished Clean up modules from old kernels.
Feb 06 08:46:07 manjaro18 systemd[1]: linux-modules-cleanup.service: Consumed 13.498s CPU time.
1 Like

Thank you for chiming in @linux-aarhus

So if I were to uninstall kernel-modules-hook (kernel-alive is already uninstalled), and if @andreas85 is correct that both /etc/systemd/system/basic.target.wants/ files are related to these two packages…

$ ls -la /etc/systemd/system/basic.target.wants/
total 8
drwxr-xr-x  2 root root 4096 Feb  6 00:17 .
drwxr-xr-x 11 root root 4096 Nov 19 10:51 ..
lrwxrwxrwx  1 root root   48 Jun 14  2021 linux-module-cleanup.service -> /etc/systemd/system/linux-module-cleanup.service
lrwxrwxrwx  1 root root   53 Feb  6 00:17 linux-modules-cleanup.service -> /usr/lib/systemd/system/linux-modules-cleanup.service

I suspect the right thing to do (with both packages uninstalled) is to remove both symlinks. But if I were to do that, what keeps my /lib/modules folder clean?

The packages in question interferes with the default sync of kernel and modules.

By removing those packages you will just revert to the default handler.

If you want to clean your system for references to other kernels note which kernels are installed

mhwd-kernel -li

Remove all except the currently running kernel - verify by running

uname -r

You can use pacman or mhwd-kernel - the latter will remove extra package like proprietary nvidia and virtualbox-host modules - repeat as many times as necessary

mhwd -r linuxXX

Reboot your system to the only kernel - then locate remnants or other kernels and remove them. You can recognize files or folders by their naming.

Okay, I’m willing to try/learn, as there is definitely much to learn here. I’ve uninstalled kernel-modules-hook because I wanted to prove my thought that another broken symlink would be the result; and perhaps also prove @andreas85 was right about the linux-modules-cleanup.service file belonging to kernel-modules-hook… and I scored a bingo!

I think this also suggests that the kernel-modules-hook install missed creating the symlink to run it’s service (or as @andreas85 pointed out, left the service disabled)… and that my $ systemctl enable linux-modules-cleanup.service command indeed would have corrected that miss.

So I am going to pause until the next kernel 5.15.x update so that I can see/confirm that “other default handlers” kicked in and pruned the previous 5.15 kernel’s modules folder.

What scares me about pursuing this option is one question/thought… I’m sure these packages exist because someone ran into an issue these packages resolved (i.e. kernel-modules-hook states that it “keeps your system fully functional after a kernel update”)… so what issues might I be opening myself up to?

Trying to think about why I might have a need for either of these applications, I came to one thought… that delaying rebooting after applying an update (perhaps when running a server) could have some consequences… and if that’s true, I’d never run into those consequences as I always reboot my desktop (not a server) after every major update I apply (and they usually request it).

One final question though… it feels right to drop these broken symlinks… any reason not to?

The removal of the package could have removed then but the package doesn’t really know if the service was enabled or not - so before removing a package with a service which has been enabled manually one should disable the service - which will remove the symlink.

sudo systemctl disable --now some.service

When the package has been removed before the service was disabled - you can still remove the symlink using

sudo systemctl disable --now linux-modules-cleanup.service linux-module-cleanup.service

You may get a message - you can safely ignore it.

1 Like

Great knowledge, thank you for sharing! I have no idea how it knew what symlinks to remove with the packages uninstalled, but that command worked perfectly.

$ sudo systemctl disable --now linux-modules-cleanup.service linux-module-cleanup.service
Removed /etc/systemd/system/basic.target.wants/linux-modules-cleanup.service.
Removed /etc/systemd/system/basic.target.wants/linux-module-cleanup.service.
Failed to stop linux-modules-cleanup.service: Unit linux-modules-cleanup.service not loaded.
Failed to stop linux-module-cleanup.service: Unit linux-module-cleanup.service not loaded.

And as in my case as I only had these two service files, it also purged the folder they were in…

$ ls -la /etc/systemd/system/basic.target.wants/
ls: cannot access '/etc/systemd/system/basic.target.wants/': No such file or directory

You are right - doing your own thinking - making an informed choice - that’s is the way to go.

2 Likes

@andreas85 if you are willing to participate as the other side of the choice coin… I’ll stay as I am with kernel-module-hook uninstalled as it looks like you’ve enabled its cleanup service… and we’ll compare notes after the next 5.15 kernel update (so long as your not jumping to 5.16 before then).

If enabling the clean-up service resolved the “rogue” module folders for you then I’ll use that experience to mark this thread as resolved.

I will of course add my experience so people can see the outcome of both choices (default handlers versus kernel-modules-hook with its service enabled), and make an informed choice for themselves.

1 Like

I’m not feeling “scared” after reading three posts from the Please use kernel-modules-hook instead of kernel-alive! thread. I was still bright green back then and so much was going over my head… only slightly less bright green now :wink:

I) kernel-alive was removed as a default from all ISO’s that contained it… (kernel-modules-hook was never a default)

II) keeping it included could be “counterproductive for troubleshooting”… (I presume the same could be said for kernel-modules-hook if it was defaulted)

III) the “why it was used” explanation that must have planted the seeds I recalled and paraphrased here… (P.S. I finally read/understand the “root tip” now :slight_smile: )

I will try to describe some of my findings

When you update your kernel, the “old” kernel is removed, and also its modules. And it is replaced with a new kernel and new modules. So far so good. But while this is done and until you reboot this “old” kernel is in RAM and running.

These Problems may result:

When a module needs to be loaded into the “old” kernel

If for any reason there is a need to load an additional module into the running kernel, these modules need to be available.
When a “new” module would be tried, this will fail because the kernel detects this is none of my modules :wink:
I do not know for sure when this may happen, but someone pointed out that it may take place when detecting a specific filesystem on USB that is inserted.

When this “old” kernel needs to be reloaded

You may think the kernel is in memory, why should it be loaded again?
If your system hibernates, the RAM is written onto disk. This includes the “old” kernel.
It seems that it is necessary when the system wakes. There is a exact copy of the running kernel (/boot/vmlinuz-5.15-x86_64) at /lib/modules/5.15.16-1-MANJARO/vmlinuz . This kernel seems to be used when waking from hibernation. If it is missing, waking may fail. And this is inside modules directory …

  • If you reboot after the update there will be no problem at all
  • If your pc is used after the update without rebooting (like a server, or when used as multiseat) you better keep using kernel-modules-hook. But enable the cleanup :wink:
1 Like

So today was a new stable update with a shiny new kernel… and I was finally able to see whether or not the /lib/modules folder was cleared of the previous kernel’s folders/files in a “default” configuration… i.e. without kernel-alive and kernel-modules-hook. And the verdict is… yes it was!

$ ls -la /lib/modules 
total 280
drwxr-xr-x  11 root root   4096 Feb 14 13:05 .
drwxr-xr-x 240 root root 241664 Feb 14 13:05 ..
drwxr-xr-x   3 root root   4096 Feb 14 13:05 5.10.98-1-MANJARO
drwxr-xr-x   3 root root   4096 Nov 29 21:01 5.13.19-2-MANJARO
drwxr-xr-x   3 root root   4096 Dec 10 14:52 5.14.21-2-MANJARO
drwxr-xr-x   3 root root   4096 Feb 14 13:05 5.15.21-1-MANJARO
drwxr-xr-x   2 root root   4096 Feb 14 13:05 extramodules-5.10-MANJARO
drwxr-xr-x   2 root root   4096 Nov 29 21:01 extramodules-5.13-MANJARO
drwxr-xr-x   2 root root   4096 Dec 10 14:52 extramodules-5.14-MANJARO
drwxr-xr-x   2 root root   4096 Feb 14 13:05 extramodules-5.15-MANJARO
drwxr-xr-x   5 root root   4096 Feb  5 11:10 .old

Huzzah, only one 5.15.* kernel folder!

This did however show a minor necessary manual cleanup/removal of the /lib/modules/.old folder left behind by the uninstall of kernel-modules-hook with the command $ sudo rm -dr /lib/modules/.old/

$ ls -la /lib/modules
total 276
drwxr-xr-x  10 root root   4096 Feb 14 16:58 .
drwxr-xr-x 240 root root 241664 Feb 14 13:05 ..
drwxr-xr-x   3 root root   4096 Feb 14 13:05 5.10.98-1-MANJARO
drwxr-xr-x   3 root root   4096 Nov 29 21:01 5.13.19-2-MANJARO
drwxr-xr-x   3 root root   4096 Dec 10 14:52 5.14.21-2-MANJARO
drwxr-xr-x   3 root root   4096 Feb 14 13:05 5.15.21-1-MANJARO
drwxr-xr-x   2 root root   4096 Feb 14 13:05 extramodules-5.10-MANJARO
drwxr-xr-x   2 root root   4096 Nov 29 21:01 extramodules-5.13-MANJARO
drwxr-xr-x   2 root root   4096 Dec 10 14:52 extramodules-5.14-MANJARO
drwxr-xr-x   2 root root   4096 Feb 14 13:05 extramodules-5.15-MANJARO

How did your test go with kernel-modules-hook after enabling ($ systemctl enable linux-modules-cleanup.service) it’s service @andreas85?

There’s been a few weeks of idleness, so I think it’s time to put together the best solution post I can… as the solution is multi-fold depending on your course of action.

To answer the original question/issue about why clean-up in the /lib/modules/ folder was not happening… it really boils down to knowing the tool you are trying to use; whether that be kernel-alive or (in my case) kernel-modules-hook. As it turns out, even though I’d installed the package, I had not started/enabled its service for it to be used/called…

I feel that step alone would have resolved my original question… but was kernel-modules-hook even required in my case?

So seeing as I always reboot after applying system updates manually, tools like kernel-alive and kernel modules-hook aren’t required… and in my use case, the best solution was to remove the package/service entirely.

  1. Don’t forget to shutdown the services; which will purge its symlinks before removal
  1. and uninstall the application/service…

Did uninstalling kernel-modules-hook enable/allow the default hooks/process during an update to clean-up the /lib/modules folder? Yes it did! (with one minor clean-up required; a folder created by the package)

version 5.13 & 5.14 should be removed

sudo mhwd-kernel -r linux513 
sudo mhwd-kernel -r linux514
sudo ls -la /lib/modules