Boot fails after latest Manjaro update / systemd

After an update a week ago the system does not boot any longer (currently working via USB stick boot). The situation is very weird, actually everything is setup correct, still mounting of /boot and /boot/efi does not work. Basic situation:

[manjaro-cinnamon /]# cat /etc/fstab 
# /etc/fstab: static file system information.
#
# <file system>                 <mount point>	<type>	<options>					<dump>	<fsck>
/dev/mapper/root		/		ext4	defaults,noatime				0	1
/dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a	/boot	ext2	defaults,noatime,x-systemd.device-timeout=10s	0	1
/dev/disk/by-partuuid/a875fad4-ea8b-4777-af84-2824aea21320	/boot/efi	vfat	umask=0077,x-systemd.device-timeout=10s		0	1


[manjaro-cinnamon /]# cat /etc/crypttab
# /etc/crypttab: mappings for encrypted partitions.
#
#
# <name>		<device>					<password>		<options>
root	  		/dev/disk/by-partuuid/b29f47cf-bf7c-4b01-998c-220a0ad4745d	-	plain,cipher=aes-cbc-essiv:sha256,hash=ripemd160,size=256,discard



[manjaro-cinnamon /]# ls -l /dev/disk/by-partuuid/
total 0
lrwxrwxrwx 1 root root 15 Apr 20 18:57 71f8f961-d84b-48b4-9f03-ed2fef926e5a -> ../../nvme1n1p2
lrwxrwxrwx 1 root root 15 Apr 20 18:57 7735d644-2341-49d8-992b-ae2dd9b224fd -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Apr 20 18:57 a875fad4-ea8b-4777-af84-2824aea21320 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 15 Apr 20 18:57 b29f47cf-bf7c-4b01-998c-220a0ad4745d -> ../../nvme1n1p3


[manjaro-cinnamon /]# journalctl -b -2 -p err
Apr 18 08:21:55 archlinux kernel: x86/cpu: SGX disabled by BIOS.
Apr 18 08:22:32 xeus2k1 systemd[1]: Timed out waiting for device /dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a.
Apr 18 08:22:32 xeus2k1 systemd[1]: Timed out waiting for device /dev/disk/by-partuuid/a875fad4-ea8b-4777-af84-2824aea21320.
Apr 18 08:22:32 xeus2k1 systemd[1]: Failed to start Generate shutdown-ramfs.


[manjaro-cinnamon /]# journalctl -b -2
...
Apr 18 08:21:56 archlinux (udev-worker)[273]: nvme1n1p2: Successfully created symlink '/dev/disk/by-label/boot' to '/dev/nvme1n1p2'
Apr 18 08:21:56 archlinux (udev-worker)[273]: nvme1n1p2: Successfully created symlink '/dev/disk/by-id/nvme-ADATA_LEGEND_960_2N112L251AP9_1-part2' to '/dev/nvme1n1p2'
Apr 18 08:21:56 archlinux (udev-worker)[270]: nvme1n1p3: Successfully created symlink '/dev/disk/by-path/pci-0000:73:00.0-nvme-1-part3' to '/dev/nvme1n1p3'
Apr 18 08:21:56 archlinux (udev-worker)[273]: nvme1n1p2: Successfully created symlink '/dev/disk/by-id/nvme-ADATA_LEGEND_960_2N112L251AP9-part2' to '/dev/nvme1n1p2'
Apr 18 08:21:56 archlinux (udev-worker)[273]: nvme1n1p2: Successfully created symlink '/dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a' to '/dev/nvme1n1p2'
Apr 18 08:21:56 archlinux (udev-worker)[270]: nvme1n1p3: Successfully created symlink '/dev/disk/by-id/nvme-ADATA_LEGEND_960_2N112L251AP9_1-part3' to '/dev/nvme1n1p3'
Apr 18 08:21:56 archlinux (udev-worker)[273]: nvme1n1p2: Successfully created symlink '/dev/disk/by-uuid/743c3dd0-0af9-4e29-8157-938027deb56f' to '/dev/nvme1n1p2'
Apr 18 08:21:56 archlinux (udev-worker)[270]: nvme1n1p3: Successfully created symlink '/dev/disk/by-diskseq/2-part3' to '/dev/nvme1n1p3'
Apr 18 08:21:56 archlinux (udev-worker)[273]: nvme1n1p2: Successfully created symlink '/dev/disk/by-partlabel/boot' to '/dev/nvme1n1p2'
Apr 18 08:21:56 archlinux (udev-worker)[270]: nvme1n1p3: Successfully created symlink '/dev/disk/by-partlabel/manjaro_root' to '/dev/nvme1n1p3'
Apr 18 08:21:56 archlinux (udev-worker)[273]: nvme1n1p2: Successfully created symlink '/dev/disk/by-diskseq/2-part2' to '/dev/nvme1n1p2'
...
Apr 18 08:22:32 xeus2k1 systemd[1]: dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device: Job dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device/start timed out.
Apr 18 08:22:32 xeus2k1 systemd[1]: dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device: Job 164 dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device/start finish>
Apr 18 08:22:32 xeus2k1 systemd[1]: Timed out waiting for device /dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a.
Apr 18 08:22:32 xeus2k1 systemd[1]: systemd-fsck@dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.service: Job 162 systemd-fsck@dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fe>
Apr 18 08:22:32 xeus2k1 systemd[1]: Dependency failed for File System Check on /dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a.
Apr 18 08:22:32 xeus2k1 systemd[1]: boot.mount: Job 161 boot.mount/start finished, result=dependency
Apr 18 08:22:32 xeus2k1 systemd[1]: Dependency failed for /boot.
Apr 18 08:22:32 xeus2k1 systemd[1]: boot-efi.mount: Job 165 boot-efi.mount/start finished, result=dependency
Apr 18 08:22:32 xeus2k1 systemd[1]: Dependency failed for /boot/efi.
Apr 18 08:22:32 xeus2k1 systemd[1]: local-fs.target: Job 160 local-fs.target/start finished, result=dependency
Apr 18 08:22:32 xeus2k1 systemd[1]: Dependency failed for Local File Systems.
Apr 18 08:22:32 xeus2k1 systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
Apr 18 08:22:32 xeus2k1 systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
Apr 18 08:22:32 xeus2k1 systemd[1]: emergency.target: Trying to enqueue job emergency.target/start/replace-irreversibly

I do not understand why this fails, for me everything seems correct, the symlink is created by the kernel, still, later systemd complains that it can’t find the device. What do I miss? It looks as if systemd misses the uevent somehow.

System is up-to-date:

[manjaro-cinnamon /]# pacman -Syyu
:: Synchronizing package databases...
 core                                                                                                  149.3 KiB   447 KiB/s 00:00 [################################################################################] 100%
 extra                                                                                                   8.7 MiB  19.7 MiB/s 00:00 [################################################################################] 100%
 community                                                                                              29.0   B  2.83 KiB/s 00:00 [################################################################################] 100%
 multilib                                                                                              143.9 KiB  10.8 MiB/s 00:00 [################################################################################] 100%
:: Starting full system upgrade...
 there is nothing to do

I’m running linux Kernel 6.8.5.1, but I tried downgrade of systemd and kernel to 6.6 without any success, the problems stays. Any hint or suggestion would be great, thank you!

P.S.: From the emergency shell, i can do mount /boot and mount /boot/efi without any problem, it works. Mounting the root system works as well (after entering the password it get mounted and is available on the emergency shell).

That repo was deprecated almost a year ago. You aren’t maintaining your system. At the very least you should learn how to merge your pacnew files.

4 Likes

P.S.: This is suspicious for me, but I do not yet fully understand what happens:

Apr 18 08:22:32 xeus2k1 systemd-fstab-generator[381]: Parsing /etc/fstab...
Apr 18 08:22:32 xeus2k1 systemd-fstab-generator[381]: Found entry what=/dev/mapper/root where=/ type=ext4 makefs=no growfs=no pcrfs=no noauto=no nofail=no
Apr 18 08:22:32 xeus2k1 systemd-gpt-auto-generator[383]: Disabling root partition auto-detection, root= is defined.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-getty-generator succeeded.
Apr 18 08:22:32 xeus2k1 systemd-hibernate-resume-generator[384]: Not running in an initrd, exiting.
Apr 18 08:22:32 xeus2k1 systemd-fstab-generator[381]: Found entry what=/dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a where=/boot type=ext2 makefs=no growfs=no pcrfs=no noauto=no nofail=no
Apr 18 08:22:32 xeus2k1 systemd-fstab-generator[381]: Found entry what=/dev/disk/by-partuuid/a875fad4-ea8b-4777-af84-2824aea21320 where=/boot/efi type=vfat makefs=no growfs=no pcrfs=no noauto=no nofail=no
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-integritysetup-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-run-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-debug-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-system-update-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/nfs-server-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-bless-boot-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/rpc-pipefs-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-fstab-generator succeeded.
Apr 18 08:22:32 xeus2k1 systemd-gpt-auto-generator[383]: Reading EFI variable /sys/firmware/efi/efivars/LoaderDevicePartUUID-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/ostree-system-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-cryptsetup-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-hibernate-resume-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-gpt-auto-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-execu[374]: /usr/lib/systemd/system-generators/systemd-veritysetup-generator succeeded.
Apr 18 08:22:32 xeus2k1 (sd-gens)[373]: (sd-executor) succeeded.
Apr 18 08:22:32 xeus2k1 systemd[1]: (sd-gens) succeeded.
Apr 18 08:22:32 xeus2k1 systemd[1]: Looking for unit files in (higher priority first):
Apr 18 08:22:32 xeus2k1 systemd[1]:         /etc/systemd/system.control
Apr 18 08:22:32 xeus2k1 systemd[1]:         /run/systemd/system.control
Apr 18 08:22:32 xeus2k1 systemd[1]:         /run/systemd/transient
Apr 18 08:22:32 xeus2k1 systemd[1]:         /run/systemd/generator.early
Apr 18 08:22:32 xeus2k1 systemd[1]:         /etc/systemd/system
Apr 18 08:22:32 xeus2k1 systemd[1]:         /etc/systemd/system.attached
Apr 18 08:22:32 xeus2k1 systemd[1]:         /run/systemd/system
Apr 18 08:22:32 xeus2k1 systemd[1]:         /run/systemd/system.attached
Apr 18 08:22:32 xeus2k1 systemd[1]:         /run/systemd/generator
Apr 18 08:22:32 xeus2k1 systemd[1]:         /usr/local/lib/systemd/system
Apr 18 08:22:32 xeus2k1 systemd[1]:         /usr/lib/systemd/system
Apr 18 08:22:32 xeus2k1 systemd[1]:         /run/systemd/generator.late
Apr 18 08:22:32 xeus2k1 systemd[1]: Modification times have changed, need to update cache.
Apr 18 08:22:32 xeus2k1 systemd[1]: systemd-fsck-silent@.service: unit self-alias: systemd-fsck-silent@.service <E2><86><92> systemd-fsck-silent@.service, ignoring.
Apr 18 08:22:32 xeus2k1 systemd[1]: unit_file_build_name_map: normal unit file: /etc/systemd/system/second.service
Apr 18 08:22:32 xeus2k1 systemd[1]: systemd-fsck-silent-root.service: unit self-alias: systemd-fsck-silent-root.service <E2><86><92> systemd-fsck-silent-root.service, ignoring.
Apr 18 08:22:32 xeus2k1 systemd[1]: unit_file_build_name_map: normal unit file: /run/systemd/generator/-.mount
Apr 18 08:22:32 xeus2k1 systemd[1]: unit_file_build_name_map: normal unit file: /run/systemd/generator/systemd-cryptsetup@root.service
Apr 18 08:22:32 xeus2k1 systemd[1]: unit_file_build_name_map: normal unit file: /run/systemd/generator/boot.mount
Apr 18 08:22:32 xeus2k1 systemd[1]: unit_file_build_name_map: normal unit file: /run/systemd/generator/boot-efi.mount
Apr 18 08:22:32 xeus2k1 systemd[1]: unit_file_build_name_map: normal unit file: /usr/lib/systemd/system/systemd-localed.service
...
Apr 18 08:22:32 xeus2k1 systemd[1]: unit_file_build_name_map: normal unit file: /usr/lib/systemd/system/virtsecretd-admin.socket
Apr 18 08:22:32 xeus2k1 systemd[1]: Too many messages being logged to kmsg, ignoring
Apr 18 08:22:32 xeus2k1 kernel: loop: module loaded
Apr 18 08:22:32 xeus2k1 kernel: fuse: init (API version 7.39)
Apr 18 08:22:32 xeus2k1 kernel: EXT4-fs (dm-0): re-mounted 8a193d5b-5afc-4c16-aacf-97036e9cc99d r/w. Quota mode: none.
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.480:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="dovecot-anvil" pid=444 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.480:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="klogd" pid=438 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.490:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="torbrowser_tor" pid=442 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.494:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=431 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.494:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=431 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.494:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=430 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.497:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="dovecot-config" pid=446 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.500:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="dovecot-deliver" pid=447 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.500:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="syslogd" pid=440 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 kernel: audit: type=1400 audit(1713421323.504:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="samba-rpcd" pid=435 comm="apparmor_parser"
Apr 18 08:22:32 xeus2k1 systemd[1]: dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device: Job dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device/start timed out.
Apr 18 08:22:32 xeus2k1 systemd[1]: dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device: Job 164 dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device/start finished, result=timeout
Apr 18 08:22:32 xeus2k1 systemd[1]: Timed out waiting for device /dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a.
Apr 18 08:22:32 xeus2k1 systemd[1]: systemd-fsck@dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.service: Job 162 systemd-fsck@dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.service/start finished, result=dependency
Apr 18 08:22:32 xeus2k1 systemd[1]: Dependency failed for File System Check on /dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a.
Apr 18 08:22:32 xeus2k1 systemd[1]: boot.mount: Job 161 boot.mount/start finished, result=dependency
Apr 18 08:22:32 xeus2k1 systemd[1]: Dependency failed for /boot.
Apr 18 08:22:32 xeus2k1 systemd[1]: boot-efi.mount: Job 165 boot-efi.mount/start finished, result=dependency
Apr 18 08:22:32 xeus2k1 systemd[1]: Dependency failed for /boot/efi.
Apr 18 08:22:32 xeus2k1 systemd[1]: local-fs.target: Job 160 local-fs.target/start finished, result=dependency
Apr 18 08:22:32 xeus2k1 systemd[1]: Dependency failed for Local File Systems.
Apr 18 08:22:32 xeus2k1 systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
Apr 18 08:22:32 xeus2k1 systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
Apr 18 08:22:32 xeus2k1 systemd[1]: emergency.target: Trying to enqueue job emergency.target/start/replace-irreversibly

For me, it seems like systemd parses the fstab file at 8:22:32 and then in the same second clains the timeout?

This user seems to describe the exact same problem (sadly, without a solution, except for reinstallation):

https://unix.stackexchange.com/questions/748137/timed-out-waiting-for-device-efi-partition-xubuntu-22-04-lts

Is a long gone package that does not exist in the repos or the AUR.

This indicates you have unsupported (and incompatible) packages.

I might guess there are further problems like unmanaged pacnews.

(to print them all you can use pacdiff -o)

1 Like
[manjaro-cinnamon /]# pacdiff -o
/etc/passwd.pacnew
/etc/shells.pacnew
/etc/locale.gen.pacnew
/etc/default/grub.pacnew
/etc/libvirt/qemu.conf.pacnew
/etc/lightdm/lightdm.conf.pacnew
/etc/mkinitcpio.conf.pacnew
/etc/qemu/bridge.conf.pacnew

How can I find out which packages are outdated? I do all the updates when they come and only modified minor things like locals or libvirt and mkinitcpio.conf, but I needed this, because I have not found any solution to escape a colon in kernel parameters and sadly the encryption is an older one with :sha256. The diff for mkinitcpio.conf is

[manjaro-cinnamon /]# diff /etc/mkinitcpio.conf /etc/mkinitcpio.conf.pacnew 
6,7c6,7
< #     MODULES=(piix ide_disk reiserfs)
< MODULES=""
---
> #     MODULES=(usbhid xhci_hcd)
> MODULES=()
19c19
< FILES=""
---
> FILES=()
32c32
< ##   No raid, lvm2, or encrypted root is needed.
---
> ##   No RAID, lvm2, or encrypted root is needed.
37c37
< #    HOOKS=(base udev autodetect block filesystems)
---
> #    HOOKS=(base udev autodetect modconf block filesystems fsck)
41c41
< #    HOOKS=(base udev block filesystems)
---
> #    HOOKS=(base udev modconf block filesystems fsck)
43,45c43,45
< ##   This setup assembles a pata mdadm array with an encrypted root FS.
< ##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
< #    HOOKS=(base udev block mdadm encrypt filesystems)
---
> ##   This setup assembles a mdadm array with an encrypted root file system.
> ##   Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
> #    HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
47,48c47,48
< ##   This setup loads an lvm2 volume group on a usb device.
< #    HOOKS=(base udev block lvm2 filesystems)
---
> ##   This setup loads an lvm2 volume group.
> #    HOOKS=(base udev modconf block lvm2 filesystems fsck)
51,54c51,52
< #    usr, fsck and shutdown hooks.
< #HOOKS="base udev autodetect modconf block keyboard keymap resume filesystems fsck"
< #HOOKS="base udev autodetect modconf block keyboard keymap encrypt filesystems fsck"
< HOOKS="systemd modconf block keyboard sd-vconsole sd-encrypt filesystems fsck"
---
> #    usr and fsck hooks.
> HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck)
64c62
< COMPRESSION="lz4"
---
> #COMPRESSION="lz4"
69a68,73
> 
> # MODULES_DECOMPRESS
> # Decompress kernel modules during initramfs creation.
> # Enable to speedup boot process, disable to save RAM
> # during early userspace. Switch (yes/no).
> #MODULES_DECOMPRESS="yes"


[manjaro-cinnamon /]# diff /etc/shells /etc/shells.pacnew
6,8d5
< /bin/zsh
< /usr/bin/zsh
< /usr/bin/git-shell
10,11d6
< /usr/bin/rbash
< /usr/bin/bash
12a8,10
> /usr/bin/bash
> /usr/bin/rbash
> 

[manjaro-cinnamon /]# diff /etc/shells /etc/shells.pacnew
6,8d5
< /bin/zsh
< /usr/bin/zsh
< /usr/bin/git-shell
10,11d6
< /usr/bin/rbash
< /usr/bin/bash
12a8,10
> /usr/bin/bash
> /usr/bin/rbash
> 

I have not even any clue why I have the shell.conf.pacnew, never touched it.

Those should probably be handled.
The wiki (and other threads on the forum) will help you do so if you dontk now how.
A common approach might be

DIFFPROG=meld pacdiff -s

Or for more GUI friendliness you can use manjaro-pacnew-checker.

The packages are still an issue as well. And there may be more than systemd-fsck-silent.
You can similarly print all ‘foreign’ packages;

pacman -Qmq

pacnew files are created when there are changes you need to manually review and merge.

Please consult the wiki.

The pacman command above will print foreign packages.

My previous comment was about hanging on to a ‘bad’ package, not that your packages were out of date.
We can check with sorting mirrors and rseyncing I suppose;

sudo pacman-mirrors -f && sudo pacman -Syu

I know how to use meld, except for the shell I personally do not see any need to further merge (I just keep the pacnew files to review later).

[manjaro-cinnamon /]# pacman -Qmq
celt
clementine
debtap
electron12
electron13
electron17
electron18
electron19
etcher
foo2zjs-nightly
gcolor2
gklib
gmime
gnome-system-log
grub-theme-manjaro-dev
ipw2100-fw
ipw2200-fw
js78
lib32-faudio
lib32-libwrap
lib32-openssl-1.0
libsidplay
libwrap
manjaro-documentation-en
manjaro-firmware
metis
openssl-1.0
pidgin-libnotify
programmer-calculator
progsreiserfs
python-pep517
python-progress
python2
python2-ptyprocess
systemd-fsck-silent
xcursor-chameleon-pearl-deepin
xplayer
xplayer-plparser

I understand when conflicts appear, just, when I never touched a file do not understand, why I get a conflict that I need to merge, anyway, old packages are more likely to be a problem, just wonder how to find this out.

All of those are ‘bad’ aliens … they dont exist in the repos or the AUR.

I dont know if you need to make sure to have systemd replace systemd-fsck-silent or if it just needs to be removed.
Maybe similar for ipw* vs the currently available AUR package ipw2x00-firmware.
etc.

Lets see, I removed the packages and updated systemd, it rebuild initramfs, i as well merged most of the pacnew files, except for those where I cant merge (like passwd, …). Let me reboot and see

Thank you so far, sadly, this did not help, still the problem stays.

I now simply commented out /boot and /boot/efi in fstabs and copied the content to root, which mounts successful, just for testing. Now the boot moves on and fails at other jobs:

[manjaro-cinnamon /]# journalctl -S "2024-04-21 22:50:38" --priority=5
Apr 21 22:50:38 archlinux kernel: Linux version 6.8.5-1-MANJARO (builduser@fv-az846-576) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.42.0) #1 SMP PREEMPT_DYNAMIC Wed Apr 10 20:15:45 UTC 2024
Apr 21 22:50:38 archlinux kernel: random: crng init done
Apr 21 22:50:38 archlinux kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-6.8-x86_64 root=/dev/mapper/root rw apparmor=1 security=apparmor systemd.log_level=debug
Apr 21 22:50:38 archlinux kernel: Unknown kernel command line parameters "BOOT_IMAGE=/vmlinuz-6.8-x86_64", will be passed to user space.
Apr 21 22:50:38 archlinux kernel: x86/cpu: SGX disabled by BIOS.
Apr 21 22:50:38 archlinux kernel: MMIO Stale Data CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/processor_mmio_stale_data.html for more details.
Apr 21 22:50:38 archlinux kernel: audit: type=2000 audit(1713732637.066:1): state=initialized audit_enabled=0 res=1
Apr 21 22:50:38 archlinux kernel: SCSI subsystem initialized
Apr 21 22:50:38 archlinux kernel: VFS: Disk quotas dquot_6.6.0
Apr 21 22:50:38 archlinux kernel: Initialise system trusted keyrings
Apr 21 22:50:38 archlinux kernel: Key type blacklist registered
Apr 21 22:50:38 archlinux kernel: integrity: Platform Keyring initialized
Apr 21 22:50:38 archlinux kernel: integrity: Machine keyring initialized
Apr 21 22:50:38 archlinux kernel: Key type asymmetric registered
Apr 21 22:50:38 archlinux kernel: Asymmetric key parser 'x509' registered
Apr 21 22:50:38 archlinux kernel: Loading compiled-in X.509 certificates
Apr 21 22:50:38 archlinux kernel: Loaded X.509 cert 'Build time autogenerated kernel key: 954f3e5c74778db1297a5d36ed89ee57fcb3f17f'
Apr 21 22:50:38 archlinux kernel: Key type .fscrypt registered
Apr 21 22:50:38 archlinux kernel: Key type fscrypt-provisioning registered
Apr 21 22:50:39 archlinux kernel: usb: port power management may be unreliable
Apr 21 22:50:42 archlinux kernel: scsi 4:0:0:0: Direct-Access     JetFlash Transcend 128GB  1100 PQ: 0 ANSI: 6
Apr 21 22:50:42 archlinux kernel: sd 4:0:0:0: [sda] 241637376 512-byte logical blocks: (124 GB/115 GiB)
Apr 21 22:50:42 archlinux kernel: sd 4:0:0:0: [sda] Write Protect is off
Apr 21 22:50:42 archlinux kernel: sd 4:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr 21 22:50:42 archlinux kernel: sd 4:0:0:0: [sda] Attached SCSI disk
Apr 21 22:50:44 archlinux kernel: Key type trusted registered
Apr 21 22:50:44 archlinux kernel: Key type encrypted registered
Apr 21 22:50:45 archlinux systemd[1]: Switching root.
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732645.997:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="dovecot-anvil" pid=445 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732645.997:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="klogd" pid=439 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732646.010:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="samba-rpcd-spoolss" pid=438 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732646.013:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=431 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732646.013:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=432 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732646.013:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=432 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732646.017:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ping" pid=429 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732646.017:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="torbrowser_tor" pid=443 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732646.020:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="dovecot-config" pid=447 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 kernel: audit: type=1400 audit(1713732646.020:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="syslogd" pid=441 comm="apparmor_parser"
Apr 21 22:57:26 xeus2k1 systemd[1]: sys-subsystem-net-devices-enp113s0.device: Job sys-subsystem-net-devices-enp113s0.device/start timed out.
Apr 21 22:57:26 xeus2k1 systemd[1]: Timed out waiting for device /sys/subsystem/net/devices/enp113s0.
Apr 21 22:57:26 xeus2k1 systemd[1]: Dependency failed for br0.
Apr 21 22:57:26 xeus2k1 systemd[1]: netctl@br0.service: Job netctl@br0.service/start failed with result 'dependency'.
Apr 21 22:57:26 xeus2k1 systemd[1]: sys-subsystem-net-devices-enp113s0.device: Job sys-subsystem-net-devices-enp113s0.device/start failed with result 'timeout'.
Apr 21 22:57:26 xeus2k1 systemd[1]: systemd-sysctl.service: start operation timed out. Terminating.
Apr 21 22:57:26 xeus2k1 systemd[1]: systemd-sysctl.service: Main process exited, code=killed, status=15/TERM
Apr 21 22:57:26 xeus2k1 systemd[1]: systemd-sysctl.service: Failed with result 'timeout'.
Apr 21 22:57:26 xeus2k1 systemd[1]: Failed to start Apply Kernel Variables.
Apr 21 22:50:45 xeus2k1 systemd-modules-load[415]: Failed to find module 'tuxedo_io'
Apr 21 22:50:45 xeus2k1 systemd-modules-load[415]: Failed to find module 'tuxedo_keyboard'
Apr 21 22:57:26 xeus2k1 systemd[1]: Shutting down.

P.S.: This is the logs from with fstabs:

[manjaro-cinnamon /]# journalctl -S "2024-04-21 19:36:16" -U "Sun 2024-04-21 20:42:26" --priority=5
Apr 21 19:36:16 archlinux kernel: Linux version 6.8.5-1-MANJARO (builduser@fv-az846-576) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.42.0) #1 SMP PREEMPT_DYNAMIC Wed Apr 10 20:15:45 UTC 2024
Apr 21 19:36:16 archlinux kernel: random: crng init done
Apr 21 19:36:16 archlinux kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-6.8-x86_64 root=/dev/mapper/root rw apparmor=1 security=apparmor systemd.log_level=debug
Apr 21 19:36:16 archlinux kernel: Unknown kernel command line parameters "BOOT_IMAGE=/vmlinuz-6.8-x86_64", will be passed to user space.
Apr 21 19:36:16 archlinux kernel: x86/cpu: SGX disabled by BIOS.
Apr 21 19:36:16 archlinux kernel: MMIO Stale Data CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/processor_mmio_stale_data.html for more details.
Apr 21 19:36:16 archlinux kernel: audit: type=2000 audit(1713720976.066:1): state=initialized audit_enabled=0 res=1
Apr 21 19:36:16 archlinux kernel: SCSI subsystem initialized
Apr 21 19:36:16 archlinux kernel: VFS: Disk quotas dquot_6.6.0
Apr 21 19:36:16 archlinux kernel: Initialise system trusted keyrings
Apr 21 19:36:16 archlinux kernel: Key type blacklist registered
Apr 21 19:36:16 archlinux kernel: integrity: Platform Keyring initialized
Apr 21 19:36:16 archlinux kernel: integrity: Machine keyring initialized
Apr 21 19:36:16 archlinux kernel: Key type asymmetric registered
Apr 21 19:36:16 archlinux kernel: Asymmetric key parser 'x509' registered
Apr 21 19:36:16 archlinux kernel: Loading compiled-in X.509 certificates
Apr 21 19:36:16 archlinux kernel: Loaded X.509 cert 'Build time autogenerated kernel key: 954f3e5c74778db1297a5d36ed89ee57fcb3f17f'
Apr 21 19:36:16 archlinux kernel: Key type .fscrypt registered
Apr 21 19:36:16 archlinux kernel: Key type fscrypt-provisioning registered
Apr 21 19:36:17 archlinux kernel: usb: port power management may be unreliable
Apr 21 19:36:20 archlinux kernel: scsi 4:0:0:0: Direct-Access     JetFlash Transcend 128GB  1100 PQ: 0 ANSI: 6
Apr 21 19:36:20 archlinux kernel: sd 4:0:0:0: [sda] 241637376 512-byte logical blocks: (124 GB/115 GiB)
Apr 21 19:36:20 archlinux kernel: sd 4:0:0:0: [sda] Write Protect is off
Apr 21 19:36:20 archlinux kernel: sd 4:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr 21 19:36:20 archlinux kernel: sd 4:0:0:0: [sda] Attached SCSI disk
Apr 21 19:36:23 archlinux kernel: Key type trusted registered
Apr 21 19:36:24 archlinux kernel: Key type encrypted registered
Apr 21 19:36:24 archlinux systemd[1]: Switching root.
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.200:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="dovecot-anvil" pid=449 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.200:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="klogd" pid=443 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.217:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=436 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.217:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=436 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.220:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="syslogd" pid=445 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.223:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=435 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.223:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="torbrowser_tor" pid=447 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.223:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="dovecot-config" pid=451 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.223:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="samba-rpcd-classic" pid=441 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 kernel: audit: type=1400 audit(1713720985.223:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="dovecot-deliver" pid=452 comm="apparmor_parser"
Apr 21 19:50:58 xeus2k1 systemd[1]: dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device: Job dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device/start timed out.
Apr 21 19:50:58 xeus2k1 systemd[1]: Timed out waiting for device /dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a.
Apr 21 19:50:58 xeus2k1 systemd[1]: Dependency failed for File System Check on /dev/disk/by-partuuid/71f8f961-d84b-48b4-9f03-ed2fef926e5a.
Apr 21 19:50:58 xeus2k1 systemd[1]: Dependency failed for /boot.
Apr 21 19:50:58 xeus2k1 systemd[1]: Dependency failed for Local File Systems.
Apr 21 19:50:58 xeus2k1 systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
Apr 21 19:50:58 xeus2k1 systemd[1]: Dependency failed for /boot/efi.
Apr 21 19:50:58 xeus2k1 systemd[1]: boot-efi.mount: Job boot-efi.mount/start failed with result 'dependency'.
Apr 21 19:50:58 xeus2k1 systemd[1]: boot.mount: Job boot.mount/start failed with result 'dependency'.
Apr 21 19:50:58 xeus2k1 systemd[1]: systemd-fsck@dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.service: Job systemd-fsck@dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.service/start failed with result 'dependency'.
Apr 21 19:50:58 xeus2k1 systemd[1]: dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device: Job dev-disk-by\x2dpartuuid-71f8f961\x2dd84b\x2d48b4\x2d9f03\x2ded2fef926e5a.device/start failed with result 'timeout'.
Apr 21 19:50:58 xeus2k1 systemd[1]: dev-disk-by\x2dpartuuid-a875fad4\x2dea8b\x2d4777\x2daf84\x2d2824aea21320.device: Job dev-disk-by\x2dpartuuid-a875fad4\x2dea8b\x2d4777\x2daf84\x2d2824aea21320.device/start timed out.
Apr 21 19:50:58 xeus2k1 systemd[1]: Timed out waiting for device /dev/disk/by-partuuid/a875fad4-ea8b-4777-af84-2824aea21320.
Apr 21 19:50:58 xeus2k1 systemd[1]: Dependency failed for File System Check on /dev/disk/by-partuuid/a875fad4-ea8b-4777-af84-2824aea21320.
Apr 21 19:50:58 xeus2k1 systemd[1]: systemd-fsck@dev-disk-by\x2dpartuuid-a875fad4\x2dea8b\x2d4777\x2daf84\x2d2824aea21320.service: Job systemd-fsck@dev-disk-by\x2dpartuuid-a875fad4\x2dea8b\x2d4777\x2daf84\x2d2824aea21320.service/start failed with result 'dependency'.
Apr 21 19:50:58 xeus2k1 systemd[1]: dev-disk-by\x2dpartuuid-a875fad4\x2dea8b\x2d4777\x2daf84\x2d2824aea21320.device: Job dev-disk-by\x2dpartuuid-a875fad4\x2dea8b\x2d4777\x2daf84\x2d2824aea21320.device/start failed with result 'timeout'.
Apr 21 19:36:25 xeus2k1 systemd-modules-load[418]: Failed to find module 'tuxedo_io'
Apr 21 19:36:25 xeus2k1 systemd-modules-load[418]: Failed to find module 'tuxedo_keyboard'
Apr 21 19:50:58 xeus2k1 systemd[1]: Shutting down.