System Doesn't boot - My usual fixes don't work

[rob215x]

I’m working on a friend’s machine that’s running Manjaro. He hasn’t done any updates in months so I just did all of the updates. Everything looked like it updated successfully until it finally got to building Google Chrome and it just hung for 30 minutes, so I rebooted.

Upon reboot, there’s a bunch of “[FAILED] Failed to start …” lines and there are so many, the top few have already scrolled off the screen

1. I cannot CTRL + ALT + F2 or CTRL + ALT + F3

2. I CAN get to the GRUB menu, but I’m not sure what to do here. If I choose any of the Manjaro options, I just end up back at the “[FAILED] Failed to start…” screen.

3. I launched from my USB installer and did the following from [HowTo] Recovering from an interrupted update/upgrade

manjaro-chroot -a

then…

[ -f /var/lib/pacman/db.lck ] && rm -f /var/lib/pacman/db.lck
pacman-mirrors -f && pacman -Syyu
update-grub
exit

That all worked, except for “update-grub” which gave me “command not found”.

I rebooted and I’m right back to the list of “[FAILED] Failed to start…” messages. So now I’m stuck. I’ll take a photo of the screen in a minute and post it here.

Thanks!

[cscs]

@rob215x

Everything looked like it updated successfully until it finally got to building Google Chrome and it just hung for 30 minutes, so I rebooted.

chrome can take hours to build.

Also … it is usually suggested to separate repo and AUR updates.

I dont know what you actually did … but if you held the power button down then that could cause any number of issues depending on what exactly was going on at the time.

@rob215x

That all worked, except for “update-grub” which gave me “command not found”.

The equivalent is

sudo grub-mkconfig -o /boot/grub/grub.cfg

[rob215x]

@cscs thanks, I’ll try the USB stick again with your grub command.

Here’s a phone pic I just took of the screen when I try to boot normally…

UPDATE:
I tried the procedure with the USB installer and it didn’t help. I still get the same screen as pictured above.

What can I try next?

Is there anything I can do from the GRUB menu?

[Aragorn]

1. Please don’t gratuitously post any images. They take up space in the forum database, they cannot be copied from, and they cannot be indexed.

2. Your root filesystem is most likely hosed. Boot up from the USB stick, and without mounting your root filesystem, run a full fsck on it. See the man page for options.

[rob215x]

Thanks for your suggestion, I’ll try that next. As for the image, I’ve been on this forum for a while and I fully understand that images are worse than text you can actually index and copy/paste. However, I have never seen such a screen before and in this case, I thought a picture was worth 1000 words.

[rob215x]

Booted from USB again and…

[manjaro manjaro]# fsck /dev/sda2
fsck from util-linux 2.39.3
e2fsck 1.47.0 (5-Feb-2023)
/dev/sda2: clean, 555301/30507008 files, 86523432/122018696 blocks
[manjaro manjaro]#

[Aragorn]

Well, if your filesystem is clean, then it is the files themselves which are damaged. It will be virtually impossible to fix this without reinstalling, which is probably going to be the easiest and quickest solution.

Furthermore, given that this was an unmaintained and un-updated system, its owner will probably be better off running a point-release distribution. Manjaro is high-maintenance, and there’s no way around that.

[rob215x]

and I also did:

[manjaro manjaro]# fsck -f /dev/sda2
fsck from util-linux 2.39.3
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda2: 555311/30507008 files (3.7% non-contiguous), 86523432/122018696 blocks

So, it doesn’t seem like anything is wrong with the disk. But I also didn’t build this machine so I don’t know why Manjaro is on /dev/sda2. Apparently there is no /dev/sda1. Does that matter?

[Aragorn]

@rob215x

Apparently there is no /dev/sda1. Does that matter?

It may have been there in the past, or it could be an unformatted bios_grub partition in case the partition table is GPT but the machine boots in legacy BIOS mode. It either way doesn’t matter.

[rob215x]

@Aragorn

Furthermore, given that this was an unmaintained and un-updated system, its owner will probably be better off running a point-release distribution. Manjaro is high-maintenance, and there’s no way around that.

Yeah, I agree you’re probably right. I always like to learn from these experiences and post solutions that can help others later.

At least I can easily copy all of his personal files off and just do a fresh install.

Thanks

[scotty65]

@rob215x

Here’s a phone pic I just took of the screen when I try to boot normally…

That looks like the issue caused by the move to systemd v256:

[Stable Update] 2024-07-01 - Kernels, Systemd, KDE Gear, LibreOffice, Firefox, Wine, Qemu

* Support for cgroup v1 ('legacy' and 'hybrid' hierarchies) is now considered obsolete and systemd by default will refuse to boot under it. To forcibly reenable cgroup v1 support, SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1 must be set on kernel command line. The meson option 'default-hierarchy=' is also deprecated, i.e. only cgroup v2 ('unified' hierarchy) can be selected as build-time default.

The fix was here (but don’t type in all the X’s - they are just placeholders for whatever else was on that line in the /etc/grub/default file):

[Stable Update] 2024-07-01 - Kernels, Systemd, KDE Gear, LibreOffice, Firefox, Wine, Qemu

@scotty65 thank you for highlighting that, I was lost in search results:) After manjaro-chroot login to my installation, I’ve edited the /etc/default/grub file and edited the relevant line as below: GRUB_CMDLINE_LINUX_DEFAULT="XXXXXXXXXXX SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1 systemd.unified_cgroup_hierarchy=0" and then sudo grub-install && sudo update-grub and then rebooted my computer. Now I can go back to my Monday routine, peacefully:))

[beermad]

@Aragorn

Furthermore, given that this was an unmaintained and un-updated system, its owner will probably be better off running a point-release distribution. Manjaro is high-maintenance, and there’s no way around that.

And if the owner doesn’t want to continue doing updates regularly, perhaps Manjaro is the wrong distro for them?