Totally broken system after December 2025 Stable Update

Support System Updates
1

Hello guys,

I messed up my system with the last update :tada:

What happened?
I did a normal pamac update, entered my password, and confirmed the install.
Then I went to do some other work. Coming back to my machine I found a black screen. By opening and closing the lid the system was going to standby normally and waking again (visible through the notebook leds), but always just to a black screen.
I couldn’t enter any tty, exept tty7, which was showing that the session was locked, and that I would be redirected to a login screen, only that I wasn’t… So after letting it rest for some time I figured, nothing is happening and a forced a reset.

The system wouldn’t boot (kernel error), so I tried to chroot into the install. First I failed, because the drive is encrypted, but with some searching I managed to get in.

Now I wanted to do a full system update again, but pacman threw an error. I managed to remove and reinstall the manjaro and arch keyrings, just to find pacman giving me ssl cert errors now (for example error: failed retrieving file ‘core.db’ from manjarolinux-mirror.cloud.mu : error adding trust anchors from file: /etc/ssl/certs/ca-certificates.crt). That folder is basically empty, no certs (or symliks to certs), only ca-bundeles.crt and ca-certificates.crt and a java folder.
pamac doesn’t work either (error while loading shared libraries: /usr/lib/libgmodule-2.0.so.0: file too short), so I failed to reinstall the ssl certs with pamac reinstall ca-certificates. And pacman-mirrors is doing nothing?!

Here I am stuck, and I am not sure in which direction to proceed. Propably I am missing something obvious, but I am just a regular user, so I am not to sure what I am doing here anyway :sweat_smile:

I have a backup of all my files, and I can still chroot into the system, so I am not to worried. Still I would like to make the system work again (if possible without reinstalling), and would also take any advice, what I could have done before the reset to avoid the trouble.

Any help/hints/insight appreciated.

Tim

2

The first will be to try to fix the interrupted update. And after that - back to the initial problem which is still a mystery (or maybe it was only an interrupted update).
Boot from usb. Check the disk. Chroot and follow the guide for interrupted update. You may need to download pacman-static. Depending of if errors occur, you may need other tweaks, like --overwrite flag. As a good measure also check for missing files with pacman -Qk and reinstall with overwrite if needed.

3
2 Likes
5

Thank you two for your quick reply. I am not sure, that I was clear enough before…
Like I wrote, I was basically proceeding like the HowTo suggests, but at the point of a fresh update with pacman -Syyu the problems started (see initial post). To me it seems the pacman-mirrors command doesn’t do anything…
So my question was not: what to do generally, if you get an update interrupted, but specifically, what to do in my situation, if chrooting works, but neither pacman nor pamac seem to work. Maybe that is more clear?

Or did I miss something?

pacman -Qk runs normally, and doesn’t show any missing files, if I didn’t overlook something. What did you mean with the –overwrite flags and pacman-static?

6

pacman-mirrors on itself does nothing but display your current branch and status of the mirrors in the mirror list.

Use man pacman-mirrors or pamanc-mirrors -h to learn how it can rewrite your mirror list.

As for black screen - it could point to known issues with Nvidia GPU’s.

See the latest stable Announcements > Stable Updates or even the Announcements > Testing Updates to learn of potential issues.

Especially pay attention to the second comment - which holds information about known issues and solutions - including an update preppers guide.

7

well, I ment when I said: it doesn’t do anything. Neither does it give the mirror list, nor does -h option show the help or any other option do anything. Really just nothing happens :grinning_face:

And I don’t have a nvidia gpu, its just some intel onboard graphic…

And these days I read the announcements before the update (learnt from past mistakes), but of course reading and understanding everything are two different things as well :face_with_peeking_eye:

8

Now that is strange…

That is even more strange - AMD and Intel iGPU very reliable…

I am in the dark on this one.

Could indicate a file system error…

If you can chroot into the system - I suggest you run a file system check on your root - how you do will depend on the filesystem.

9

its LUKS encrypted ext4
so I do cryptsetup and manual mount for chrooting

So to check the possibly broken filesystem I would do

sudo fsck /dev/sda2

on the live system terminal (not chrooted)?

lsblk –fs gives this output:

sda
├─sda1
│ vfat FAT32 NO_LABEL 60B5-8A6A
├─sda2
│ crypto 1 ffc0a8e3-431f-4e00-a614-fb14f0ba1b0f
│ └─mydrive
│ ext4 1.0 36fb0418-e109-41e5-b442-74fba7f9f99d 53.4G 43% /mnt
├─sda3
│ crypto 1 3646510a-e376-4eb9-97a5-feec2aec0623
└─sda4
ext4 1.0 unencryptediso e82df732-e38d-4a3a-bfb8-c78d2d2dea2f

while mydrive is what I mounted under /mnt and what I was chrooting in

10

No - not when it is a luks container.

Then you do it on the open container /dev/mapper/<container> and not while mounted

umount /mnt -R

Then run fsck on the container

fsck /dev/mapper/mydrive
3 Likes
11

ok, fsck ran, found and fixed some errors.

After that I mounted and chrooted again, but the errors are the same (no reaction from pacman-mirrors and pacman -Syyu gives ssl errors so no download)

12

I expected that much :slight_smile:

First exit the chroot but keep the mount

Also mount your efi partition - I see there is one

mount /dev/sda1 /mnt/boot/efi

I would check if the package lists in the /mnt is intact - then sanitize them and use them to a complete rebuild of the filesystem.

Using these as command templates

If the original files are intact (/mnt/rootfs-pkgs.txt & /mnt/desktopfs-pkgs.txt)

pacman -Syy --needed --root /mnt $(cat /mnt/rootfs-pkgs.txt | awk '{print $1;}')

pacman -Syy --needed --root /mnt $(cat /mnt/desktopfs-pkgs.txt | awk '{print $1;}')
1 Like
13

I think there is maybe a flag missing to force update?

The commands complete fine, but after every package it prints
is up to date – skipping

so nothing actually happens.

edit: after scrolling through the terminal output there are some lines that go like

error: target not found: qpdfview

Does that mean, the lists are corrupt too, or are this just old packages, that don’t matter anymore?

edit again: sorry, I just realized I did this on the wrong terminal, not chrooted :exploding_head:
will try again and report…

14

This list is likely valid - but qpdfview is no longer in the repo.

open the file with an editor and remove the line with qpdfview

15

sorry, I am confused now
I don’t know if you saw my last edit so I will ask directly:
after creating the efi mount, should I chroot again to run these commands or not?

Because when I answered first, I did not chroot but then realized that it means, pacman is working on the live system, not the one I want to repair.

AND it is not just the qpdfview package. There are more, but I don’t know how to filter them out of this long list…

16

That’s good - but no snapshot?

17

No, I don’t have a snapshot. Never managed to configure it properly with my setup…

Any ideas what I can try to proceed? Can’t really believe, that the system is unrepairable. At least I never reached that point on any distribution I have ever used (so far :smiley: )

18

You have repaired the file system inside the container (no chroot required)

If the ISO you use to rescue the system is the latest Manjaro ISO matching the system you want to repair then you can use the file list from the iso

  • Open the luks container
  • Mount the mapped container on /mnt
  • mount the efi partition on /mnt/boot/efi

Do not enter a chroot shell - stay with the live system.

Then you can do the same except the path to the package lists will be the ISO

pacman -Syy --needed --root /mnt $(cat /rootfs-pkgs.txt | awk '{print $1;}')
pacman -Syy --needed --root /mnt $(cat /desktopfs-pkgs.txt | awk '{print $1;}')
19

@tial, here are my 2 cents: In your case, I wouldn’t waste any more time beating a dead horse. Since you’ve backed up your most important files, wipe the disk clean, reformat it, create the necessary partitions and reinstall everything. :innocent:

You could install timeshift and create system snapshots then next time you can use this to restore an older system stage (saved my ■■■ two or three times).

1 Like
20

Most welcome, though all I did was to enhance the reply from @Teo with a link. However, let’s back-track a little:

No guarantees, but perhaps this will help clear this up; at least, from the mirrors perspective. Try with the following pamac command:

pamac upgrade --no-aur --force-refresh
23

@linux-aarhus
The ISO is not the latest, it is propably the one I used to install the system some time ago. So I guess I better use the list from the broken system?!

Do not enter a chroot shell - stay with the live system.

Ok. Thats what I did in the first place.
Like I wrote then, I get many messeages in the form of packageXY is up to date – skipping and some messages like error: target not found: packageXY. So nothing happens at all.

You suggested to remove the lines, that cause errors (like qpdfview), but I don’t know how to filter the warning/error output of pacman (grep doesn’t seem to work?!)

@Wollie

thanks for your input!
Of course that option does exist, but like written before, I don’t really believe yet, that the system is irreparable after a failed update. That has never happened to me in many years on many different distributions. Even without special knowledge (I am not in IT or something), I was always able to repair a broken linux system with some searching on the internet…
Even if it is time consuming, I kind of don’t want to give up out of curiosity :smiley:
So wiping and reinstalling would be my last resort.

@soundofthunder

thanks again, for another idea, but I can’t try that. Like written in the first post, pamac is not working (pamac: error while loading shared libraries: /usr/lib/libgmodule-2.0.so.0: file too short)
pacman does seem to work, but can’t make any online connection because of the cert errors. pacman-mirrors doesn’t do anything (no error either)