Multiple computers have failed in my homelab last week.
- 1 laptop failed after docker install (fresh install mostly)
- 1 server failed while executing a job (no user interaction, so apparently the script is failing for
- Probably other computers are affected as well
I was able to find a workaround for the second computer. Very simple:
- do not reboot or update the machine
- fix the issue first by opening TTY (Ctrl+Alt+F2) ((my gui session was broken and failing to unlock))
sudo pacman -S lib32-glibc sudo pacman -S glibc
The first being a dependant of the second, I was lazy to find the correct Pacman command
Apparently this is a common issue with Arch linux this month. From my understanding the issue might come from a path change for lib32-glibc. Apparently this works because an hotfix has been released, but I assume rollback to a previous version would work too.
On my laptop I did try to update with pacman first then reboot, from my understanding libc is a sensitive package so the update went south and it has ended up in a kernel panic, then I booted on the iso of manjaro and mounted / “root” with chroot to perform the update, kernel panic was solved but system was stuck in emergency mode, I ran out of time due to work so I’ve taken the “ali killswitch way” by making a backup of /home/$USER and wiping the system.
So now my watch begins for finding potential issues on the rest of the lab