Can't load kernel after Timeshift restore / broken installation

I have restored a Timeshift image, and now get this error message right after boot:

sbin/init/: error while loading libzstd.so.1

Then get a message about “Kernel Panic mode”.

I have booted from a live USB (manjaro) and restorig another Timeshift image, including the Timeshift image backup I had made JUST BEFORE the problematic restore. The same error remains every time.

If I run manjaro-chroot -a, as mentionned in other topics to fix boot issues, I get:

grub-probe: error: cannot find a GRUB drive for /dev/sdc1

I believe this error is FINE since this corresponds to the bootable USB I am using to run manjaro-chroot/debug.

It also shows:

Mounting (Manjaro Linux) [/dev/sda2]
--> mount: [/mnt]
--> mount: [/mnt/boot/efi]
--> mount: [/mnt/home]

Then following this wiki, I cannot seem to even run grub-install:

[manjaro /]# grub-install
grub-install: error while loading shared libraries: lipthread.so.0: cannot open shared object: no such file or directory

Similarly, if I try to run:


[manjaro /]# pacman -Syu grub
pacman: error while loading shared libraries: libzstd.so./: no such file or directory

Any ideas?

System info:

Please note the system info was obtained not from the broken system, but from the bootable usb. Running the command with manjaro-chroot -a on the borken system yields an error (missing libs).

[manjaro@manjaro libs_desktop]$ inxi --admin --verbosity=7 --filter --no-host --width
System:
  Kernel: 5.15.32-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 11.2.0
    parameters: BOOT_IMAGE=/boot/vmlinuz-x86_64 lang=en_US keytable=us tz=UTC
    misobasedir=manjaro misolabel=MANJARO_XFCEM_2126 quiet
    systemd.show_status=1 apparmor=1 security=apparmor driver=free
    nouveau.modeset=1 i915.modeset=1 radeon.modeset=1
  Desktop: Xfce v: 4.16.0 tk: Gtk v: 3.24.29 info: xfce4-panel wm: xfwm
    v: 4.16.1 vt: 7 dm: LightDM v: 1.30.0 Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Desktop System: Hewlett-Packard product: HP Z420 Workstation v: N/A
    serial: <superuser required> Chassis: type: 6 serial: <superuser required>
  Mobo: Hewlett-Packard model: 1589 v: 0.00 serial: <superuser required>
    UEFI: Hewlett-Packard v: J61 v03.92 date: 02/09/2018
Battery:
  Device-1: hidpp_battery_0 model: Logitech Wireless Keyboard ERGO K860
    serial: <filter> charge: 100% (should be ignored) rechargeable: yes
    status: discharging
  Device-2: hidpp_battery_1
    model: Logitech M720 Triathlon Multi-Device Mouse serial: <filter>
    charge: 100% (should be ignored) rechargeable: yes status: discharging
Memory:
  RAM: total: 31.28 GiB used: 3.4 GiB (10.9%)
  RAM Report:
    permissions: Unable to run dmidecode. Root privileges required.
CPU:
  Info: model: Intel Xeon E5-1650 0 bits: 64 type: MT MCP arch: Sandy Bridge
    family: 6 model-id: 0x2D (45) stepping: 7 microcode: 0x71A
  Topology: cpus: 1x cores: 6 tpc: 2 threads: 12 smt: enabled cache:
    L1: 384 KiB desc: d-6x32 KiB; i-6x32 KiB L2: 1.5 MiB desc: 6x256 KiB
    L3: 12 MiB desc: 1x12 MiB
  Speed (MHz): avg: 3119 high: 3197 min/max: 1200/3800 scaling:
    driver: intel_cpufreq governor: schedutil cores: 1: 3192 2: 3192 3: 3193
    4: 3192 5: 3197 6: 3192 7: 2571 8: 2936 9: 3192 10: 3192 11: 3192 12: 3192
    bogomips: 76653
  Flags: acpi aes aperfmperf apic arat arch_perfmon avx bts clflush cmov
    constant_tsc cpuid cx16 cx8 dca de ds_cpl dtes64 dtherm dts epb ept est
    flexpriority flush_l1d fpu fxsr ht ibpb ibrs ida lahf_lm lm mca mce
    md_clear mmx monitor msr mtrr nonstop_tsc nopl nx pae pat pbe pcid
    pclmulqdq pdcm pdpe1gb pebs pge pln pni popcnt pse pse36 pti pts rdtscp
    rep_good sep smx ss ssbd sse sse2 sse4_1 sse4_2 ssse3 stibp syscall tm tm2
    tpr_shadow tsc tsc_deadline_timer vme vmx vnmi vpid x2apic xsave xsaveopt
    xtopology xtpr
  Vulnerabilities:
  Type: itlb_multihit status: KVM: VMX disabled
  Type: l1tf
    mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable
  Type: meltdown mitigation: PTI
  Type: spec_store_bypass
    mitigation: Speculative Store Bypass disabled via prctl and seccomp
  Type: spectre_v1
    mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2 mitigation: Retpolines, IBPB: conditional, IBRS_FW,
    STIBP: conditional, RSB filling
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: NVIDIA GK208B [GeForce GT 710] vendor: Micro-Star MSI
    driver: nouveau v: kernel pcie: gen: 2 speed: 5 GT/s lanes: 8 ports:
    active: DVI-D-1,VGA-1 empty: HDMI-A-1 bus-ID: 05:00.0 chip-ID: 10de:128b
    class-ID: 0300
  Device-2: Logitech C505e HD Webcam type: USB
    driver: snd-usb-audio,uvcvideo bus-ID: 3-2:2 chip-ID: 046d:08e4
    class-ID: 0102 serial: <filter>
  Display: x11 server: X.Org v: 1.21.1.3 compositor: xfwm v: 4.16.1 driver:
    X: loaded: modesetting alternate: fbdev,vesa gpu: nouveau display-ID: :0.0
    screens: 1
  Screen-1: 0 s-res: 4480x1440 s-dpi: 96 s-size: 1185x381mm (46.65x15.00")
    s-diag: 1245mm (49.01")
  Monitor-1: DVI-D-1 pos: primary,left model: LG (GoldStar) QHD
    serial: <filter> built: 2020 res: 2560x1440 hz: 60 dpi: 93 gamma: 1.2
    size: 698x392mm (27.48x15.43") diag: 801mm (31.5") ratio: 16:9 modes:
    max: 2560x1440 min: 640x480
  Monitor-2: VGA-1 pos: right model: BenQ GL2780 serial: <filter>
    built: 2020 res: 1920x1080 hz: 60 dpi: 82 gamma: 1.2
    size: 598x336mm (23.54x13.23") diag: 686mm (27") ratio: 16:9 modes:
    max: 1920x1080 min: 720x400
  Message: Unable to show GL data. Required tool glxinfo missing.
Audio:
  Device-1: Intel C600/X79 series High Definition Audio
    vendor: Hewlett-Packard driver: snd_hda_intel v: kernel bus-ID: 00:1b.0
    chip-ID: 8086:1d20 class-ID: 0403
  Device-2: NVIDIA GK208 HDMI/DP Audio vendor: Micro-Star MSI
    driver: snd_hda_intel v: kernel pcie: gen: 2 speed: 5 GT/s lanes: 8
    bus-ID: 05:00.1 chip-ID: 10de:0e0f class-ID: 0403
  Device-3: Logitech Headset H390 type: USB
    driver: hid-generic,snd-usb-audio,usbhid bus-ID: 2-1.1:3 chip-ID: 046d:0a44
    class-ID: 0300
  Device-4: Logitech C505e HD Webcam type: USB
    driver: snd-usb-audio,uvcvideo bus-ID: 3-2:2 chip-ID: 046d:08e4
    class-ID: 0102 serial: <filter>
  Sound Server-1: ALSA v: k5.15.32-1-MANJARO running: yes
  Sound Server-2: JACK v: 1.9.20 running: no
  Sound Server-3: PulseAudio v: 15.0 running: yes
Network:
  Device-1: Intel 82579LM Gigabit Network vendor: Hewlett-Packard
    driver: e1000e v: kernel port: e040 bus-ID: 00:19.0 chip-ID: 8086:1502
    class-ID: 0200
  IF: eno1 state: up speed: 1000 Mbps duplex: full mac: <filter>
  IP v4: <filter> type: dynamic noprefixroute scope: global
    broadcast: <filter>
  IP v6: <filter> type: noprefixroute scope: link
  WAN IP: <filter>
Bluetooth:
  Message: No bluetooth data found.
Logical:
  Message: No logical block device data found.
RAID:
  Message: No RAID data found.
Drives:
  Local Storage: total: 1.03 TiB used: 108.96 GiB (10.3%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/sda maj-min: 8:0 vendor: Kingston model: SA400S37120G
    size: 111.79 GiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s
    type: SSD serial: <filter> rev: 61K1 scheme: GPT
  ID-2: /dev/sdb maj-min: 8:16 vendor: Western Digital
    model: WD1002FAEX-00Y9A0 size: 931.51 GiB block-size: physical: 512 B
    logical: 512 B type: N/A serial: <filter> rev: 1D05 scheme: GPT
  ID-3: /dev/sdc maj-min: 8:32 type: USB vendor: Lexar
    model: USB Flash Drive size: 14.92 GiB block-size: physical: 512 B
    logical: 512 B speed: <unknown> type: SSD serial: <filter> rev: 1100
    scheme: MBR
  SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure?
  Optical-1: /dev/sr0 vendor: hp model: DVDROM DH40N rev: DD01
    dev-links: cdrom
  Features: speed: 40 multisession: yes audio: yes dvd: yes rw: none
    state: running
  Optical-2: /dev/sr1 vendor: hp model: DVD-ROM SH-116AB rev: HF20
    dev-links: N/A
  Features: speed: 16 multisession: yes audio: yes dvd: yes rw: none
    state: running
Partition:
  ID-1: /mnt raw-size: 50.28 GiB size: 49.24 GiB (97.93%)
    used: 49.18 GiB (99.9%) fs: ext4 dev: /dev/sda2 maj-min: 8:2 label: N/A
    uuid: 014bafdf-6822-4da9-bae5-091b96a3cc45
  ID-2: /mnt/boot/efi raw-size: 513 MiB size: 512 MiB (99.80%)
    used: 312 KiB (0.1%) fs: vfat dev: /dev/sda1 maj-min: 8:1 label: N/A
    uuid: A062-E7EC
  ID-3: /mnt/home raw-size: 61.01 GiB size: 59.8 GiB (98.02%)
    used: 59.77 GiB (100.0%) fs: ext4 dev: /dev/sda3 maj-min: 8:3 label: N/A
    uuid: 5e28f708-2b12-42d5-b3d2-12d764978a13
Swap:
  Alert: No swap data was found.
Unmounted:
  ID-1: /dev/sdb1 maj-min: 8:17 size: 491.84 GiB fs: ext4 label: hdd-1
    uuid: f47c183c-86db-427c-8f49-1df774a2a80e
  ID-2: /dev/sdb2 maj-min: 8:18 size: 439.67 GiB fs: ext4 label: hdd-2
    uuid: 69c6b64c-42db-4688-a428-60407b12da96
  ID-3: /dev/sdc1 maj-min: 8:33 size: 2.54 GiB fs: iso9660
  ID-4: /dev/sdc2 maj-min: 8:34 size: 4 MiB fs: vfat label: MISO_EFI
    uuid: 2086-4E82
USB:
  Hub-1: 1-0:1 info: Full speed or root hub ports: 3 rev: 2.0 speed: 480 Mb/s
    chip-ID: 1d6b:0002 class-ID: 0900
  Hub-2: 1-1:2 info: Intel Integrated Rate Matching Hub ports: 6 rev: 2.0
    speed: 480 Mb/s chip-ID: 8087:0024 class-ID: 0900
  Device-1: 1-1.3:3 info: Lexar Media LJDTT16G [JumpDrive 16GB]
    type: Mass Storage driver: usb-storage interfaces: 1 rev: 2.0
    speed: 480 Mb/s power: 200mA chip-ID: 05dc:a81d class-ID: 0806
    serial: <filter>
  Hub-3: 2-0:1 info: Full speed or root hub ports: 3 rev: 2.0
    speed: 480 Mb/s chip-ID: 1d6b:0002 class-ID: 0900
  Hub-4: 2-1:2 info: Intel Integrated Rate Matching Hub ports: 8 rev: 2.0
    speed: 480 Mb/s chip-ID: 8087:0024 class-ID: 0900
  Device-1: 2-1.1:3 info: Logitech Headset H390 type: Audio,HID
    driver: hid-generic,snd-usb-audio,usbhid interfaces: 4 rev: 1.1
    speed: 12 Mb/s power: 100mA chip-ID: 046d:0a44 class-ID: 0300
  Device-2: 2-1.3:4 info: Logitech Unifying Receiver
    type: Keyboard,Mouse,HID driver: logitech-djreceiver,usbhid interfaces: 3
    rev: 2.0 speed: 12 Mb/s power: 98mA chip-ID: 046d:c52b class-ID: 0300
  Hub-5: 3-0:1 info: Hi-speed hub with single TT ports: 4 rev: 2.0
    speed: 480 Mb/s chip-ID: 1d6b:0002 class-ID: 0900
  Device-1: 3-2:2 info: Logitech C505e HD Webcam type: Video,Audio
    driver: snd-usb-audio,uvcvideo interfaces: 4 rev: 2.0 speed: 480 Mb/s
    power: 500mA chip-ID: 046d:08e4 class-ID: 0102 serial: <filter>
  Device-2: 3-4:3 info: Logitech Unifying Receiver type: Keyboard,Mouse,HID
    driver: logitech-djreceiver,usbhid interfaces: 3 rev: 2.0 speed: 12 Mb/s
    power: 98mA chip-ID: 046d:c52b class-ID: 0300
  Hub-6: 4-0:1 info: Super-speed hub ports: 4 rev: 3.0 speed: 5 Gb/s
    chip-ID: 1d6b:0003 class-ID: 0900
Sensors:
  System Temperatures: cpu: 38.0 C mobo: N/A gpu: nouveau temp: 48.0 C
  Fan Speeds (RPM): N/A
Info:
  Processes: 292 Uptime: 19h 20m wakeups: 27 Init: systemd v: 250
  tool: systemctl Compilers: gcc: 11.2.0 clang: 13.0.1 Packages: pacman: 983
  lib: 284 Shell: Bash v: 5.1.16 running-in: xfce4-terminal inxi: 3.3.15

EDIT:

Seems I have issues with the manjaro install, because still with manjaro-chroot:

[manjaro /]# pacman --help:
pacman: error while loading shared libraries: libzstd.so./: no such file or directory

Say if I try python:

[manjaro /]# python
pacman: error while loading shared libraries: libpython3.9.so.1.0.so./: no such file or directory

EDIT #2:

Seems that for whatever reasons, the symlinks for a bunch of libs are wrong under /usr/lib/, which leads to them not being found by system binaries (pacman, grep, etc.) that needs them. For instances:

ls -la /usr/lib | grep libz.so
libz.so -> libz.so.1.2.11
libz.so.1 -> libz.so.1.2.11
libz.so.1.2.12

E.g. symlinks points to 1.2.11, but 1.2.12 is actually available. If I fix that symlink, the errors changes (to another symlink that isn’t up to date). So it appears I could recover pacman (and potentially whatever missing link prevents me from booting) by fixing the symlinks. Will update here.

try running from chroot:
pacman -Syu

1 Like

See edit. pacman complains that libzstd.so doesn’t exit. Any pacman, or system command it seems, doesn’t work with a similar error.

try to install them with:
pacman -S lib32-zstd
pacman -S zstd

well I can’t - because

pacman <anything> complains that libzstd.so.1 isn’t here…

and can you install kernel for example:
sudo mhwd-kernel -i linux510

Well, then I get others libs (libx86emu.so.3) complains. See edit#2, seems I need to fix the missing/wrong symlink to fix my distrubution.

and if you try overwrite:
sudo pacman -Syyu --overwrite "*"
and post the error (if any) output here

Same error message. Doesn’t matter what arguments are passed to pacman; for whatever reasons the symlink to libzstd.so.1 is wrong. Thus it never gets to try to do anything; the library isn’t available.

Welcome to Manjaro! :smiling_face_with_three_hearts:

  1. Please read the information behind this link. It will help you to post necessary information. [HowTo] Provide System Information
  2. Please press the three dots below your post and then press the :pencil2:
  • If you give us information about your system, we can see what we’re talking about and make better suggestions.
  • You can do this by using inxi in a terminal or in console.
inxi --admin --verbosity=7 --filter --no-host --width
  • Personally identifiable information such as serial numbers and MAC addresses are filtered out by this command
  • Presenting the information in this way allows everyone to be familiar with the format and quickly find the items they need without missing anything.
  1. Copy the output from inxi (including the command) and paste it into your post.
  • To make it more readable, add 3 backticks ``` on an extra line before and after the pasted text.

You can also use Maxi

to collect information about the boot process. This works,

  • when you are in your running system (with CTRL+ALT+F2).
  • And also in a live environment

P.S. If i do know right, there exists a statically linked version of pamac or pacman.

1 Like

Thanks, added the info.

You are right about pacman-static; I’ve found the repo on AUR. I doesn’t seem obvious to me how to run it. I mean I can boot from the usb and and run it from there/manjaro-chroot I suppose; however it’s not clear to me yet if I’m suppose to do that with something like:

sudo pacman-static ....

Or what. I might very well look into that after. For now, I have a laptop with manjaro install as well, so I will first try to manually repair the broken symlink/lib files with those? ACtually, thinking of it, I am going to ask a question about how to use pacman static to ‘rescue’ my broken install. I’m going to search a bit more for some sort of wiki/post, and if I don’t find it then I guess we’ll create it :wink:

you can try this command:
sudo pacman-static -S pacman --overwrite "*"
hopefully it will work

1 Like

Okay - but then my question is, I have to git clone/mkgpg from the bootable install, not my broken install. So… when I run that command, will it overwrite:

  • the bootable pacman stuff (which won’t help me recover my broken system)?
  • or the broken system one?

What would you do in order to have this command “act” on the broken system?

good question… and i really dont know… you could install it from official repositories in the live usb manjaro, then chroot into the broken install and run the command?

1 Like

I did never use this, but i would:

  • install pacman-static in live manjaro (no git necessary - this is in core) with pacman :wink:
  • copy the binary to the installed manjaro (into /usr/local/bin)
  • chroot
  • start pacman-static from where it is (/usr/local/bin/pacman-static …)
    :footprints:
1 Like

ah… then potentially I would be able to run pacman-static from manjaro-chroot. It would presumbly allow me to update (and/or force update… I know I know don’t use the --force, but I’m already so deeply broken who cares), and it would update the (broken) /usr/lib stuff.

This somewhat along the lines of what I’m doing manually, if it works - e.g. I’m currently copying files from an healthy manjaro install I have on another machine onto this one. Obviously pretty tedious. I’ll try that I update.

Good thing I already have pacman-static installed in the bootable, so shouldn’t be too long tot est that.

I wish you good luck

For the future i do suggest to use what i learned to love:

  • btrfs
  • snapper
  • flat layout
  • manual rollback
This is more work, but your data will be save

You find good Information about Btrfs in the wiki

and in

1 Like

You should change libz.so1.2.12 back to libz.so1.2.11

mv /usr/lib/libz.so.1.2.12 /usr/lib/libz.so.1.2.11

Reasonnably sure the approach would actually be to update the symlink here, because the file you propose to rename is the actually runnable. Changing its name doesn’t change the underlying/code version, so this seems likely to create more problems than it solves…

All right, thanks everyone for the help. I actually couldn’t pull it off - even with manjaro-chroot and fixing the broken symlink, I ended up going into the glibc rabbit hole (apparently it was affected as well). That proved one step too far, and I couldn’t fix pacman, even using pacman-static. So I ended up swallowing the pile & re-installing manjaro. most of my stuff is backed up on some cloud thingy, just lost a few configurations.

I’m still not sure why the timeshift restore failed, and why it ended up being so complicated. Well, thanks for the support, I still learned a lot of about how an OS works, so there’s that.

Will look into the btrfs link you provided, thank you. CLosing this topic!

1 Like