Totally broken system after December 2025 Stable Update

Have you been updating regularly? Monitoring the update announcement threads? I’m trying to get a feel for the general state of your system.

If possible, please also enter a chroot environment and generate system information with inxi:

Also, note the “Running inxi within a chroot environment” below.

System Information

While information from *-fetch type apps might be fine for someone wishing to buy your computer, for Support purposes it’s better to ask your system directly; :eyes:

Output of the inxi command (with appropriate parameters, and formatted according to forum guidelines) will generate information useful for those wishing to help:

Suggested inxi command (use either):

inxi -zv8 (short-form)
inxi --filter --verbosity=8 (long-form)
inxi man pages (manual)
Running `inxi` within a `chroot` environment
  • Add --color=0 to the long-form command, or…
  • Change the short-form command to inxi -zv8c0
Your privacy is respected

You can also pipe the inxi output to a text file, for example, like this:

inxi -zv8c0 > ~/my_inxi_output.txt

Updating mostly once a month. Monitoring the announcements as good as I can.
I am wondering if with the previous update I might have messed up some configuration when I merged the pacnew files?! But then again, why did everything run smoothly until yesterday, even the package download, and only during or after installing something went wrong?! I am very clueless…

output of inxi –filter –verbosity=8

https://notebin.de/?966786b4ceacb02d#6t79guicMH6mBbLyHyndewzkMaKKMReeNprEHDkFoV2c

System:
  Kernel: 6.6.10-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
    clocksource: tsc avail: acpi_pm parameters: BOOT_IMAGE=/boot/vmlinuz-x86_64
    lang=en_US keytable=de tz=Africa/Harare utc=yes misobasedir=manjaro
    misolabel=MANJARO_XFCE_2313 quiet systemd.show_status=1 splash apparmor=1
    security=apparmor driver=free nouveau.modeset=1 i915.modeset=1
    radeon.modeset=1
  Desktop: Xfce v: N/A tools: avail: light-locker dm: N/A Distro: Manjaro
    base: Arch Linux
Machine:
  Type: Laptop System: Acer product: TravelMate B116-M v: V1.23
    serial: <filter>
  Mobo: Acer model: BA10_BA v: V1.23 serial: <filter>
    part-nu: TravelMate B116-M_1043_1.23
    uuid: 5c5de08b-feea-9840-8b92-2c600cea9d33 Firmware: UEFI vendor: Insyde
    v: 1.23 date: 06/22/2016
Battery:
  ID-1: BAT1 charge: 41.1 Wh (100%) condition: 41.1/48.9 Wh (84%) volts: 16.82
    min: 15.2 model: LGC AC14B8K type: Li-ion serial: <filter> charging:
    status: full cycles: N/A
Memory:
  System RAM: total: 4 GiB available: 3.74 GiB used: 2.31 GiB (61.6%)
    igpu: 32 MiB
  Array-1: capacity: 8 GiB slots: 2 modules: 1 EC: None
    max-module-size: 4 GiB note: est.
  Device-1: ChannelA-DIMM0 type: DDR3 detail: synchronous size: 4 GiB
    speed: 1600 MT/s volts: min: 1.5 max: 1.5 width (bits): data: 8 total: 8
    manufacturer: Kingston part-no: ACR16D3LS1KNG/4G serial: <filter>
PCI Slots:
  Message: No PCI Slot data found.
CPU:
  Info: model: Intel Pentium N3700 socket: CHV bits: 64 type: MCP
    arch: Airmont level: v2 built: 2015-17 process: Intel 14nm family: 6
    model-id: 0x4C (76) stepping: 3 microcode: 0x35E
  Topology: cpus: 1x dies: 1 clusters: 2 cores: 4 smt: <unsupported> cache:
    L1: 224 KiB desc: d-4x24 KiB; i-4x32 KiB L2: 2 MiB desc: 2x1024 KiB
  Speed (MHz): avg: 480 min/max: 480/2400 base/boost: 1600/1660 scaling:
    driver: intel_cpufreq governor: schedutil volts: 4.0 V ext-clock: 83 MHz
    cores: 1: 480 2: 480 3: 480 4: 480 bogomips: 12805
  Flags: 3dnowprefetch acpi aes aperfmperf apic arat arch_perfmon bts
    clflush cmov constant_tsc cpuid cx16 cx8 de ds_cpl dtes64 dtherm dts epb
    ept erms est flexpriority fpu fxsr ht ida lahf_lm lm mca mce mmx monitor
    movbe msr mtrr nonstop_tsc nopl nx pae pat pbe pclmulqdq pdcm pebs pge
    pni popcnt pse pse36 pti rdrand rdtscp rep_good sep smep ss sse sse2
    sse4_1 sse4_2 ssse3 syscall tm tm2 tpr_shadow tsc tsc_adjust
    tsc_deadline_timer tsc_known_freq tsc_reliable vme vmx vnmi vpid
    xtopology xtpr
  Vulnerabilities:
  Type: gather_data_sampling status: Not affected
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Vulnerable: Clear CPU buffers attempted, no microcode;
    SMT disabled
  Type: meltdown mitigation: PTI
  Type: mmio_stale_data status: Unknown: No mitigations
  Type: retbleed status: Not affected
  Type: spec_rstack_overflow status: Not affected
  Type: spec_store_bypass status: Not affected
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: Retpolines, STIBP: disabled, RSB filling,
    PBRSB-eIBRS: Not affected
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx
    Integrated Graphics vendor: Acer Incorporated ALI driver: i915 v: kernel
    arch: Gen-8 process: Intel 14nm built: 2014-15 ports: active: eDP-1
    empty: DP-1, DP-2, HDMI-A-1, HDMI-A-2 bus-ID: 00:02.0 chip-ID: 8086:22b1
    class-ID: 0300
  Device-2: Realtek HD Webcam - driver: uvcvideo type: USB rev: 2.0
    speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-4:3 chip-ID: 0bda:57cc
    class-ID: 0e02 serial: <filter>
  Display: server: X.org v: 1.21.1.21 driver: X: loaded: modesetting
    alternate: fbdev,vesa dri: crocus gpu: i915 display-ID: :0.0
  Monitor-1: eDP-1 model: AU Optronics 0x235c built: 2013 res: 1366x768
    dpi: 136 gamma: 1.2 chroma: red: x: 0.569 y: 0.333 green: x: 0.329 y: 0.569
    blue: x: 0.153 y: 0.133 white: x: 0.314 y: 0.329
    size: 256x144mm (10.08x5.67") diag: 294mm (11.6") ratio: 16:9
    modes: 1366x768
  API: EGL v: 1.5 hw: drv: intel crocus platforms: device: 0 drv: crocus
    device: 1 drv: swrast surfaceless: drv: crocus inactive: gbm,wayland,x11
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: mesa v: 25.3.1-arch1.2
    note: incomplete (EGL sourced) renderer: Mesa Intel HD Graphics 405 (BSW),
    llvmpipe (LLVM 21.1.6 128 bits)
  API: Vulkan Message: No Vulkan data available.
  Info: Tools: api: eglinfo, glxinfo, vulkaninfo de: xfce4-display-settings
    x11: xdpyinfo, xprop, xrandr
Audio:
  Device-1: Intel Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series
    High Definition Audio vendor: Acer Incorporated ALI driver: snd_hda_intel
    v: kernel bus-ID: 00:1b.0 chip-ID: 8086:2284 class-ID: 0403
  API: ALSA v: k6.6.10-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator tools: alsactl,alsamixer,amixer
  Server-1: JACK v: 1.9.22 status: off tools: N/A
  Server-2: PipeWire v: 1.4.9 status: off with: 1: pipewire-pulse
    status: off 2: wireplumber status: off 3: pipewire-alsa type: plugin
    tools: pactl,pw-cat,pw-cli,wpctl
Network:
  Device-1: Intel Wireless 7265 driver: iwlwifi v: kernel pcie: gen: 1
    speed: 2.5 GT/s lanes: 1 bus-ID: 02:00.0 chip-ID: 8086:095a class-ID: 0280
  IF: wlp2s0 state: up mac: <filter>
  IP v4: <filter> type: dynamic noprefixroute scope: global
    broadcast: <filter>
  IP v6: <filter> type: noprefixroute scope: link
  Device-2: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet
    vendor: Acer Incorporated ALI driver: r8169 v: kernel pcie: gen: 1
    speed: 2.5 GT/s lanes: 1 port: 1000 bus-ID: 03:00.0 chip-ID: 10ec:8168
    class-ID: 0200
  IF: enp3s0 state: down mac: <filter>
  Info: services: No services found.
  WAN IP: <filter>
Bluetooth:
  Device-1: Intel Bluetooth wireless interface driver: btusb v: 0.8 type: USB
    rev: 2.0 speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 1-5:4 chip-ID: 8087:0a2a
    class-ID: e001
  Report: rfkill ID: hci0 rfk-id: 2 state: down bt-service: N/A rfk-block:
    hardware: no software: no address: see --recommends
Logical:
  Message: No logical block device data found.
RAID:
  Message: No RAID data found.
Drives:
  Local Storage: total: 126.73 GiB used: 43.98 GiB (34.7%)
  ID-1: /dev/sda maj-min: 8:0 vendor: LITE-ON model: CV1-CC128
    size: 119.24 GiB block-size: physical: 512 B logical: 512 B sata: 3.1
    speed: 6.0 Gb/s tech: SSD serial: <filter> fw-rev: 201
  SMART: yes state: enabled health: PASSED on: 12 hrs cycles: 2005
    read: 35.3 MiB written: 30.9 MiB
  ID-2: /dev/sdb maj-min: 8:16 vendor: SanDisk model: Cruzer size: 7.49 GiB
    block-size: physical: 512 B logical: 512 B type: USB rev: 2.0 spd: 480 Mb/s
    lanes: 1 mode: 2.0 tech: N/A serial: <filter> fw-rev: 7.01
  SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure?
  Message: No optical or floppy data found.
Partition:
  ID-1: / raw-size: 105.82 GiB size: 103.6 GiB (97.90%)
    used: 43.98 GiB (42.4%) fs: ext4 block-size: 4096 B dev: /dev/dm-1
    maj-min: 254:1 mapped: mydrive label: N/A uuid: N/A
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%)
    used: 4.3 MiB (1.4%) fs: vfat block-size: 512 B dev: /dev/sda1 maj-min: 8:1
    label: NO_LABEL uuid: 60B5-8A6A
Swap:
  Alert: No swap data was found.
Unmounted:
  ID-1: /dev/dm-0 maj-min: 254:0 mapped: ventoy size: 3.52 GiB fs: N/A
    label: MANJARO_XFCE_2313 uuid: 2024-01-13-03-08-57-00
  ID-2: /dev/sda2 maj-min: 8:2 size: 105.82 GiB fs: N/A label: N/A
    uuid: ffc0a8e3-431f-4e00-a614-fb14f0ba1b0f
  ID-3: /dev/sda3 maj-min: 8:3 size: 8.25 GiB fs: N/A label: N/A
    uuid: 3646510a-e376-4eb9-97a5-feec2aec0623
  ID-4: /dev/sda4 maj-min: 8:4 size: 4.88 GiB fs: ext4 label: unencryptediso
    uuid: e82df732-e38d-4a3a-bfb8-c78d2d2dea2f
  ID-5: /dev/sdb1 maj-min: 8:17 size: 7.45 GiB fs: N/A label: Ventoy
    uuid: 79BB-1DB4
  ID-6: /dev/sdb2 maj-min: 8:18 size: 32 MiB fs: N/A label: VTOYEFI
    uuid: 3F32-27F5
USB:
  Hub-1: 1-0:1 info: hi-speed hub with single TT ports: 7 rev: 2.0
    speed: 480 Mb/s (57.2 MiB/s) lanes: 1 mode: 2.0 chip-ID: 1d6b:0002
    class-ID: 0900
  Device-1: 1-1:2 info: SanDisk Cruzer Micro Flash Drive type: mass storage
    driver: usb-storage interfaces: 1 rev: 2.0 speed: 480 Mb/s (57.2 MiB/s)
    lanes: 1 mode: 2.0 power: 200mA chip-ID: 0781:5151 class-ID: 0806
    serial: <filter>
  Device-2: 1-4:3 info: Realtek HD Webcam - type: video driver: uvcvideo
    interfaces: 2 rev: 2.0 speed: 480 Mb/s (57.2 MiB/s) lanes: 1 mode: 2.0
    power: 500mA chip-ID: 0bda:57cc class-ID: 0e02 serial: <filter>
  Device-3: 1-5:4 info: Intel Bluetooth wireless interface type: bluetooth
    driver: btusb interfaces: 2 rev: 2.0 speed: 12 Mb/s (1.4 MiB/s) lanes: 1
    mode: 1.1 power: 100mA chip-ID: 8087:0a2a class-ID: e001
  Hub-2: 2-0:1 info: super-speed hub ports: 6 rev: 3.0
    speed: 5 Gb/s (596.0 MiB/s) lanes: 1 mode: 3.2 gen-1x1 chip-ID: 1d6b:0003
    class-ID: 0900
Sensors:
  System Temperatures: cpu: 48.0 C mobo: N/A
  Fan Speeds (rpm): N/A
Repos:
  Packages: pm: pacman pkgs: 1633 libs: 472 tools: pamac pm: flatpak pkgs: 0
  Active pacman repo servers in: /etc/pacman.d/mirrorlist
    1: https://manjarolinux-mirror.cloud.mu/stable/$repo/$arch
    2: https://manjaro.mirror.liquidtelecom.com/stable/$repo/$arch
Processes:
  Message: No process data available.
Info:
  Processes: 0 Power: uptime: 54m states: freeze,mem,disk suspend: deep
    avail: s2idle wakeups: 0 hibernate: platform avail: shutdown, reboot,
    suspend, test_resume image: 1.47 GiB Init: systemd v: 258
    default: graphical tool: systemctl
  Compilers: clang: 21.1.6 gcc: 15.2.1 Shell: N/A (sudo) inxi: 3.3.40

Mod edit:- Copied inxi output here as a convenience for others. No charge.

Please realise I have never been in a similar situation so my suggestions is a trial-and-error approach - I am sometimes guessing.

What we do know

  • the dysfunctional system has file system errors
  • file system errors causes applications and files in general to be incomplete or missing
  • you have fixed the file system - repaired the errors
  • the package lists on the ISO and in the the original file system root contains packages no longer in the repo.

Ensure your system is booted in EFI mode
Ensure you have both root and EFI mounted

Something like this when you run lsblk

sda
├─sda1        /mnt/boot/efi
├─sda2
│ └─mydrive   /mnt

Now we need to get the file system back into shape - where files are complete and in the right place.

For that to happen we need a valid list of packages we can feed to pacman and unpack onto the damaged file system which is mounted on /mnt

This can be done be retrieving the latest ISO - booting and starting the process all over OR we can instead download the package list which - usually - exist side-by-side with the ISO.

Besides downloading using CDN77 the images are initially stored on sourceforge before they are transferred to ‘where-ever’ they are stored.

Since you are - apparently - using xfce - the current preview is at Manjaro Linux - Browse /xfce/25.1-pre2 at SourceForge.net with a package list next to it.

Assuming you are using the full ISO - and you have the system booted using the live iso - use this command - in a virtual terminal - to retrieve the file

wget -O xfce-pkglist.txt https://sourceforge.net/projects/manjarolinux/files/xfce/25.1-pre2/manjaro-xfce-25.1-pre2-251215-linux618.iso.pkgs/download

When the list has been downloaded you are ready for next step.

We will need to rethink the approach of getting the dysfunctional file system restored.

It was a mistake of me suggesting the --needed argument - as this will skip the package if the metadata suggests it is up-to-date.

What we need to do is to force the file system to be rebuilt.

Open a terminal and switch to root context

su

First move the /usr tree in the mount point to a backup

mv /mnt/usr /mnt/usr.bak

Then run pacman with instruction to operate on the /mnt mountpoint feeding the filtered package list (version numbers removed)

pacman -Syy --noconfirm --root /mnt $(cat xfce-pkglist.txt | awk '{print $1;}')

After the package sync - then enter a chrooted environment using /mnt

manjaro-chroot /mnt /bin/bash

Rebuild init and grub

mkinitcpio -P
install-grub

I cannot think of anything that could be done now other than exiting chroot and rebooting.

exit
reboot

When / If you have successfully restored your system to a functional state - remove the backup copy we made

sudo rm -rf /usr.bak
2 Likes

Ok.
The pacman command gives an error, which I could propably resolve if I remove the –noconfirm flag, right? like this it doesn’t let me choose…

looking for conflicting packages…
:: pamac-gtk3 and pamac-gtk are in conflict. Remove pamac-gtk? [y/N]
error: unresolvable package conflicts detected
error: failed to prepare transaction (conflicting dependencies)
:: pamac-gtk3 and pamac-gtk are in conflict

Mod edit:- Corrected display of pre-formatted text. No charge.
Please re-evaluate the method you use for pasting pre-formatted text.
See How to Post Command Output as Preformatted Text :eyes:

yes - removing the --no-confirm will cause pacman to ask

Great, I resolved the conflicts and now the update is running!
Will get back, as soon as it is done (might need an hour or two due to slow internet!
Thanks for not giving up and trying to help me so much! Highly appreciated :green_heart:

1 Like

Good. As we have moved the existing system files (/usr) to another location - any applications you may have added besides the ISO defaults will not exist.

Do not restore anything from the /usr.bak folder.

Install any extra applications from the repo.

Assuming your home folder has not gone sideways - any user data and configurations will be intact - or you can restore from a backup.

Next problem :smiley:

Download stuck at 70%, system frozen, no response to anything…

The only thing I could imagine, is a full flash drive of the live system!? Stupid me didn’t check the remaining space before starting the download..

I guess it is another hard reset to find out?

That shouldn’t be a problem as we operate on the mountpoint

That could be an issue - since we are using a rescue iso - there is no harm done in removing the usr.bak folder inside the container as it will be recreated by pacman

sudo rm -rf /mnt/usr.bak

Then rerun the command

It should be possible to use the booted ISO system - if the system is completely locked - no input possible - see if you can get to a TTY - then kill the pacman process

pkill pacman

I fear I can’t access any TTY..

Could you expand on that a little?
Did you try every available TTY?
What happens with each attempt?
Do you see any command output? What is it?

That is strange - but it does point into the direction of more serious problems.

At this point - try some repeated Ctrl+Alt+Del at least 7 - see if the system reboots - if not you’ll have to hard reset.

I cannot guess - but from this internet endpoint - I’d seriously consider to salvage what can be salvaged from your home folder inside the luks container.

Then wipe the system - do a clean reinstallation using the latest preview from sourceforge - linked above.

If you care for giving it another shot - then remove the backup copy you made in side the container - and rerun the rebuilding of the file system.

I tried Ctrl alt f1-12 and Ctrl alt shift f1-12. No reaction to anything

Did you give it enough time to react? I find sometimes that I must wait maybe 10-20 seconds for a response – not normally, but at times when a related issue might exist.

Ctrl+Alt+F1 <=> F6 are the usual combo’s.

1 Like

I did… Did a hard reset now and will start afresh with @linux-aarhus instructions from before. and will check free space before starting the downloads.

1 Like

Fair enough.

For general reference, there is a safer way…

1 Like

Is that actually so? pacman is working on the mountpoint, but does it download to the mountpoint? Because if the packages are downloaded on the flash drive, it will probably run full. I just checked and it has 2.6Gb free space. That would not be enough for the full update.

The mountpoint shows more than 50Gb free space, so that should not be a problem.

Also the live system doesn’t seem to have swap and I only have 4Gb of RAM. Could that lead to problems in the process? Any use in mounting the swap partition of the broken system?

1 Like

That should be easy to check by looking at the content of the respective directories:

ls -hl /var/cache/pacman/pkg
should be empty or nearly empty since this is on the flash drive
(strangely there are a few packages present in my Xfce minimal ISO from October)

ls -hl /mnt/var/cache/pacman/pkg
should contain all the packages that are about to be installed
this is on the hard drive

I have some Manjaro VM’s and each one only has got 4 GB of RAM and no swap - never had a problem due to this
although I did the last big update in a TTY without being logged in to the graphical session.

my guess was right, it is downloading to the flash. Can I change pacman download location?
If there is no easy way to change that, I would have to get a bigger flash from somebody and prepare it with the iso.

edit: stupid me again :grimacing: getting a bigger flash doesn’t help. The ISO is of that size. Could I expand that with gparted?!

edit: It made me thinking now. If pacman is not downloading to the /mnt folder, is the command actually working and it will update the system in /mnt, or am I wasting time and bandwidth to try to update the live system? :rofl:

Since you said, you are also guessing, can somebody else confim, that the command would actually work like this (pacman on the live system working on the packages in the mounted system)?

pacman -Syy --noconfirm --root /mnt $(cat xfce-pkglist.txt | awk ‘{print $1;}’)

edit3: I figured, that I can change the download location in pacman.conf, so I changed it to the folder in /mnt and rerun the update.

lets see :face_with_peeking_eye:

This is what I meant with trial-and-error :slight_smile:

I didn’t realise that could be an issue.

You can add --cachedir /mnt/var/cache/pacman/pkg to the pacman command.

I am having in a learning experience too.

2 Likes