Updating kernels fails with Damaged tar archive error

Strontium · 21 March 2025 10:30

I just tried a system update and all the kernels fail to update with the following error:

==> Building image from preset: /etc/mkinitcpio.d/linux612.preset: 'fallback'
==> Using default configuration file: '/etc/mkinitcpio.conf'
  -> -k /boot/vmlinuz-6.12-x86_64 -g /boot/initramfs-6.12-x86_64-fallback.img -S autodetect
==> Starting build: '6.12.17-1-MANJARO'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [microcode]
  -> Running build hook: [modconf]
  -> Running build hook: [kms]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [block]
  -> Running build hook: [filesystems]
==> WARNING: Possibly missing '/bin/bash' for script: /usr/bin/mount.unionfs
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-6.12-x86_64-fallback.img'
  -> Early uncompressed CPIO image generation successful
bsdtar: Error reading archive -: Damaged tar archive (bad header checksum)
bsdtar: Error exit delayed from previous errors.
==> ERROR: Initcpio image generation FAILED: 'bsdtar (step 2)' reported an error

Kernel update worked on the 10th of March, because I had a backup of that kernel prior to the update (or my machine would now be totally broken).

I can’t find any resource that explains why this error occurs now, or how to correct it.
My /etc/mkinitcpio.conf is the same as the latest /etc/mkinitcpio.conf.pacnew file present.

I have 6.6, 6.12 and 6/13 kernels installed (Manjaro standard kernels) and they all have the same issue.

soundofthunder · 21 March 2025 10:33

Possible bad mirror?

Manjaro Mirrors

Aragorn · 21 March 2025 10:39

Did you refresh your mirrors first? You should make that a habit with every bundled update.

Strictly speaking it should not be necessary, but we’re not living in a perfect universe, and Murphy has no mercy.

sudo pacman-mirrors-f && sudo pacman -Syu linux66 linux612 linux613 mkinitcpio
sudo mkinitcpio -P
sudo update-grub

That could be part of the problem.

For one — although the following won’t be the cause of your issue — your HOOKS line includes consolefont, but as you can see, consolefont is not defined. So you should either remove the consolefont hook or set a proper font in /etc/vconsole.conf.

Strontium · 21 March 2025 12:29

I had an original slightly modified mkinitcpio.conf which removed consolefont. When i started getting this error, the first thing i tried was making it the same as the distribution default. I am aware of the consolefont is just a noisy warning and doesn’t actually hurt anything, well, it never has before.

Updating mirrors does not help at all:

::United_States   : https://mirrors.sonic.net/manjaro/stable
::Uruguay         : https://manjaro.repo.cure.edu.uy/stable
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
:: Synchronising package databases...
 core is up to date
 extra is up to date
 multilib is up to date
warning: linux66-6.6.83-1 is up to date -- reinstalling
warning: linux612-6.12.19-1 is up to date -- reinstalling
warning: linux613-6.13.7-1 is up to date -- reinstalling
warning: mkinitcpio-39.2-4 is up to date -- reinstalling
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (4) linux612-6.12.19-1  linux613-6.13.7-1  linux66-6.6.83-1  mkinitcpio-39.2-4

Total Installed Size:  424.57 MiB
Net Upgrade Size:        0.00 MiB

:: Proceed with installation? [Y/n]
(4/4) checking keys in keyring                                                         [--------------------------------------------------] 100%
(4/4) checking package integrity                                                       [--------------------------------------------------] 100%
(4/4) loading package files                                                            [--------------------------------------------------] 100%
(4/4) checking for file conflicts                                                      [--------------------------------------------------] 100%
(4/4) checking available disk space                                                    [--------------------------------------------------] 100%
:: Running pre-transaction hooks...
(1/1) Remove upgraded DKMS modules
==> dkms remove --no-depmod corefreq/2.0.1 -k 6.13.7-1-MANJARO
==> dkms remove --no-depmod openrazer-driver/3.10.0 -k 6.13.7-1-MANJARO
==> dkms remove --no-depmod vboxhost/7.1.6_non_OSE -k 6.13.7-1-MANJARO
==> dkms remove --no-depmod corefreq/2.0.1 -k 6.6.83-1-MANJARO
==> dkms remove --no-depmod openrazer-driver/3.10.0 -k 6.6.83-1-MANJARO
==> dkms remove --no-depmod vboxhost/7.1.6_non_OSE -k 6.6.83-1-MANJARO
==> dkms remove --no-depmod corefreq/2.0.1 -k 6.12.19-1-MANJARO
==> dkms remove --no-depmod openrazer-driver/3.10.0 -k 6.12.19-1-MANJARO
==> dkms remove --no-depmod vboxhost/7.1.6_non_OSE -k 6.12.19-1-MANJARO
:: Processing package changes...
(1/4) reinstalling mkinitcpio                                                          [--------------------------------------------------] 100%
(2/4) reinstalling linux66                                                             [--------------------------------------------------] 100%
(3/4) reinstalling linux612                                                            [--------------------------------------------------] 100%
(4/4) reinstalling linux613                                                            [--------------------------------------------------] 100%
:: Running post-transaction hooks...
( 1/10) Reloading system manager configuration...
( 2/10) Restarting marked services...
( 3/10) Creating temporary files...
( 4/10) Arming ConditionNeedsUpdate...
( 5/10) Updating module dependencies...
( 6/10) Install DKMS modules
==> dkms install --no-depmod vboxhost/7.1.6_non_OSE -k 6.13.7-1-MANJARO
==> dkms install --no-depmod vboxhost/7.1.6_non_OSE -k 6.12.19-1-MANJARO
==> dkms install --no-depmod openrazer-driver/3.10.0 -k 6.13.7-1-MANJARO
==> dkms install --no-depmod openrazer-driver/3.10.0 -k 6.6.83-1-MANJARO
==> dkms install --no-depmod vboxhost/7.1.6_non_OSE -k 6.6.83-1-MANJARO
==> dkms install --no-depmod corefreq/2.0.1 -k 6.6.83-1-MANJARO
==> dkms install --no-depmod corefreq/2.0.1 -k 6.13.7-1-MANJARO
==> dkms install --no-depmod openrazer-driver/3.10.0 -k 6.12.19-1-MANJARO
==> dkms install --no-depmod corefreq/2.0.1 -k 6.12.19-1-MANJARO
==> depmod 6.13.7-1-MANJARO
==> depmod 6.6.83-1-MANJARO
==> depmod 6.12.19-1-MANJARO
( 7/10) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux612.preset: 'default'
==> Using default configuration file: '/etc/mkinitcpio.conf'
  -> -k /boot/vmlinuz-6.12-x86_64 -g /boot/initramfs-6.12-x86_64.img
==> Starting build: '6.12.19-1-MANJARO'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [microcode]
  -> Running build hook: [modconf]
  -> Running build hook: [kms]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
  -> Running build hook: [block]
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating bzip2-compressed initcpio image: '/boot/initramfs-6.12-x86_64.img'
  -> Early uncompressed CPIO image generation successful
bsdtar: Error reading archive -: Damaged tar archive (bad header checksum)
bsdtar: Error exit delayed from previous errors.
==> ERROR: Initcpio image generation FAILED: 'bsdtar (step 2)' reported an error

mithrial · 21 March 2025 16:02

Is there enough space on /boot?

Nachlese · 21 March 2025 16:09

… or on / (root)

lsblk -f

and/or

df -h /

soundofthunder · 21 March 2025 19:34

@Strontium

Please provide system information as described (below);

System Information

Output of this command (formatted according to forum requirements) may be useful for those wishing to help:

inxi --filter --verbosity=8

or the short form:

inxi -zv8

Privacy:- Those with privacy concerns, note that when -z, or --filter is used, all personally identifiable information is filtered out from the resulting inxi output.

mithrial · 21 March 2025 20:11

Is bz2 the default compression? Maybe there is a wrong flag somewhere in the compression settings.

BG405 · 21 March 2025 20:16

I just checked my last upgrade log and found this:

==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-6.13-x86_64.img'
  -> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful

… so I guess this must have been changed, then.

Nachlese · 21 March 2025 21:44

Nothing has been changed.
But what is used is configurable in /etc/mkinitcpio.conf
default is still gzip
but it can be changed to … whatever

in /etc/mkinitcpio.conf is this default:

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
#COMPRESSION="zstd"
#COMPRESSION="cat"

I recall that lower kernel versions could not deal with some of the available options
like xz or zstd, for example
Not sure whether that is in play here.

I use “cat” in some of my VM’s
it’s more space consumed, but also faster during updates …

soundofthunder · 21 March 2025 21:53

From my mkinitcpio.conf:

@Strontium

Please post the command output of;

cat /etc/mkinitcpio.conf

Nachlese · 21 March 2025 21:58

Hmm - the above was copy/pasted from my, very much default, /etc/mkinitcpio.conf
(from one of my VM’s - an Xfce4 instance)

That line or message,
or call it a caveat, perhaps,
is not in there.

But I have seen it.
Somewhere …

perhaps it was lost during a .pacnew merge … I guess we will never know.

I’ll look at the same file in other Manjaro VM’s … see where there are differences …

soundofthunder · 21 March 2025 22:07

Aside:-

This also highlights the importance of managing .pacnew files (Pacnew and Pacsave).

There can be subtle differences in files from one machine to another; possibly due to customisations by the User; or possibly from never having changed it since the original installation (potentially many years ago).

These compression methods are listed in (my) mkinitcpio.conf, for example; and I’ve never touched the file except to change hooks.

#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

Aragorn · 22 March 2025 02:40

If the contents of /boot are on a btrfs filesystem with compression enabled, then it is better not to use any compression for mkinitcpio, because with two compression algorithms working on the same file, the end result could end up bigger than the original.

Strontium · 22 March 2025 08:21

Ok, so you see BZ2 in the above log because I was trying to see if changing the compressor helps. It does not.
The filesystem is ZFS, been running this same setup for years. /boot is ZFS. It use refind for the UEFI loader, because I have redundant UEFI across my three drives. And uses ZFSBoot to actually boot the OS. So UEFI → Refind → ZFSBoot → Manjaro /boot.

/boot is not its own volume and I have ~728G free on my root pool.
If it’s compressed or not won’t matter too much to ZFS because it won’t waste time by compressing detectably uncompressible data.

Ok, I have worked out whats causing the problem. bsdtar from libarchive 3.7.7-2` is creating broken archives.

I hacked mkinitcpio with the following inside the create_image() function before it runs the image creation pipeline.

 # Debug pipeline
    LC_ALL=C.UTF-8 find . -mindepth 1 -printf '%P\0' \
        | LC_ALL=C.UTF-8 sort -z \
        | LC_ALL=C.UTF-8 bsdtar --uid 0 --gid 0 --null -cnf - -T - > /tmp/badimage.tar

And I then try and display its contents using bsdtar, and I get this:

❯ bsdtar -t -f /tmp/badimage.tar
VERSION
bin
buildconfig
config
...
usr/lib/firmware/amdgpu/gc_10_3_7_pfp.bin
usr/lib/firmware/amdgpu/gc_10_3_7_rlc.bin
usr/lib/firmware/amdgpu/gc_11_0_0_imu.bin
usr/lib/firmware/amdgpu/gc_11_0_0_me.bin
bsdtar: Damaged tar archive (bad header checksum)
bsdtar: Retrying...
bsdtar: Damaged tar archive (bad header checksum)
bsdtar: Retrying...
bsdtar: Damaged tar archive (bad header checksum)
bsdtar: Retrying...
bsdtar: Damaged tar archive (bad header checksum)
bsdtar: Retrying...
bsdtar: Damaged tar archive (bad header checksum)
bsdtar: Retrying...

So the image bsdtar is creating is BAD, and the next command in the pipeline detects that and fails.

This is the libarchive version I have installed:

❯ pacman -Q --info libarchive
Name            : libarchive
Version         : 3.7.7-2
Description     : Multi-format archive and compression library
Architecture    : x86_64
URL             : https://libarchive.org/
Licenses        : BSD
Groups          : None
Provides        : libarchive.so=13-64
Depends On      : acl  libacl.so=1-64  bzip2  libbz2.so=1.0-64  libxml2  libxml2.so=2-64  lz4  openssl  libcrypto.so=3-64  xz  liblzma.so=5-64
                  zlib  libz.so=1-64  zstd  libzstd.so=1-64
Optional Deps   : None
Required By     : appstream-glib  cmake  debuginfod  engrampa  evince  flatpak  fwupd  gvfs  libappimage  libgxps  mkinitcpio  ostree  pacman
                  pkgfile  rpi-imager  smbclient  tesseract  vagrant  vlc
Optional For    : systemd
Conflicts With  : None
Replaces        : None
Installed Size  : 1225.71 KiB
Packager        : Christian Hesse <eworm@archlinux.org>
Build Date      : Wed 12 Mar 2025 03:48:38
Install Date    : Fri 21 Mar 2025 14:34:31
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : Signature

Downgrading libarchive with:

sudo pacman -U https://archive.archlinux.org/packages/l/libarchive/libarchive-3.7.7-1-x86_64.pkg.tar.zst

after that, i can again run mkinitramfs without issue.
So, at least on my system, libarchive-3.7.7-2 is broken.
I note the latest libarchive is 3.7.7-4 in https://archive.archlinux.org/packages/l/libarchive/libarchive-3.7.7-4-x86_64.pkg.tar.zst

Trying to update libarchive to the latest version pacman -Syu libarchive only gives me 3.7.7-2 which is broken again when i re-install that.
I also just tried 3.7.7-3 and 3.7.7-4 and they both work correctly. So its only 3.7.7-2 which is broken.

Strontium · 22 March 2025 08:23

TLDR.
The Solution is, don’t use bsdtar 3.7.7-2. If its in the official repos, like mine is, it can be force upgraded to a later version that works with:

sudo pacman -U https://archive.archlinux.org/packages/l/libarchive/libarchive-3.7.7-4-x86_64.pkg.tar.zst
sudo mkinitcpio -P

soundofthunder · 22 March 2025 10:10

You’re correct - libarchive v3.7.7-2 is currently in the Stable repository; v3.7.7-3 is in both Unstable and Testing - I dare say it will reach Stable soon. @Yochanan

Branch compare; libarchive

Yochanan · 22 March 2025 21:32

I’m working on GNOME & NVIDIA stuff at the moment, we’ll see.

By the way, there’s now libarchive 3.7.7-4 available in the unstable branch.

DeLinuxCo · 23 March 2025 22:16

This issue appears to be causing the BUILDISO to fail during installation from a live ISO, Been trying to test a new ISO and keep getting this error at the end of the installation. Although, the ISO builds with no issue.

==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-6.13-x86_64-fallback.img'
  -> Early uncompressed CPIO image generation successful
bsdtar: Error reading archive -: Damaged tar archive (bad header checksum)
bsdtar: Error exit delayed from previous errors.
==> ERROR: Initcpio image generation FAILED: 'bsdtar (step 2)' reported an error
2025-03-23 - 07:41:36 [6]: void Calamares::ViewManager::onInstallationFailed(const QString&, const QString&)
2025-03-23 - 07:41:36 [6]:     Calamares will quit when the dialog closes. 
2025-03-23 - 07:41:37 [6]: void Config::doNotify(bool, bool)
2025-03-23 - 07:41:37 [6]:     Notification not sent; completion: failed 
2025-03-23 - 07:41:37 [6]: void {anonymous}::PowerManagementInterface::uninhibitSleep()
2025-03-23 - 07:41:37 [6]:     Sleep was never inhibited.

Entire Log: https://termbin.com/nw5g

Strontium · 24 March 2025 06:28

My experience is this will be causing any initramfs images to be corrupt. They are much shorter than a valid one. So while the ISO might build correctly, it probably won’t boot properly because of the corrupt initramfs.

For me it seems 3.7.7-2 of libarchive is broken and not safe for creating initramfs images. And transitively so is mkinitramfs because it depends on it.

I wonder how libarchive 3.7.7-2 made it into stable with this issue? It’s clearly not unique to me.

If one has a look at the changes.
3.7.7-2 introduced a backport from upstream: upgpkg: 3.7.7-2: fast-forward upstream/patch/3.7 & backport fixes for CVEs (7ea72c6a) · Commits · Arch Linux / Packaging / Packages / libarchive · GitLab
3.7.7-3 reverted it. upgpkg: 3.7.7-3: drop changes from upstream/patch/3.7 (993255c5) · Commits · Arch Linux / Packaging / Packages / libarchive · GitLab

One can imagine why this was reverted.