Pacman fails because of missing keys

marinheiro · 25 January 2023 22:44

Very newbie problem, probably simple to fix (I hope)

Running the pacman gui I get a panel which says ‘Failed to commit transaction’, with no more explanation.

Running pacman from the commandline I get error messages about keys that can’t be installed eg:

error: key "CAA6A59611C7F07E" could not be imported
error: required key missing from keyring
error: failed to commit transaction (unexpected error)

and the update fails to continue. Trying to follow various suggestions about updating keys from older threads doesn’t improve things, and I’m afraid I’m just going to mess the system up further from lack of knowledge.

Graham

marinheiro · 26 January 2023 15:41

So, I am stuck in a loop: if I try to update or restore the keyring, pacman fails because of missing keys. Following the instructions on the wiki:

Remove /etc/pacman.d/gnupg
sudo pacman -Sy gnupg archlinux-keyring manjaro-keyring

..........
warning: gnupg-2.2.40-1 is up to date -- reinstalling
warning: archlinux-keyring-20221220-1 is up to date -- reinstalling
warning: manjaro-keyring-20221028-4 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (3) archlinux-keyring-20221220-1  gnupg-2.2.40-1
             manjaro-keyring-20221028-4

Total Installed Size:  10.85 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n] Y
(3/3) checking keys in keyring                     [##########################] 100%
downloading required keys...
:: Import PGP key 139B09DA5BF0D338, "David Runge <dvzrv@archlinux.org>"? [Y/n] Y
error: key "139B09DA5BF0D338" could not be looked up remotely
:: Import PGP key 6D42BDD116E0068F, "Christian Hesse <eworm@archlinux.org>"? [Y/n]
error: key "6D42BDD116E0068F" could not be looked up remotely
:: Import PGP key CAA6A59611C7F07E, "Philip Müller <philm@manjaro.org>"? [Y/n] Y
error: key "CAA6A59611C7F07E" could not be looked up remotely
error: required key missing from keyring
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.

I can’t reinitialise the keys either:

sudo pacman-key --init
==> Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key...
gpg: agent_genkey failed: Unknown system error
gpg: key generation failed: Unknown system error
gpg: Done
==> Updating trust database...
gpg: no need for a trustdb check

Any other suggestions?

Thanks
Graham

mithrial · 26 January 2023 15:52

marinheiro · 26 January 2023 16:12

That fails for me, unfortunately: exits with an error because the tmp directory contains no tar files. I tried running the line which should have fetched them manually and got this:

sudo pacman --sync --refresh --downloadonly --noconfirm --cachedir /tmp/tmp.WaXMbDud7u --config /tmp/tmp.WaXMbDud7u/pacman.conf archlinux-keyring manjaro-keyring
:: Synchronising package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
resolving dependencies...

Packages (2) archlinux-keyring-20221220-1  manjaro-keyring-20221028-4

Total Download Size:  0.00 MiB

:: Proceed with download? [Y/n] 
(2/2) checking keys in keyring                        [############################] 100%
(2/2) checking package integrity                      [############################] 100%

mithrial · 26 January 2023 16:24

Come on, what is the error?

marinheiro · 26 January 2023 16:47

Sorry, I guess I wasn’t clear. The error message from the script is:

error: '/tmp/tmp.WaXMbDud7u/*.tar.*': could not find or read package

And this results from:

Total Download Size:  0.00 MiB

when attempting to download archlinux-keyring-20221220-1 and manjaro-keyring-20221028-4 though these do both exist.

mithrial · 26 January 2023 16:52

Is your disk full?

marinheiro · 26 January 2023 17:01

277G available

Sorry, 16G available in /tmp.

marinheiro · 26 January 2023 21:10

I manually downloaded copies of the two keyring files and used those in the script. They read ok, but the script still doesn’t work (ie. if I let it carry on to the actual system update, it still fails on missing keys). The problem now seems to be with

pacman-key --init

[INFO] Remove pacman's gnupg
[INFO] Re-initilize pacman's gnupg
Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key...
gpg: agent_genkey failed: Unknown system error
gpg: key generation failed: Unknown system error
gpg: Done

Later in the script when it tries to reinstall the two keyring files I get the error:

==> ERROR: There is no secret key available to sign with.
==> Use 'pacman-key --init' to generate a default secret key.

I don’t know how to get any more helpful error messages from pacman-key --init, there’s no ‘verbose’ option.

dragan · 26 January 2023 23:00

Just by looking at it (I’m not too intimate with troubleshooting pacman/keyrings) this seems like a gpg issue, try looking at the HowTo Find error messages and filter for gpg errors, it looks to me like your gpg setup is screwed up in some way, just a thought (try troubleshooting that e.g. by doing some basic checkup, listing w/o changing anything, see gpg / GnuPG on archwiki). It’s what I’d do, but careful if you don’t know what you’re doing, as normally you don’t need to touch gpg much, unless it’s misbehaving (maybe someone has something quicker and more to the point).

marinheiro · 26 January 2023 23:09

Yep, I came to the same conclusion: I tried fixing gpg directly using this reddit link, but that made no difference - and I got nervous about the amount of damage i might do to my system following random threads from reddit…

I also tried to work out how pacman-key was calling gpg during initialisation to see if I could add a ‘–verbose’ parameter to gpg itself, but failed there too.

megavolt · 27 January 2023 03:27

Hello @marinheiro

The problem here is that pacman doesn’t clean the whole cache and therefore doesn’t download the packages again in a custom cache folder.

I fixed it here: https://notabug.org/megavolt/random-scripts/src/master/fix-gpg-pacman.sh#L32-L37

Please try again:

bash <(curl -s "https://notabug.org/megavolt/random-scripts/raw/master/fix-gpg-pacman.sh"

marinheiro · 27 January 2023 09:51

That doesn’t change the failure to download the new keyring packages, I’m afraid. The error message here is exactly the same as before.

However, the conclusion I came to is that the fundamental problem is actually before this point in the script:

[INFO] Re-initilize pacman's gnupg 
Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key..
gpg: agent_genkey failed: Unknown system error
gpg: key generation failed: Unknown system error
gpg: Done

which suggests 2 problems:

The script should bail out when genkey fails, but it just carries on
The failure of genkey itself: I can’t find any more information about this ‘unknown system error’ (journalctl doesn’t have any information)

Graham

megavolt · 27 January 2023 11:57

Well I tested it on my system. That error should not be there:

That is something else:

marinheiro:

[INFO] Re-initilize pacman's gnupg 
Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key..
gpg: agent_genkey failed: Unknown system error
gpg: key generation failed: Unknown system error
gpg: Done

Guess pacman’s gnupg folder was not removed completely. And yes, on my system it didn’t do it as well. Removing the quotation marks solved it, at least for me.

Sorry, that problems never happens on my systems with Manjaro and I have to follow what people report about this and yeah, reproducing such problems with gpg is not easy. Some people are really talented at breaking their system, I am not.

Please try again:

bash <(curl -s "https://notabug.org/megavolt/random-scripts/raw/master/fix-gpg-pacman.sh")

marinheiro · 27 January 2023 12:37

=> Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key...
gpg: agent_genkey failed: Unknown system error
gpg: key generation failed: Unknown system error
gpg: Done
==> Updating trust database...
gpg: no need for a trustdb check
[INFO] Removing package cache 
sudo: unable to execute /bin/bash: Argument list too long

megavolt · 27 January 2023 12:44

I really have no idea what is again a problem here… look, it should work like that:

$ bash <(curl -s "https://notabug.org/megavolt/random-scripts/raw/master/fix-gpg-pacman.sh")
[INFO] Removing lock files of pamac and pacman
[INFO] Switching to global mirror (Manjaro's CDN)
::INFO Downloading mirrors from Manjaro
::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json
::INFO => Mirror status: https://repo.manjaro.org/status.json
::INFO User generated mirror list
::------------------------------------------------------------
::INFO Custom mirror file saved: /var/lib/pacman-mirrors/custom-mirrors.json
::INFO Using custom mirror file
::INFO Querying mirrors - This may take some time
  0.760 Global         : https://mirrors.fossho.st/manjaro/
  0.470 Global         : https://mirrors.manjaro.org/repo/
::INFO Writing mirror list
::Global          : https://mirrors.manjaro.org/repo/unstable/$repo/$arch
::Global          : https://mirrors.fossho.st/manjaro/unstable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
::INFO To reset custom mirrorlist 'sudo pacman-mirrors -id'
::INFO To remove custom config run  'sudo pacman-mirrors -c all'
::INFO Downloading mirrors from Manjaro
::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json
::INFO => Mirror status: https://repo.manjaro.org/status.json
::INFO Using custom mirror file
::INFO Querying mirrors - This may take some time
  0.202 Global         : https://mirrors.fossho.st/manjaro/
  0.468 Global         : https://mirrors.manjaro.org/repo/
::INFO Writing mirror list
::Global          : https://mirrors.fossho.st/manjaro/unstable/$repo/$arch
::Global          : https://mirrors.manjaro.org/repo/unstable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
[INFO] Remove pacman's gnupg
removed '/etc/pacman.d/gnupg/S.gpg-agent'
removed '/etc/pacman.d/gnupg/S.gpg-agent.browser'
removed '/etc/pacman.d/gnupg/S.gpg-agent.extra'
removed '/etc/pacman.d/gnupg/S.gpg-agent.ssh'
removed '/etc/pacman.d/gnupg/gpg-agent.conf'
removed '/etc/pacman.d/gnupg/gpg.conf'
removed '/etc/pacman.d/gnupg/openpgp-revocs.d/F6893E86D66E78B1BABABC674D2E5D15A237D3C7.rev'
removed directory '/etc/pacman.d/gnupg/openpgp-revocs.d'
removed '/etc/pacman.d/gnupg/private-keys-v1.d/873A4530121DDEBC9CD9E42EB7CB5171C92A18BF.key'
removed directory '/etc/pacman.d/gnupg/private-keys-v1.d'
removed '/etc/pacman.d/gnupg/pubring.gpg'
removed '/etc/pacman.d/gnupg/pubring.gpg~'
removed '/etc/pacman.d/gnupg/secring.gpg'
removed '/etc/pacman.d/gnupg/tofu.db'
removed '/etc/pacman.d/gnupg/trustdb.gpg'
[INFO] Re-initilize pacman's gnupg
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
==> Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key...
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/982F0889726C40260BD443C628696AC328387DCA.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
[INFO] Removing package cache
[INFO] Downloading the newest packages which contains the gpg keys in /tmp/tmp.pKPMb3wu93
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
resolving dependencies...

Packages (2) archlinux-keyring-20221220-1  manjaro-keyring-20221028-4

Total Download Size:  1.66 MiB

:: Proceed with download? [Y/n]
:: Retrieving packages...
 archlinux-keyring-20221220-...  1153.7 KiB  3.00 MiB/s 00:00 [##################################] 100%
 manjaro-keyring-20221028-4-any   541.6 KiB  2.40 MiB/s 00:00 [##################################] 100%
 Total (2/2)                     1695.3 KiB  2.78 MiB/s 00:01 [##################################] 100%
(2/2) checking keys in keyring                                [##################################] 100%
(2/2) checking package integrity                              [##################################] 100%
[INFO] Installing Keyring Packages
loading packages...
warning: archlinux-keyring-20221220-1 is up to date -- reinstalling
warning: manjaro-keyring-20221028-4 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (2) archlinux-keyring-20221220-1  manjaro-keyring-20221028-4

Total Installed Size:  2.30 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n]
(2/2) checking keys in keyring                                [##################################] 100%
(2/2) checking package integrity                              [##################################] 100%
(2/2) loading package files                                   [##################################] 100%
(2/2) checking for file conflicts                             [##################################] 100%
(2/2) checking available disk space                           [##################################] 100%
:: Running pre-transaction hooks...
(1/1) Creating Timeshift snapshot before upgrade...
==> skipping timeshift-autosnap due skipAutosnap in /etc/timeshift-autosnap.conf set to TRUE.
:: Processing package changes...
(1/2) reinstalling archlinux-keyring                          [##################################] 100%
==> Appending keys from archlinux.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signed 6 keys.
==> Importing owner trust values...
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: inserting ownertrust of 4
gpg: setting ownertrust to 4
==> Disabling revoked keys in keyring...
  -> Disabled 57 keys.
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   6  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   6  signed:  95  trust: 0-, 0q, 0n, 6m, 0f, 0u
gpg: depth: 2  valid:  75  signed:  26  trust: 75-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2023-04-21
==> Updating trust database...
gpg: next trustdb check due at 2023-04-21
(2/2) reinstalling manjaro-keyring                            [##################################] 100%
==> Appending keys from manjaro.gpg...
gpg: error reading key: No public key
gpg: error reading key: No public key
gpg: error reading key: No public key
gpg: error reading key: No public key
gpg: error reading key: No public key
gpg: error reading key: No public key
gpg: error reading key: No public key
gpg: error reading key: No public key
==> Locally signing trusted keys in keyring...
  -> Locally signed 18 keys.
==> Importing owner trust values...
gpg: inserting ownertrust of 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: inserting ownertrust of 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: inserting ownertrust of 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
==> Disabling revoked keys in keyring...
  -> Disabled 8 keys.
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:  22  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:  22  signed:  95  trust: 0-, 0q, 0n, 22m, 0f, 0u
gpg: depth: 2  valid:  75  signed:  26  trust: 75-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2023-04-21
:: Running post-transaction hooks...
(1/2) Reloading system manager configuration...
(2/2) Arming ConditionNeedsUpdate...
[INFO] Removing temporary directory: /tmp/tmp.pKPMb3wu93
removed '/tmp/tmp.pKPMb3wu93/manjaro-keyring-20221028-4-any.pkg.tar.xz'
removed '/tmp/tmp.pKPMb3wu93/archlinux-keyring-20221220-1-any.pkg.tar.zst'
removed '/tmp/tmp.pKPMb3wu93/pacman.conf'
removed directory '/tmp/tmp.pKPMb3wu93'
[INFO] Switching to a local mirror by GeoIP
::INFO Downloading mirrors from Manjaro
::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json
::INFO => Mirror status: https://repo.manjaro.org/status.json
::INFO User generated mirror list
::------------------------------------------------------------
::INFO Custom mirror file saved: /var/lib/pacman-mirrors/custom-mirrors.json
::INFO Using default mirror file
::INFO Querying mirrors - This may take some time
  0.344 Germany        : https://mirror.alpix.eu/manjaro/
  0.254 Germany        : https://mirrors.xtom.de/manjaro/
  0.311 Germany        : https://ftp.gwdg.de/pub/linux/manjaro/
  0.258 Germany        : https://mirror.23m.com/manjaro/
  0.306 Germany        : https://ftp.halifax.rwth-aachen.de/manjaro/
  0.234 Germany        : http://ftp.tu-chemnitz.de/pub/linux/manjaro/
  0.291 Germany        : https://mirror.netcologne.de/manjaro/
  0.260 Germany        : https://mirror.moson.org/manjaro/
  0.333 Germany        : https://mirror.tarnkappe.info/manjaro/
::INFO Writing mirror list
::Germany         : http://ftp.tu-chemnitz.de/pub/linux/manjaro/unstable
::Germany         : https://mirrors.xtom.de/manjaro/unstable
::Germany         : https://mirror.23m.com/manjaro/unstable
::Germany         : https://mirror.moson.org/manjaro/unstable
::Germany         : https://mirror.netcologne.de/manjaro/unstable
::Germany         : https://ftp.halifax.rwth-aachen.de/manjaro/unstable
::Germany         : https://ftp.gwdg.de/pub/linux/manjaro/unstable
::Germany         : https://mirror.tarnkappe.info/manjaro/unstable
::Germany         : https://mirror.alpix.eu/manjaro/unstable
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
::INFO Downloading mirrors from Manjaro
::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json
::INFO => Mirror status: https://repo.manjaro.org/status.json
::INFO Using custom mirror file
::INFO Querying mirrors - This may take some time
  0.323 Germany        : https://ftp.gwdg.de/pub/linux/manjaro/
  0.441 Germany        : https://mirror.alpix.eu/manjaro/
  0.232 Germany        : https://mirrors.xtom.de/manjaro/
  0.260 Germany        : https://mirror.moson.org/manjaro/
  0.214 Germany        : http://ftp.tu-chemnitz.de/pub/linux/manjaro/
::INFO Writing mirror list
::Germany         : http://ftp.tu-chemnitz.de/pub/linux/manjaro/unstable/$repo/$arch
::Germany         : https://mirrors.xtom.de/manjaro/unstable/$repo/$arch
::Germany         : https://mirror.moson.org/manjaro/unstable/$repo/$arch
::Germany         : https://ftp.gwdg.de/pub/linux/manjaro/unstable/$repo/$arch
::Germany         : https://mirror.alpix.eu/manjaro/unstable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
[INFO] Performing a full upgrade with pacman
Do you want to continue? [Yy/Nn] (Be aware that a full upgrade needs enough ram on a live session)y
Continue...
:: Synchronizing package databases...
 core                             164.3 KiB   625 KiB/s 00:00 [##################################] 100%
 extra                           1859.3 KiB  3.47 MiB/s 00:01 [##################################] 100%
 community                          7.5 MiB  4.74 MiB/s 00:02 [##################################] 100%
 multilib                         169.3 KiB  1078 KiB/s 00:00 [##################################] 100%
:: Starting full system upgrade...
 there is nothing to do
[INFO] Done. Note that you need to refresh the database for pamac also.

Maybe you blindly copy&paste and didn’t see that the “)” at the end was missing?

I added the installation of gnupg. Maybe your gnupg is just broken for what ever reason.

ok well… to skip the gpg verification for now try this:

sudo pacman-mirrors -c Global
sudo cp /etc/pacman.conf  /etc/pacman.conf.backup
sudo sed --in-place --regexp-extended 's/^(SigLevel).+$/\1 = Never/g' /etc/pacman.conf
sudo pacman -Syyuu

Now it should update from Manjaro’s global mirror without gpg verification. I would not recommend that, but using the global minimize the risk of installing manipulated packages. I would not do that on random mirrors.

marinheiro · 27 January 2023 19:08

So I tried that, but it ended a bit disastrously.

The update ran through around 390 packages of 410 then hung for 3 hours. top showed it wasn’t doing anything, it wasn’t short of memory, and there was plenty of disk space. Eventually the system stopped responding completely, I had no idea why, and all I could do was power off. On reboot the system stops with

/boot/vmlinuz-5.15-x86_64 not found

I boot from a live usb, and see that in fact /boot has no kernel in it. I would really like not to wreck my system totally if I can possibly recover it; so I’m wondering what is the best route from here: can I just copy vmlinuz-6.1-x86_64 and linux61-x86_64.kver from the usb boot directory into the disk /boot and edit grub during the reboot, or will that mess things up further?

mithrial · 27 January 2023 19:58

Reinstall your kernel in the chroot.

pacman -Syu linux515

andreas85 · 27 January 2023 20:30

In an similar situation i did copy (from an already updated pc) :

/boot/vmlinuz* (kernel)
/boot/initramfs*.fallback.img (modules)
(not /boot/initramfs.* ! because this may need to be different for every pc)
/boot/linux*.kver (kernelinfo)

and aditionally everything inside

/lib/modules/*MANJARO (modules+kernel)
/lib/modules/extramodules*.MANJARO (modules)

I had to select “fallback” in grub. And it did boot (After that i repeated the update successfully)

marinheiro · 28 January 2023 13:20

OK, I am progressively messing up the system at a deeper and deeper level. Put it down as a learning experience.

I tried andreas85’s straightforward ‘copy’ method but it didn’t work, so chrooted and ran pacman -Syu linux515 instead. This reinstalled the older kernel as expected. I then tried rebooting and found the system no longer recognizes the drive as being bootable. At this point I realised I had chrooted forgetting this is an EFI based system. I find my EFI data is on /dev/sda2; I believe I need to chroot with /dev/sda2 mounted on /boot/efi. But… the live usb has already mounted /dev/sda on /run/miso/bootmnt, and won’t let me unmount it since ‘target is busy’ . How do I fix this?