[UNSTABLE] pacman keyring issues - keyring reset after reboot

We have some issues with pacman and the pacman keyring.

This topic is to compile the experience you have.

The keyring issue is triggered with the rebuild of pacman only on unstable branch

package release 6.0.2-18

We suspect is has something to do with dirmngr and keyboxd sockets as these sockets is not created with previous versions.

It is likely upstream has applied changes we are not fully aware of the implications the changes introduce.

If you are using unstable branch - please add a comment with output from

09:04:55 β—‹ [fh@tiger] ~
 $ tree /etc/pacman.d
/etc/pacman.d
β”œβ”€β”€ gnupg
β”‚   β”œβ”€β”€ pubring.kbx
β”‚   β”œβ”€β”€ S.dirmngr
β”‚   β”œβ”€β”€ S.gpg-agent
β”‚   β”œβ”€β”€ S.gpg-agent.browser
β”‚   β”œβ”€β”€ S.gpg-agent.extra
β”‚   β”œβ”€β”€ S.gpg-agent.ssh
β”‚   β”œβ”€β”€ S.keyboxd
β”‚   └── trustdb.gpg
β”œβ”€β”€ mesa-nonfree.pre.repo.conf
└── mirrorlist

2 directories, 10 files

sudo pacman -Syu
[...]

(23/23) checking keys in keyring                                   [------------------------------------] 100%
warning: Public keyring not found; have you run 'pacman-key --init'?
downloading required keys...
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: keyring is not writable
error: required key missing from keyring
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.

There is no current solution - below topic is the first occurence on record …

For notes and extras

Thanks to @zbe for the clues to what appears to be the cause of this.

When one has manjaro-tools installed - it installs a mount unit which mounts /etc/pacman.d/gnupg on tmpfs.

Why this hasn’t been an issue before we have still to figure out

For those curios about the technical aspect - here goes

Upstream has added keyboxd@etc-pacman.d-gnupg.socket and dirmngr@etc-pacman.d-gnupg.socket sockets.

Manjaro ISO tools uses a mount unit and a service to initialize keyring on live media using a tmpfs mount targeting /etc/pacman.d/gnupg. This mount unit was triggered by the sockets pacman uses.

The mount unit etc-pacman.d-gnupg.mount was activated by when pacman accessed the sockets - thus making the original content inaccessible - therefore pacman issued an error message - keyring is not initialized.

Only those on unstable branch AND having the manjaro-tools installed is affected by this.

1 Like

Given my experience, for now a one-liner might be

sudo pacman-key --init && sudo pacman-key --populate $(pacman -Qsq keyring | awk -F- '{ print $1 }')

@cscs well that is how you init the keyring. however on reboot the folder content gets deleted and the user has to re-init every system boot. This should not happen to begin with …

Yes, as I noted in the other thread, I know it is required on every reboot.
Its simply a reliable and simple way to get in working order upon reboot.
As that was not yet listed in this thread.

And no, I dont yet know why, as I also stated in the other thread, I havent had time to investigate.

This does not happen to every installation: I have 3 machines running current unstable with pacman 6.0.2-18 and none are showing the described (mis-)behaviour.

tree output
# tree /etc/pacman.d/
/etc/pacman.d/
β”œβ”€β”€ gnupg
β”‚   β”œβ”€β”€ gpg-agent.conf
β”‚   β”œβ”€β”€ gpg.conf
β”‚   β”œβ”€β”€ openpgp-revocs.d
β”‚   β”‚   └── 1695FE4A82AE804B30C9D9D70AE38B4E19820166.rev
β”‚   β”œβ”€β”€ private-keys-v1.d
β”‚   β”‚   └── A5392200DCDA20FD237A31B46B2D5F34A09CECAD.key
β”‚   β”œβ”€β”€ pubring.gpg
β”‚   β”œβ”€β”€ pubring.gpg~
β”‚   β”œβ”€β”€ S.dirmngr
β”‚   β”œβ”€β”€ secring.gpg
β”‚   β”œβ”€β”€ S.gpg-agent
β”‚   β”œβ”€β”€ S.gpg-agent.browser
β”‚   β”œβ”€β”€ S.gpg-agent.extra
β”‚   β”œβ”€β”€ S.gpg-agent.ssh
β”‚   β”œβ”€β”€ S.keyboxd
β”‚   β”œβ”€β”€ tofu.db
β”‚   └── trustdb.gpg
β”œβ”€β”€ hooks
β”‚   └── cache-cleanup.hook
└── mirrorlist

5 directories, 17 files
1 Like

Did you try to reboot one of them?

All of them shutdown/boot up at least daily, so they’ve been rebooted at least 3 times after pacman 6.0.2-18 was installed on 2024-02-10.
One of them I’ve rebooted multiple times just before posting.

Yes it does.

1 Like

And installing new packages works fine?
(at least in my cases regular sync/upgrade appeared normal)

For further clarity heres after a reboot:

$ sudo pacman -Syu
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 multilib is up to date
:: Starting full system upgrade...
 there is nothing to do
$ sudo pacman -Syu cowsay
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 multilib is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (1) cowsay-3.04-4

Total Installed Size:  0.03 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                                                                                 [##################################################################] 100%
warning: Public keyring not found; have you run 'pacman-key --init'?
downloading required keys...
error: keyring is not writable
error: required key missing from keyring
error: failed to commit transaction (could not find or read file)
Errors occurred, no packages were upgraded.
$ sudo pacman-key --init && sudo pacman-key --populate $(pacman -Qsq keyring | awk -F- '{ print $1 }')
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
==> Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key...
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created

[...]

==> Disabling revoked keys in keyring...
  -> Disabled 55 keys.
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:  24  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:  24  signed:  98  trust: 0-, 0q, 0n, 24m, 0f, 0u
gpg: Note: third-party key signatures using the SHA1 algorithm are rejected
gpg: (use option "--allow-weak-key-signatures" to override)
gpg: depth: 2  valid:  74  signed:  22  trust: 74-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2024-04-10
$ sudo pacman -Syu cowsay
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 multilib is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (1) cowsay-3.04-4

Total Installed Size:  0.03 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                                                                                 [##################################################################] 100%
(1/1) checking package integrity                                                                               [##################################################################] 100%
(1/1) loading package files                                                                                    [##################################################################] 100%
(1/1) checking for file conflicts                                                                              [##################################################################] 100%
(1/1) checking available disk space                                                                            [##################################################################] 100%
:: Processing package changes...
(1/1) installing cowsay                                                                                        [##################################################################] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...

Quite strange - that is how mine look before I reboot - after reboot it is empty - except for the sockets.

What is different between a functioning system and a failing system - mine is failing - quite obviously

# tree /etc/pacman.d
/etc/pacman.d
β”œβ”€β”€ gnupg
β”‚   β”œβ”€β”€ S.gpg-agent
β”‚   β”œβ”€β”€ S.gpg-agent.browser
β”‚   β”œβ”€β”€ S.gpg-agent.extra
β”‚   └── S.gpg-agent.ssh
β”œβ”€β”€ mesa-nonfree.pre.repo.conf
└── mirrorlist

After --init

# tree /etc/pacman.d
/etc/pacman.d
β”œβ”€β”€ gnupg
β”‚   β”œβ”€β”€ gpg-agent.conf
β”‚   β”œβ”€β”€ gpg.conf
β”‚   β”œβ”€β”€ openpgp-revocs.d
β”‚   β”‚   └── F846EFC0CF5A613EDE1131C725E50E17F6446AA9.rev
β”‚   β”œβ”€β”€ private-keys-v1.d
β”‚   β”‚   └── 93467442400061BC1053320841B102CB232E0A8E.key
β”‚   β”œβ”€β”€ pubring.gpg
β”‚   β”œβ”€β”€ pubring.gpg~
β”‚   β”œβ”€β”€ secring.gpg
β”‚   β”œβ”€β”€ S.gpg-agent
β”‚   β”œβ”€β”€ S.gpg-agent.browser
β”‚   β”œβ”€β”€ S.gpg-agent.extra
β”‚   β”œβ”€β”€ S.gpg-agent.ssh
β”‚   └── trustdb.gpg
β”œβ”€β”€ mesa-nonfree.pre.repo.conf
└── mirrorlist

reboot

 $ tree /etc/pacman.d
/etc/pacman.d
β”œβ”€β”€ gnupg
β”‚   β”œβ”€β”€ S.gpg-agent
β”‚   β”œβ”€β”€ S.gpg-agent.browser
β”‚   β”œβ”€β”€ S.gpg-agent.extra
β”‚   └── S.gpg-agent.ssh
β”œβ”€β”€ mesa-nonfree.pre.repo.conf
└── mirrorlist
$ tree /etc/pacman.d
/etc/pacman.d
β”œβ”€β”€ gnupg
β”‚   β”œβ”€β”€ S.dirmngr
β”‚   β”œβ”€β”€ S.gpg-agent
β”‚   β”œβ”€β”€ S.gpg-agent.browser
β”‚   β”œβ”€β”€ S.gpg-agent.extra
β”‚   β”œβ”€β”€ S.gpg-agent.ssh
β”‚   β”œβ”€β”€ S.keyboxd
β”‚   β”œβ”€β”€ crls.d
β”‚   β”‚   └── DIR.txt
β”‚   β”œβ”€β”€ gpg-agent.conf
β”‚   β”œβ”€β”€ gpg.conf
β”‚   β”œβ”€β”€ openpgp-revocs.d
β”‚   β”‚   └── B415465A1FEEEEC2BE0C6328F1ECC92CA9DCB112.rev
β”‚   β”œβ”€β”€ private-keys-v1.d
β”‚   β”‚   └── 9D1EEDF50D3262DC9388B04285E5F66CB1B4E01E.key
β”‚   β”œβ”€β”€ pubring.gpg
β”‚   β”œβ”€β”€ pubring.gpg~
β”‚   β”œβ”€β”€ secring.gpg
β”‚   β”œβ”€β”€ tofu.db
β”‚   └── trustdb.gpg
β”œβ”€β”€ manjaro-sway.repo.conf
└── mirrorlist

5 directories, 18 files

I don’t have the mentioned problem…

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
dev             1.4G     0  1.4G   0% /dev
run             1.5G  1.1M  1.5G   1% /run
/dev/vda2        30G   26G  3.4G  89% /
tmpfs           1.5G     0  1.5G   0% /dev/shm
tmpfs           1.5G  4.0K  1.5G   1% /tmp
tmpfs           1.5G  2.8M  1.5G   1% /etc/pacman.d/gnupg
/dev/vda3        99M  292K   99M   1% /boot/efi
tmpfs           289M   68K  289M   1% /run/user/967
tmpfs           289M   72K  289M   1% /run/user/1000
$ pacman -Qo /usr/lib/systemd/system/etc-pacman.d-gnupg.mount
/usr/lib/systemd/system/etc-pacman.d-gnupg.mount is owned by manjaro-tools-base-git r3035.3a5c25c-1

I was thinking something like that - the gnupg folder being

The only thing actually working is moving the folder /etc/pacman.d/gnupg to /var/lib/pacman and modify pacman.conf accordingly.

Otherwise my keyring is removed on reboot.

That is a likely explanation why it is only a few which is touched by this

$ pacman -Qo /usr/lib/systemd/system/etc-pacman.d-gnupg.mount
/usr/lib/systemd/system/etc-pacman.d-gnupg.mount is owned by manjaro-tools-base-git r3035.3a5c25c-1

Sure looks like it’s responsible.
The introducing commit by @Yochanan is 6 months old by now, but maybe it wasn’t present in a released version?
Or that pacman-init service did not run as expected?

I guess this isn’t working (well, it’s disabled in my VM):

$ cat /usr/lib/systemd/system/pacman-init.service

[Unit]
Description=Initialize Pacman keyring
Wants=haveged.service
After=haveged.service
Requires=etc-pacman.d-gnupg.mount
After=etc-pacman.d-gnupg.mount

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/pacman-key --init
ExecStart=/usr/bin/pacman-key --populate archlinux manjaro

[Install]
WantedBy=multi-user.target
$ systemctl status haveged --no-pager

β—‹ haveged.service - Entropy Daemon based on the HAVEGE algorithm
Loaded: loaded (/usr/lib/systemd/system/haveged.service; enabled; preset: disabled)
Active: inactive (dead)
Condition: start condition unmet at Wed 2024-02-14 13:15:37 CET; 26min ago
Docs: man:haveged(8)
http://www.issihosts.com/haveged/

Feb 14 13:15:37 manjaro systemd[1]: Entropy Daemon based on the HAVEGE algorithm was skipped because of an unmet c…on=<5.6).
Feb 14 13:15:37 manjaro systemd[1]: Entropy Daemon based on the HAVEGE algorithm was skipped because of an unmet c…on=<5.6).
Hint: Some lines were ellipsized, use -l to show in full.

If nothing else:

sudo pacman -Rns manjaro-tools-{base,pkg,yaml,iso}-git

:stuck_out_tongue:

Another solution might be editing the service and enabling it:

$ sudo systemctl edit pacman-init

### Editing /etc/systemd/system/pacman-init.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file

[Unit]
Wants=
After=

### Edits below this comment will be discarded
[...]
2 Likes

I think this is an oversight in the tools package as this is only intended to be used for the live ISO.

manjaro-tools-base-git-r3035.3a5c25c-2 and manjaro-live-systemd-20240214-1 should fix this

2 Likes

This topic was automatically closed 36 hours after the last reply. New replies are no longer allowed.