[HowTo] Work around gpg verification issue on left behind systems

[megavolt]

Difficulty: ★☆☆☆☆

This script is the result of people here in the forum having problems with Manjaro installations that haven’t been updated for more than 2 months or booting an old ISO plus installing it which is older than 2 months. I was a bit tired of explaining how to solve that issue and created a script, which does the job.

The Script

The Script has now 3 Methods:

Method 1 - Basic

1. Remove lock files of pamac and pacman
2. Refresh Package Database
3. Populate local keyrings to the GnuPG Database of pacman
4. Refresh GnuPG Database of pacman from the Internet [optional]
5. Performing a full upgrade with pacman [optional]

Method 2 - Moderate

1. Remove lock files of pamac and pacman
2. Remove Pacman’s GnuPG Database
3. Initilize Pacman’s GnuPG Database
4. Populate local keyrings to Pacman’s GnuPG Database
5. Refresh GnuPG Database of pacman from the Internet [optional]
6. Remove cached software packages [optional]
7. Performing a full upgrade with pacman [optional]

Method 3 - Aggressive

1. Remove lock files of pamac and pacman
2. Switch to global mirror (Manjaro’s CDN)
3. Remove Pacman’s GnuPG Database
4. Initilize Pacman’s GnuPG Database
5. Remove cached software packages
6. Create a temporary folder in /tmp
7. Copy /etc/pacman.conf to TMPDIR/pacman.conf and disable temporarily gpg verification.
8. Download the newest packages which contains the gpg keyrings in TMPDIR
9. Install temporarily downloaded keyring packages
10. Remove temporary directory TMPDIR
11. Switch to a local mirror by Geolocation
12. Refresh GnuPG Database of pacman from the Internet [optional]
13. Performing a full upgrade with pacman [optional]

The third one is just the original old script. Method 1 and 2 are inspired by [HowTo] Solve Keyring Related Issues in Manjaro

Old Script

#!/usr/bin/env bash

set -o errexit  #>Exit when a command fails (returns non-zero)
set -o pipefail #>Exit when a command within a pipeline fail (returns non-zero)
set -o nounset  #>Forbid to use unset

# Reset language to default
export LANG=C

# Switch to root account
RunasRoot () { sudo --login --user=root "$@"; }

# Colors for messages
PrintInfo () { printf '\e[32m[INFO]\e[0m \e[33m%s\e[0m\n' "$1"; }
PrintQuestion () { printf '\e[35m[QUESTION]\e[0m \e[33m%s\e[0m\n' "$1"; }

# Script starts here:
PrintInfo "Removing lock files of pamac and pacman"
RunasRoot rm --force --verbose "/var/tmp/pamac/dbs/db.lck"
RunasRoot rm --force --verbose "/var/lib/pacman/db.lck"

PrintInfo "Switching to global mirror (Manjaro's CDN)"
RunasRoot pacman-mirrors --country Global
RunasRoot pacman-mirrors --fasttrack 5

PrintInfo "Remove pacman's gnupg"
RunasRoot rm --recursive --force --verbose /etc/pacman.d/gnupg/*

PrintInfo "Re-initilize pacman's gnupg"
RunasRoot pacman-key --init

PrintInfo "Removing package cache"
RunasRoot pacman --sync --clean --clean --noconfirm
if [[ $(ls /var/cache/pacman/pkg/ | wc -l) != 0 ]]; then
   RunasRoot find /var/cache/pacman/pkg/ -type f -exec rm --force --verbose "{}" \;
fi

TMPDIR="$(mktemp -d)"
PrintInfo "Downloading the newest packages which contains the gpg keys in ${TMPDIR}"
RunasRoot cp "/etc/pacman.conf" "${TMPDIR}/pacman.conf"
RunasRoot sed --in-place --regexp-extended 's/^(SigLevel).+$/\1 = Never/g' "${TMPDIR}/pacman.conf"
RunasRoot pacman --sync --refresh --downloadonly --noconfirm --cachedir "${TMPDIR}" --config "${TMPDIR}/pacman.conf" archlinux-keyring manjaro-keyring gnupg --overwrite "*"

PrintInfo "Installing Keyring Packages"
RunasRoot pacman --upgrade --noconfirm --config "${TMPDIR}/pacman.conf" $(find ${TMPDIR} -type f -name "*.tar.*")

PrintInfo "Removing temporary directory: ${TMPDIR}"
RunasRoot find "${TMPDIR}" -type f -exec rm --recursive --force --verbose "{}" \;
RunasRoot rmdir --verbose "${TMPDIR}"

PrintInfo "Switching to a local mirror by GeoIP"
RunasRoot pacman-mirrors --geoip
RunasRoot pacman-mirrors --fasttrack 5

PrintInfo "Performing a full upgrade with pacman"

while true; do
   PrintQuestion "Do you want to continue? [Yy/Nn] (Be aware that a full upgrade needs enough ram on a live session)"
   read -p "> [Yy/Nn] " yn
   case $yn in
   [Yy]*) 
      RunasRoot pacman --sync --refresh --refresh --sysupgrade --sysupgrade --noconfirm
      PrintInfo "Done. Note that you need to refresh the database for pamac also."
      break
   ;;
   [Nn]*) 
      exit
   ;;
   *)
      PrintInfo "No valid answer. Continue." && continue ;;
   esac
done

#!/usr/bin/env bash

set -o errexit  #>Exit when a command fails (returns non-zero)
set -o pipefail #>Exit when a command within a pipeline fail (returns non-zero)
set -o nounset  #>Forbid to use unset

# Reset language to defaultx
export LANG=C

# Switch to root account
RunasRoot () { sudo --login --user=root "$@"; }

# Colors for messages
PrintInfo () { printf '\e[32m[INFO]\e[0m \e[33m%s\e[0m\n' "$1"; }
PrintError () { printf '\e[31m[ERROR]\e[0m \e[33m%s\e[0m\n' "$1"; }
PrintQuestion () { printf '\e[35m[QUESTION]\e[0m \e[33m%s\e[0m\n' "$1"; }

RmLockFiles () 
{
   PrintInfo "Remove lock files of pamac and pacman"
   RunasRoot find /var/tmp/pamac/ -type f -iname "*db.lck" -exec rm --force --verbose "{}" \;
   RunasRoot find /var/lib/pacman/ -type f -iname "*db.lck" -exec rm --force --verbose "{}" \;
}

AskForUpgrade ()
{
   PrintInfo "Performing a full upgrade with pacman"
   while true; do
      PrintQuestion "Do you want to continue? [Yy/Nn] (Be aware that a full upgrade needs enough ram on a live session)"
      read -p "> [Yy/Nn] " yn
      case $yn in
      [Yy]*) 
         RunasRoot pacman --sync --refresh --refresh --sysupgrade --sysupgrade --noconfirm
         PrintInfo "Done. Note that you need to refresh the database for pamac also."
         break
      ;;
      [Nn]*) 
         break
      ;;
      *)
         PrintError "No valid answer. Continue." && continue ;;
      esac
   done
}

AskForRefreshKeys ()
{
   PrintInfo "Refresh GnuPG Database of pacman from the Internet"
   while true; do
      PrintQuestion "Do you want to continue? [Yy/Nn] (Note that this can take a while.)"
      read -p "> [Yy/Nn] " yn
      case $yn in
      [Yy]*) 
         RunasRoot pacman-key --refresh-keys && break
      ;;
      [Nn]*) 
         break
      ;;
      *)
         PrintError "No valid answer. Continue." && continue ;;
      esac
   done
}

AskForRemoveCache ()
{
   PrintInfo "Remove cached software packages [optional]"
   while true; do
      PrintQuestion "Delete them? [Yy/Nn]"
      read -p "[Yy/Nn] > " yn
      case $yn in
      [Yy]*)
         PrintInfo "Removing package cache"
         RunasRoot pacman --sync --clean --clean --noconfirm
         if [[ $(ls /var/cache/pacman/pkg/ | wc -l) != 0 ]]; then
            RunasRoot find /var/cache/pacman/pkg/ -type f -exec rm --force --verbose "{}" \;
         fi
      ;;
      [Nn]*) 
         break
      ;;
      *)
         PrintError "No valid answer." && continue ;;
      esac
   done
}

BasicMethod ()
{
   RmLockFiles

   PrintInfo "Refresh Package Database"
   RunasRoot pacman --sync --refresh --refresh

   PrintInfo "Populate local keyrings to the GnuPG Database of pacman"
   RunasRoot pacman-key --populate archlinux manjaro

   AskForRefreshKeys

   AskForUpgrade
}

ModerateMethod ()
{
   RmLockFiles

   PrintInfo "Remove Pacman's GnuPG Database"
   RunasRoot find /etc/pacman.d/gnupg/ -exec rm --recursive --force --verbose "{}" \;

   PrintInfo "Initilize Pacman's GnuPG Database"
   RunasRoot pacman-key --init

   PrintInfo "Populate local keyrings to Pacman's GnuPG Database"
   RunasRoot pacman-key --populate archlinux manjaro

   AskForRefreshKeys

   AskForRemoveCache

   AskForUpgrade
}

AggressiveMethod ()
{
   RmLockFiles
   
   PrintInfo "Switch to global mirror (Manjaro's CDN)"
   RunasRoot pacman-mirrors --country Global
   RunasRoot pacman-mirrors --fasttrack 5

   PrintInfo "Remove Pacman's GnuPG Database"
   RunasRoot find /etc/pacman.d/gnupg/ -exec rm --recursive --force --verbose "{}" \;

   PrintInfo "Initilize Pacman's GnuPG Database"
   RunasRoot pacman-key --init

   PrintInfo "Removing package cache"
   RunasRoot pacman --sync --clean --clean --noconfirm
   if [[ $(ls /var/cache/pacman/pkg/ | wc -l) != 0 ]]; then
      RunasRoot find /var/cache/pacman/pkg/ -type f -exec rm --force --verbose "{}" \;
   fi

   PrintInfo "Create a temporary folder in /tmp"
   TMPDIR="$(mktemp -d)"
   PrintInfo "${TMPDIR}"

   PrintInfo "Copy /etc/pacman.conf to ${TMPDIR}/pacman.conf and disable temporarily gpg verification."
   RunasRoot cp "/etc/pacman.conf" "${TMPDIR}/pacman.conf"
   RunasRoot sed --in-place --regexp-extended 's/^(SigLevel).+$/\1 = Never/g' "${TMPDIR}/pacman.conf"

   PrintInfo "Download the newest packages which contains the gpg keyrings in ${TMPDIR}"
   RunasRoot pacman --sync --refresh --downloadonly --noconfirm --cachedir "${TMPDIR}" --config "${TMPDIR}/pacman.conf" archlinux-keyring manjaro-keyring gnupg --overwrite "*"

   PrintInfo "Install temporarily downloaded keyring packages"
   RunasRoot pacman --upgrade --noconfirm --config "${TMPDIR}/pacman.conf" $(find ${TMPDIR} -type f -name "*.tar.*")

   PrintInfo "Remove temporary directory: ${TMPDIR}"
   RunasRoot find "${TMPDIR}" -type f -exec rm --recursive --force --verbose "{}" \;
   RunasRoot rmdir --verbose "${TMPDIR}"

   PrintInfo "Switch to a local mirror by Geolocation"
   RunasRoot pacman-mirrors --geoip
   RunasRoot pacman-mirrors --fasttrack 5

   AskForRefreshKeys

   AskForUpgrade

}

HelpPage ()
{
   printf "\n\t%s\n" "Method 1 - Basic"
   printf "\t%s\n" "1. Remove lock files of pamac and pacman"
   printf "\t%s\n" "2. Refresh Package Database"
   printf "\t%s\n" "3. Populate local keyrings to the GnuPG Database of pacman"
   printf "\t%s\n" "4. Refresh GnuPG Database of pacman from the Internet [optional]"
   printf "\t%s\n" "5. Performing a full upgrade with pacman [optional]"

   printf "\n\t%s\n" "Method 2 - Moderate"
   printf "\t%s\n" "1. Remove lock files of pamac and pacman"
   printf "\t%s\n" "2. Remove Pacman's GnuPG Database"
   printf "\t%s\n" "3. Initilize Pacman's GnuPG Database"
   printf "\t%s\n" "4. Populate local keyrings to Pacman's GnuPG Database"
   printf "\t%s\n" "5. Refresh GnuPG Database of pacman from the Internet [optional]"
   printf "\t%s\n" "6. Remove cached software packages [optional]"
   printf "\t%s\n" "7. Performing a full upgrade with pacman [optional]"

   printf "\n\t%s\n" "Method 3 - Aggressive"
   printf "\t%s\n" "1. Remove lock files of pamac and pacman"
   printf "\t%s\n" "2. Switch to global mirror (Manjaro's CDN)"
   printf "\t%s\n" "3. Remove Pacman's GnuPG Database"
   printf "\t%s\n" "4. Initilize Pacman's GnuPG Database"
   printf "\t%s\n" "5. Remove cached software packages"
   printf "\t%s\n" "6. Create a temporary folder in /tmp"
   printf "\t%s\n" "7. Copy /etc/pacman.conf to TMPDIR/pacman.conf and disable temporarily gpg verification."
   printf "\t%s\n" "8. Download the newest packages which contains the gpg keyrings in TMPDIR"
   printf "\t%s\n" "9. Install temporarily downloaded keyring packages"
   printf "\t%s\n" "10. Remove temporary directory TMPDIR"
   printf "\t%s\n" "11. Switch to a local mirror by Geolocation"
   printf "\t%s\n" "12. Refresh GnuPG Database of pacman from the Internet [optional]"
   printf "\t%s\n" "13. Performing a full upgrade with pacman [optional]"

   printf "\n\t%s\n\n" "Note: That tool is not designed to fix gpg issues with your local user GnuPG Database, which is commonly used for AUR Packages as an example."
}

HelpUsage () { printf '\e[32m%s\e[0m\n' "$0 [--usage|--help|--basic|--moderate|--aggressive|--ask] (default: [--basic])"; }

Menu ()
{
   clear
   while true; do
      printf '\e[33m%s\e[0m\n' "1. Basic"
      printf '\e[33m%s\e[0m\n' "2. Moderate"
      printf '\e[33m%s\e[0m\n' "3. Aggressive"
      printf '\e[33m%s\e[0m\n' "4. Explanations of the methods"
      printf '\n\e[33m%s\e[0m\n\n' "Type [q] to quit."
      PrintQuestion "Choose a Method by typing the number and press ENTER on you keyboard."
      read -p "[1,2,3,4,q]>  " choice
      case $choice in
         1) 
            BasicMethod && exit
         ;;
         2) 
            ModerateMethod && exit
         ;;
         3) 
            AggressiveMethod && exit
         ;;
         4)
            HelpPage && continue
         ;;
         q)
            exit
         ;;
         *)
            clear && PrintError "No valid answer. Continue." && continue ;;
      esac
   done
}

[[ -z $@ ]] && BasicMethod && exit
case $1 in 
   --basic)
      BasicMethod && exit
   ;;
   --moderate)
      ModerateMethod && exit
   ;;
   --aggressive)
      AggressiveMethod && exit
   ;;
   --help)
      HelpUsage && HelpPage && exit
   ;;
   --usage)
      HelpUsage && exit
   ;;
   --ask)
      Menu && exit
   ;;
   *)
      PrintError "\"$1\" is not a valid parameter. See \"$0 --usage\""
   ;;
esac

That script is mirrored manually to: megavolt/random-scripts: Various scripts for any usages. - NotABug.org: Free code hosting. Therefore it is possibe to run it like that:

export URL="https://notabug.org/megavolt/random-scripts/raw/master/fix-gpg-pacman.sh"
bash <(curl -s "$URL") --help
bash <(curl -s "$URL") --usage
bash <(curl -s "$URL") --ask
bash <(curl -s "$URL") --basic
bash <(curl -s "$URL") --moderate
bash <(curl -s "$URL") --aggressive

Hope the new redesign could fit more your needs. Cheers

This is a wiki article and you are invited to make the script better or add more explanations or anything else.

[mithrial]

I’ve made it more robust and applied all shellcheck suggestions.

I feel a bit worried about ignoring the certificate for installing the keys package. Security-wise, this would be among the most important packages where validation should be top priority.

Maybe hardcode the validation? But I don’t know how often the signing key for these two packages change.

[megavolt]

@mithrial

I’ve made it more robust and applied all shellcheck suggestions.

I hope you tested it yourself before applying such changes. Did you?

@mithrial

I feel a bit worried about ignoring the certificate for installing the keys package. Security-wise, this would be among the most important packages where validation should be top priority.

Well that are keys for verification. How would verify keys which are there for verification?

@mithrial

Maybe hardcode the validation? But I don’t know how often the signing key for these two packages change.

No idea what you mean by that As I know gpg keys can change quite often. The problem though is that the if you didn’t update and skip one keyring update, it will not be able to verify the keyring package and result in that error.

[mithrial]

Yes, I’ve tested it.

The approach here downloads the keyring and installs it without verifying if it’s the original package.
I believe this defeats the purpose of the keys.

[alven]

@mithrial

I believe this defeats the purpose of the keys.

I agreed, but you can improve it or suggest your idea how it could be fixed properly (with check of origin).

[megavolt]

@mithrial

Yes, I’ve tested it.

@mithrial

The approach here downloads the keyring and installs it without verifying if it’s the original package.
I believe this defeats the purpose of the keys.

I have not deep knowledge when it comes to gpg, but in my view it makes no difference if you get the keyring from an unverified package or retrieve it from a server. The verification of the package is just an extra security measure, but the missing of the package verification of keyrings would not reduce security in any way.

[mithrial]

The signature is there to verify that it has not been tampered with on the server, client or during transfer.
The public keys make sure of this.

However, if I control the keyring package (because you disabled the verification) I could add my own keys and sign any package (e.g. firefox or keepassxc) and your system will happily install my package.

[megavolt]

@mithrial

However, if I control the keyring package (because you disabled the verification) I could add my own keys and sign any package (e.g. firefox or keepassxc) and your system will happily install my package.

okay understand that, but…

1. In the script we switch explicitly to the global mirror which is not a some mirror out there which can be manipulated maybe “easier”. Here I would give Manjaro’s Maintainers a bit trust.
2. Package verification can be disabled really easy if you have root access.
3. On the script verification of packages are only temporary disabled. It only affects the keyring packages. So it is minimal invasive.

So to manipulate package, you must have access to Manjaro’s own server… It is a little security concern, but in my view it is better doing it that way as reinstalling the whole system.

If you have a better idea, please say it, but I don’t see any fast, reliable and less invasive solution as that script…

[mithrial]

Usually the key exchange should be done with another factor, e.g. hardcoding the keyring’s key into the script.

Another way might be populate the keyrings and then refresh the keys? New keys might be missing but the maintainer of the keyring won’t have new keys very often, right?

If you trust the global mirror, why do you have package signing anyway? Where does your trust stop?

[megavolt]

@mithrial

Usually the key exchange should be done with another factor, e.g. hardcoding the keyring’s key into the script.

Maybe, but we are talking about pacman where gpg verification is optional and not a requirement. It is by design a security measure on top of it although enabled by default (opt-out).

@mithrial

Another way might be populate the keyrings and then refresh the keys? New keys might be missing but the maintainer of the keyring won’t have new keys very often, right?

Well installing the package does population already, but don’t connect to a gpg server and refresh the keys. Doing pacman-key --refresh needs a lot more time by the way.

@mithrial

If you trust the global mirror, why do you have package signing anyway? Where does your trust stop?

In my view, Manjaro’s mirror is the origin. There I have more trust and I would consider disabling gpg verification there if necessary. If it is a mirror of Manjaro’s server, then I would really care to check if the package is really one from Manjaro, so that it wasn’t manipulated somewhere in between. Therefore I strongly consider to use gpg verification.

---

About the script:
Well… maybe add a line to switch to another local mirror before doing a full upgrade?

pacman-mirrors --interactive --geoip

Anyway… added it:

@megavolt

echo -e "${HL}[INFO] Switching to a local mirror by GeoIP ${ED}"
"${asSudo[@]}" pacman-mirrors --geoip
"${asSudo[@]}" pacman-mirrors --fasttrack 5

echo -e "${HL}[INFO] Performing a full upgrade with pamac without AUR${ED}"
"${asSudo[@]}" pamac upgrade --force-refresh --enable-downgrade --no-aur