[HowTo] Python upgrade: reinstall AUR and pip packages

Contributions Tutorials
1

Re-install pip and AUR Python packages
around the annual upgrade of system Python

Last major edit: 2023-06-24
Minor edits:
2024-05-31: in B4, added code that removes version numbers from pip freeze file.
2024-06-01: in B3, order of args for pamac clean.

As with any upgrade, it’s wise to take a system-snapshot before you start.
We cannot be sure there are no errors in this guide, so please use it with care.
As always, you are the one who is responsible for whatever happens to your system. :slight_smile:

The Python interpreter gets a minor-version upgrade every year.
All Python packages from system repositories get upgraded at the same time.

Python packages that are NOT from system repositories stop working.
This guide tells you how to re-install them so that they work again.

If you have only a few such packages, and you can
simply remember them and install them afresh,
you do not need this guide. :slight_smile:

This guide assumes that you already know enough
about the layout of Python installation directories,
and commands in bash, pacman, AUR helpers and pip.
If you need to know more, please read this background information.

This guide creates files containing lists of packages,
so that even if you have very many packages,
you won’t lose track of them.

This guide uses ‘OLD’ and ‘NEW’ as placeholders for the
minor version numbers of the outgoing and incoming system Python.
In mid-2023, OLD is 10 and NEW is 11.

2023-06-21 : New info about pip

PEP 668 – Marking Python base environments as “externally managed”

Details are in the background information, part B1d .

(A) Before the system update

(A1) If you have installed the latest Python from the AUR

You MUST uninstall it before running the system-Python upgrade.
Please consider the directions here: Remove latest AUR Python.

(A2) Make sure you’re running the right pip

  • pip -V

It will say something like
“pip N.N.N from /usr/lib/python3.N/site-packages/pip (python 3.N)”
where each N is a number.

The N after python3 needs to be of the current OLD version
not of the NEW version that you might have installed from AUR.

If it’s the wrong version, run

  • rm -vi ~/.local/bin/pip ~/.local/bin/pip3
  • sudo pacman -S python-pip
  • pip -V

and make sure that ‘pip -V’ reports the correct version now.
If it’s not correct, in the commands below, substitute
pip3.${OLD} or python3.${OLD} -m pip for the pip command.

(A3) List all AUR and PIP packages.

(adjust OLD= in future; ‘10’ is only for mid-2023)

    OLD='10'
    pip freeze --user > OLD_PIP_pkgs.txt
    pacman -Qmq $(pacman -Qoq /usr/lib/python3.${OLD}/site-packages) > OLD_AUR_pkgs.txt

Make safe copies of these list files
as you will NOT be able to re-create them after uninstalling the packages.

(A4) Uninstall old packages now

Why uninstall old packages at all?
If you have a lot of them, and might have other files in site-packages,
getting rid of packages helps you to see what else might be there.
(I found some notes and scripts that I had put there and forgotten about.)

Why now?
After the upgrade there will not be an old pip to uninstall them with,
nor will they be removed by ‘pip --upgrade’ under the new Python.
With AUR, some helpers remove old packages, but some don’t.

Whether pacman can safely uninstall them later, I’m not sure;
by default, pacman would work on the new python site-packages,
but the old location might be written in the info of each package,
enabling pacman to uninstall from the old location.

To remove packages now:

  • pip uninstall -r OLD_PIP_pkgs.txt
  • sudo pacman -Rsu $(< OLD_AUR_pkgs.txt)

System update

Run the system update in your usual way (with the usual safeguards),
but do not update AUR packages yet.

REBOOT

Essential. Don’t forget.

(B) After the system update

(B1) Prepare

  1. Set these constants
    (‘10’ and ‘11’ in mid-2023; adjust in future)

    (Where I write a block of code like this, it’s OK to copy
    the whole block and paste it into the terminal all at once.)

    OLD='10'; NEW='11'
sp_SYS_OLD="/usr/lib/python3.${OLD}/site-packages"
sp_HOME_OLD="$HOME/.local/lib/python3.${OLD}/site-packages"
sp_SYS_NEW="/usr/lib/python3.${NEW}/site-packages"
sp_HOME_NEW="$HOME/.local/lib/python3.${NEW}/site-packages"

  2. Make sure you’re running the right pip.

    • pip -V

    If, where it says “Python 3.N”, N is not the NEW version:

    1. rm -vi ~/.local/bin/pip ~/.local/bin/pip3
    2. sudo pacman -S python-pip
    3. pip -V

  3. If you use pikaur, install it again, as it’s a python program:

    1. sudo pacman -S --needed base-devel git
    2. git clone https://aur.archlinux.org/pikaur.git
    3. cd pikaur
    4. makepkg -fsri

(B2)

Does the old system ‘site-packages’ dir still exist?

  • [ -d "${sp_SYS_OLD}" ] && echo "${sp_SYS_OLD} DOES exist" || echo "${sp_SYS_OLD} does NOT exist"

If it says the folder does NOT exist :

  • CAUTION: Do not mess with ‘pacman -Q’ commands in this section B
    as they will list all your NEW packages.

  • There is nothing to do here – skip to part C.

List some packages:

  1. System packages that failed during the system update.
    This list should be empty, and probably will be.

    • pacman -Qnq $(pacman -Qoq ${sp_SYS_OLD}) > OLD_SYS_leftovers.txt

  2. If you did NOT get lists BEFORE the system update
    (e.g. because you updated before you read this guide)

    • pacman -Qmq $(pacman -Qoq ${sp_SYS_OLD}) > OLD_AUR_pkgs_2.txt

    • pip freeze --path ${sp_HOME_OLD} > OLD_PIP_pkgs_2.txt

    If you do NOT already have ‘OLD_AUR_pkgs.txt’ and ‘OLD_PIP_pkgs.txt’,
    remove the ‘_2’ from the names of the files you just made.
    (They were created with ‘_2’ to avoid accidentally overwriting
    lists that were made before the update and would not be re-creatable now.)

  3. If you had python311 from the AUR:

    With the packages listed in your ‘AUR311_AUR_pkgs.txt’:

    • if you left them installed, remove their names from ‘OLD_AUR_pkgs.txt’;

    • if you uninstalled them and you want them, add their names to ‘OLD_AUR_pkgs.txt’
      (but don’t duplicate any name that’s already in OLD_AUR_pkgs).

    The file ‘AUR311_SYS_pkgs.txt’ is not for using here;
    use it later as a reminder of what you might want to install from system repos.

(B3) Rebuild AUR packages

With the following commands, if a package fails to build or install:

  1. put its name and details in ‘AUR_pkgs_failed.txt’;

  2. in ‘OLD_AUR_pkgs.txt’, remove all that have been done,
    then run the command again.
    (simply commenting them out causes an error, with pikaur at least)

The commands

Each AUR helper has it’s own cache and records,
so use ONLY the ONE helper that you normally use.

Use the code under only ONE of these 4 main bullet points.

  • pikaur -S --rebuild $(< OLD_AUR_pkgs.txt)
    (I used this myself; it rebuilt packages and removed them from sp_SYS_OLD.)

  • paru -S --rebuild yes $(< OLD_AUR_pkgs.txt)
    (A manjaro moderator has said that this works, and removes old packages.)

  • yay currently needs its cache cleaned first, to force rebuilding
    and it does not remove old packages from sp_SYS_OLD.
    see (Rebuild option is ignored · Issue #2153 · Jguer/yay · GitHub)
    --rebuild was fixed recently in git, but is not in release 12.0.5
    --answerclean --rebuildall --rebuildtree don’t work at present.

    1. yay -Sc
    2. yay -S --rebuild $(< OLD_AUR_pkgs.txt)

  • pamac needs its cache cleaned first, to force rebuilding.

    1. Did you UNinstall AUR packages BEFORE the update?

      If yes, you can clean only uninstalled packages from pamac’s cache:

      • pamac clean -v --build-files --uninstalled --keep 0

      If no, you must clean all packages from pamac’s cache:

      • pamac clean -v --build-files --keep 0

    2. Build and install the listed AUR packages:

    • pamac build $(< OLD_AUR_pkgs.txt)

(B4) Re-install pip packages

Remove the version numbers:

	_file1=OLD_PIP_pkgs.txt
	_file2=pip_pkgs_to_upgrade.txt
	cat ${_file1} | awk -F== '{ print $1 }' > ${_file2}

In ‘pip_pkgs_to_upgrade.txt’,
comment out any that you don’t want now,
then run:

	pip install --user -r pip_pkgs_to_upgrade.txt --break-system-packages

If a package fails to install:

  1. put its name and details in ‘PIP_pkgs_failed.txt’

  2. in pip_pkgs_to_upgrade.txt,
    comment out all that have been done,
    then run the pip install command again.

(B5) Make diagnostic lists

(see below for how to use them)

    pip list --path ${sp_SYS_OLD} > pip_SYS_leftovers.txt
    pip list --path ${sp_HOME_OLD} > pip_USER_leftovers.txt
    ls -AF1 ${sp_SYS_OLD} | sort -f > SYS_dirlist.txt
    ls -AF1 ${sp_HOME_OLD} | sort -f > USER_dirlist.txt
    pip list --path ${sp_SYS_NEW} > pip_SYS_NEW.txt
    pip list --path ${sp_HOME_NEW} > pip_USER_NEW.txt
    pacman -Qnq $(pacman -Qoq ${sp_SYS_NEW}) > SYS_pkgs_NEW.txt
    pacman -Qmq $(pacman -Qoq ${sp_SYS_NEW}) > AUR_pkgs_NEW.txt

(C) Update the rest of your AUR packages

Use only ONE of these commands !

  • pamac upgrade --aur

  • pikaur -Sua

  • paru -Sua

  • yay -Sua

(D) Troubleshoot

(D1) About the diagnostic lists

These files contain names of packages that failed to re-install:

  • AUR_pkgs_failed.txt
  • PIP_pkgs_failed.txt

These files may contain names of packages that did not get re-installed:

  • OLD_SYS_leftovers.txt
  • pip_SYS_leftovers.txt
  • pip_USER_leftovers.txt
  • SYS_dirlist.txt
  • USER_dirlist.txt

If the files in the above 2 groups are all empty, you’ve finished here.

The files below list every package that definitely did get re-installed:

  • SYS_pkgs_NEW.txt
  • AUR_pkgs_NEW.txt
  • pip_USER_NEW.txt
  • pip_SYS_NEW.txt

If you did not do ‘pip uninstall’ before the system update,
then the ‘leftovers’ and ‘dirlist’ files may include
names of packages that did re-install.

The ‘NEW’ files tell you which have definitely been re-installed.
By comparing ‘NEW’ files with ‘leftovers’ and ‘dirlist’ files
you can eliminate false negatives from ‘leftovers’ and ‘dirlist’ files.

(D2) If you have some that really failed to install

Read the error messages that you saved in your ‘pkgs_failed’ files.
A possible cause of error is that an AUR package
has not been updated for the new Python version,
for which you need to contact the package maintainer at the AUR.

When looking at the ‘dirlist’ files, be aware that
not every item in site-packages is an actual package.
If it’s not a package, most likely it was put there as part of a package.
(The RECORD file in a ‘dist-info’ folder
lists all the files that were installed by that package.)

Look out for anything else that you have manually put there.
If you keep your own modules elsewhere, you may have put a .pth file there.

Occasionally packages get dropped from repos,
or moved between system repos and the AUR in either direction,
so it may be useful to look for a missing package in the system repos and the AUR.

If you look for a package in the PyPI or github etc, be aware that

  • package names that have the prefix “python-” in Linux repos and the AUR
    do not have that prefix in Python, the PyPI, or github etc.

  • pip is not suitable for installing packages that are top-level programs
    as pip does not put starter commands in /usr/bin/ or XDG menus.
    (eg borg, mercurial, pikaur)

There is more detail in part C4 of the background information.

(E) Remove old directories

When you’re sure they contain nothing you need,
remove the old directory trees:

  • sudo rm -rIv /usr/lib/python3.${OLD}

  • rm -rIv $HOME/.local/lib/python3.${OLD}

(F) UPGRADE pip packages

As described in Upgrade many python packages with pip

See also: Python versions, directories, and upgrades.

6 Likes