[HowTo] After a Python interpreter upgrade, reinstall non-working Python packages

Question

After a recent version upgrade of the Python interpreter,
some programs that rely on Python packages cannot run.

How do we fix this situation?

Answer

Once a year, a Manjaro system update installs a new version of the Python interpreter.
In the Manjaro repositories, all Python packages are also updated, so they continue to work.

But Python packages from other sources (AUR, PyPI, etc) stop working after this update,
and you need to re-install them yourself.

The detailed instructions below are for people who have little experience with Manjaro,
and who use only “Add/Remove Software” or pamac for installing and updating software.

If the detailed instructions do not suit you, please read the section called “The general idea”.
It assumes you have more experience, and aims to help you understand what to do for your own situation.

If any package does not re-install correctly, save the output from the terminal in a text file.
If you can’t correct the situation yourself, ask in the forum and provide that output.

DISCLAIMER

The authors cannot accept responsibility for any damage caused by following this guide.
It is your responsibility to make sure you always have backups of significant data.

If you see a mistake in this guide, please let us know so we can correct it.

Read this first

Click the arrow to open this section

Full system update :

This guide begins after you have completed the system update
that installs the new version of Python and removes the old one.

We recommend to always do system updates without AUR, reboot,
make sure the system is running fine, then do the AUR updates.

On this occasion, follow this guide before you do the AUR updates.
If you’ve already done your AUR updates, just follow this guide anyway.

Prior knowledge required :

  • The GUI “Add/Remove Software” is called “pamac” in the CLI.
    Read the Pamac wiki page.
    If you use pamac (GUI or CLI) to get AUR packages, then pamac is your “AUR helper”.

  • If you’re not sure what the AUR is, please study the AUR wiki page.

  • Open a terminal and use it (e.g. konsole in KDE).
    When pasting a command into a terminal, use Ctrl+Shift+V.
    When copying information from a terminal, use Ctrl+Shift+C.

  • Use and edit plain text files.
    Copy text from the terminal into a text file.
    Manage multiple text files so that new info does not remove previous info.

CAUTION

  • Always use the same AUR Helper
    because each one keeps its own cache and data.

  • Always include ‘--user’ with ‘pip install
    Never do ‘sudo pip install
    as it could over-write a system package of the same name.
    (If you have installed any package with ‘sudo pip’,
    this guide will re-install it with ‘pip --user’.)

The general idea

Click the arrow to open this section
Background info

There are 2 sub-folders with partial path “python3.N/site-packages”.
(‘N’ represents Python’s minor version number, which changes once a year.)
One is in system space, under “/usr/lib/”, and
the other is in user space under “$HOME/.local/lib/”.

Python comes with a lot of modules under ‘/usr/lib/python3.N/’,
but when first installed, both ‘site-packages’ directories are empty.

The OS and the user can install extra python modules,
which come in python packages, and are installed into ‘site-packages’.

Manjaro has nearly 2000 Python packages in its system repos.
There are also many in the AUR, and many more in the PyPI.

Python packages from system repos and from the AUR
get installed into the site-packages under /usr/lib/.

Python packages that you install from PyPI, with ‘pip’,
should be installed with ‘--user’, without sudo,
so that they get installed into site-packages under $HOME.

Letting pip install into system site-packages risks overwriting a system package of the same name,
because pip and the OS know nothing about each other’s packaging systems.
The AUR system avoids such name conflicts, as it co-operates with the ALPM (ArchLinuxPackageManager).

Intro to what you need to do

The following examples use ${OLD} and ${NEW} as placeholders for Python versions.
(In late 2022 / early 2023, OLD is 3.10 and NEW is 3.11)

When Manjaro updates the Python interpreter to a new minor version,
it also updates all of the Python packages in its system repos.

After you install such an update,
/usr/lib/python${OLD}/ has been mostly removed, and
/usr/lib/python${NEW}/ has been created, and populated.

All Python packages from system repos have been
removed from ${OLD} and re-installed in ${NEW} site-packages,
so that these packages continue to work as expected.

But Python packages from AUR, PyPI, or anywhere else,
did not get updated, and are still in ${OLD} site-packages.
The Python${OLD} interpreter has been uninstalled,
so the code in these packages cannot function,
and any program that depends on them cannot run.

Note: Python source code is mostly compatible with
a minor version change, but Python bytecode is not,
and the packages contain bytecode.

To get them running again,
the packages that came from AUR need to be rebuilt, and
the packages that came from PyPI need to be re-installed.

Suggested actions
First

Set these vars:

OLD='3.10'; NEW='3.11'

Make sure the latest pip is installed at system level:

  • sudo pacman -S python-pip
AUR

The old AUR packages that need to be rebuilt can be written to a file by:

  • pacman -Qoq /usr/lib/python${OLD} | pacman -Qmq - > AUR_package_names.txt

You can use that file as a basis to make lists for the next two commands.

Always use the same AUR helper (yay or pikaur) that you normally use.


Subsection: PIKAUR may need re-installing

If you do not use pikaur, skip this pikaur sub-section.

Pikaur is a Python app, so has probably stopped working along with the rest of them.

Check it by simply asking it to show its version:
pikaur -V
If it says to do
pikaur -S --rebuild --aur pikaur
go ahead and do it.

If it said “rebuild Pikaur manually”, and you know how to, do it.
If that worked, skip the rest of this pikaur sub-section.

If you’re not comfortable with building apps from source,
or you tried to rebuild and failed, you can install it from the PyPI:
pip install --user --upgrade pikaur
Normal pikaur commands won’t work right now:
python -m pikaur -V’ should work,
pikaur -V’ should not.
To get it working normally, let pikaur install itself from the AUR:
python -m pikaur -S pikaur
Now ‘pikaur -V’ should work.

But now you have two working installations of pikaur:
one in user space from the PyPI, one in system space from the AUR.
Get rid of the one in user space:
rm -rIv $HOME/.local/lib/python${NEW}/site-packages/pikaur*
Clean it from the cache as well: ‘pip cache remove pikaur*’.
“pip uninstall pikaur” could uninstall both of them.
Don’t know if it could selectively uninstall only one, as I has already deleted one.


To remove any AUR packages that you don’t want now:

  • pikaur -Rsun $(< AUR_to_uninstall.txt)

  • yay -Rsun $(< AUR_to_uninstall.txt)

To rebuild those that you do want:

  • pikaur -S --rebuild $(< AUR_to_rebuild.txt)

  • yay -S --rebuildall --answerclean All $(< AUR_to_rebuild.txt)

Use your lists, plus output on the terminal,
to identify the packages that have successfully updated,
and make sure those packages are no longer present
in the old system site-packages directory.
For any that is still there, delete it.
Otherwise, in the next step, pip might also re-install it.

PIP

When re-installing packages for the new Python with pip,
there are basically 2 ways to do it:

  • re-install the same version, or
  • re-install with upgrade to the latest version.

The examples below cover both ways.

For reference, here’s the documentation for pip.
Here’s a simpler overview of upgrading pip packages.

Some web sites advise not to upgrade packages in bulk because of dependency conflicts,
but it seems that pip resolves conflicts safely since version 20.3, as decribed here:
if it cannot resolve conflicts, it prints details, then exits without changing anything.

Also see the official pip documentation about dependency resolution.

Resolving conflicts is beyond the scope of this guide, but it’s basically about
editing the version numbers in freeze lists (requirements files)
so that every package has a dependency it can work with.
If that’s not possible, you’ll need to think about python environments.

Note that upgrading pip packages is not something to do only when upgrading Python,
as their version requirements are more likely related to other python packages than to Python itself.
Python upgrades might be less risky if you upgrade pip packages in between Python upgrades.

Command examples

Check if you already have any missing or conflicting package dependencies.

  • pip check

(If it reports any problem, it’s possible that the ‘upgrade’ option below could resolve it.)

List PyPI packages with version numbers:

  • pip freeze --path $HOME/.local/lib/python${OLD}/site-packages > pip_freeze_user.txt

  • pip freeze --path /usr/lib/python${OLD}/site-packages > pip_freeze_system.txt

List PyPI packages without version numbers
(the awk part removes “==version_number” from each line) :

  • pip freeze --path $HOME/.local/lib/python${OLD}/site-packages | awk -F "==" '{ print $1 }' > pip_pkgs_user.txt

  • pip freeze --path /usr/lib/python${OLD}/site-packages | awk -F "==" '{ print $1 }' > pip_pkgs_system.txt

Make safe copies of the “freeze” files, as they contain version info that you won’t be able to get later.

Copy whatever is in each …system.txt file to the corresponding …user.txt file.

In each …user.txt file, you can comment out (with #)
any package that you don’t want to re-install right now.

If you want to re-install the same version of each package:

  • pip install --user -r pip_freeze_user.txt

If you want to re-install and upgrade at the same time

  • pip install --user --upgrade -r pip_pkgs_user.txt

I think the result of each of the last two commands can be undone by the other,
so if you don’t like the result of one, try running the other.
Both could be undone by ‘pip uninstall -r pip_pkgs_user.txt’
(if you un-comment all the names you want to uninstall).

If you want to upgrade some packages and only re-install others, you could
comment out different names in ‘pip_freeze_user.txt’ and ‘pip_pkgs_user.txt’,
then run both commands.

Check again:

  • pip check
Leftover packages?

Simply looking into the site-packages directories,
it can be difficult to recognize packages, because:

  • file names don’t always match package names;
  • one package can have many files;
  • one package can pull in others as dependencies.

But if you want to have a look anyway …

with a file manager, or with these commands,
look at the old site-packages directories:

  • ls -lA /usr/lib/python${OLD}/site-packages
  • ls -lA $HOME/.local/lib/python${OLD}/site-packages

and the new site-packages directories:

  • ls -lA /usr/lib/python${NEW}/site-packages
  • ls -lA $HOME/.local/lib/python${NEW}/site-packages

If there is anything left in old system site-packages, do:

  • pacman -Qoq /usr/lib/python${OLD}

If pacman prints any package name,
then that package was installed via pacman.
If you really want that package, try installing it again,
first with pacman, and if it fails, rebuilding with your AUR helper.

Checking further is not really necessary.
Left-over files in old site-packages are quite useless, so you may as well delete them all.
If some app later tells you a dependency is missing, you can simply re-install it at that time.

Detailed instructions for pamac users

Click the arrow to open this section

Each command here starts with a bullet point.

Copy that command (without the bullet),
paste it into the terminal, then press Enter.

Each command is meant to be a single line of text.
If a long command auto-wraps onto more lines in your browser,
copy all of it at once; when pasted it will be one line;
if your terminal is narrow, it may auto-wrap there also, but that’s OK.

First commands
  1. Make sure the latest pip is installed at system level:

    • sudo pacman -S python-pip
  2. These variables (OLD and NEW), are the versions of old and new Python.
    ‘3.10’ and ‘3.11’ are for early 2023.
    If you’re doing this at a later time, make sure the values have been updated.
    (The next upgrade could be in late 2023, but is likely to be in early 2024.)

    • OLD='3.10'; NEW='3.11'
Diagnostics 1

These commands will give you clues about what needs to be done.

We’ll run them again at the end, to check that everything was fixed.

  • pip check

  • checkrebuild -v

  • pacman -Qoq /usr/lib/python${OLD}

  • ls -lA /usr/lib/python${OLD}/site-packages/

  • ls -lA $HOME/.local/lib/python${OLD}/site-packages/

If none of them lists any files or packages,
everything’s fine, there’s nothing to do here.
Go to the “Finish up” section and run the commands there.

Re-install AUR packages

Do you have any old AUR python packages?

  • pacman -Qoq /usr/lib/python${OLD}

  • pacman -Qoq /usr/lib/python${OLD} | pacman -Qmq -

The first command shows AUR packages and any system package that did not get upgraded.
The second command shows only AUR packages.

If nothing is listed by those commands, skip to the pip section.

Re-install

  1. Make a list file:
    pacman -Qoq /usr/lib/python${OLD} | pacman -Qmq - > AUR_list_1.txt

  2. Read the file.
    If there is any package you don’t want to re-install,
    move (not just copy) its name to another list file, “AUR_list_2.txt”

  3. Completely uninstall those you do not want:

    • pamac remove -oun $(< AUR_list_2.txt)
  4. Remove then rebuild the packages that you do want.
    (Removing them first ensures that you’ll get a new version,
    and also that pip will not try to re-install them in the next step.)

    1. pamac remove -ou $(< AUR_list_1.txt)

    2. pamac clean --uninstalled --build-files --keep 0

    3. pamac build $(< AUR_list_1.txt)

Re-install pip packages
  1. Make lists from the old python site-packages directories:

    • pip freeze --path $HOME/.local/lib/python${OLD}/site-packages > pip_freeze_user.txt

    • pip freeze --path /usr/lib/python${OLD}/site-packages > pip_freeze_system.txt

    Two new list files have been created. Read them.

    If they are both empty, there’s nothing to do here.

  2. Make a list for re-installing:

    Copy whatever is in ‘pip_freeze_system.txt’ to ‘pip_freeze_user.txt’

    Comment out (with #) the name of any package you don’t want to re-install at present.

  3. Re-install

    • pip install --user -r pip_freeze_user.txt

    Read the output.

Note that this did not upgrade any pip package to a newer version.
If upgrading is required, it’s a separate issue, as it’s more likely to be
related to other Python programs rather than to the new Python interpreter.
Upgrading packages with pip can be done at any time,
and it might be best to do it in between Python upgrades.

Diagnostics 2
  • pip check

  • checkrebuild -v

  • pacman -Qoq /usr/lib/python${OLD}

  • ls -lA /usr/lib/python${OLD}/site-packages/

  • ls -lA $HOME/.local/lib/python${OLD}/site-packages/

There might still be files in the old site-packages directories,
but if the other checks show nothing, it’s probably OK to delete the files.
These old files are useless now anyway, and if some app later says
it’s missing a dependency, you can re-install the dependency at that time.

If you look in the new site-packages directories,
you should see the packages you have re-installed just now.
Pip packages should be under ‘$HOME/.local/’, and AUR packages under ‘/usr/’.

  • ls -lA $HOME/.local/lib/python${NEW}/site-packages/

  • ls -lA /usr/lib/python${NEW}/site-packages/

Caution: in site-packages, it can be difficult to recognize exactly which
files and directories were installed by a particular package, because:

  • the file/directory name is not always the same as the package name;
  • one package can have many files and directories;
  • one package can pull in others as dependencies.
Finish up
  1. Do the full AUR upgrade (to get packages that are not Python) :

    • pamac upgrade --aur
  2. When you’re sure you don’t need the old ‘site-packages’ directories, delete them:

    • sudo rm -rIv /usr/lib/python${OLD}

    • rm -rIv $HOME/.local/lib/python${OLD}