When Python has its annual update, how to re-install your own packages

Early versions of this topic said to do ‘pip freezebefore the Python upgrade.
Later I realized ‘pip list’ is better for upgrading, and can be done after the Python upgrade.

Last major revision: UTC 2022-01-11 04:30

If you are familiar with this subject,
you might prefer to use the short version at the end of this post.

EDIT 2022-05-07:

  1. I now think that it’s usually better to re-install from PyPI (pip with --user)
    and avoid AUR if possible.

  2. This guide covers only simple situations;
    if you have complex requirements you’ll need to study elsewhere;
    see post 17 below.


Full version

Although this post is rather long, most of the details are about

  • explaining things for people who are not familiar with the subject,

  • dealing with errors that might never happen,

so you’ll probably find that it’s not as hard as it looks.

Background info

A new Python version is released in October every year.

Python is installed in ‘/usr/lib/pythonX.Y/’,
where ‘X.Y’ stands for a version number.

You can browse ‘/usr/’ directories in Dolphin;
you have access to read but not write.

There can be more than one version installed.
The one that ‘/usr/bin/python’ points to is called “system python”.
This is currently version 3.10.

System python” really is part of the operating system;
if you mess it up too much, the system won’t run.

Optional Python packages get installed into a sub-folder called ‘site-packages’.

There is another ‘site-packages’ folder, in your user space, at
‘$HOME/.local/lib/pythonX.Y/site-packages/’.
There is nothing else in ‘$HOME/.local/lib/pythonX.Y/’, only ‘site-packages’.

Optional packages installed by
pacman, pamac, AUR helpers, (and ‘sudo pip install’, which you should not do)
go into ‘/usr/lib/pythonX.Y/site-packages/’.

Optional packages installed by ‘pip install --user
go into ‘$HOME/.local/lib/pythonX.Y/site-packages/’.

Separate version directories, such as
“…/lib/python3.9/…” and
“…/lib/python3.10/…”,
are completely separate from each other;
there is one for each version of installed Python.
After a Python upgrade, Python and pip
use only the new one, not the old one.

The Manjaro system update removes from the old version
all files that the system originally installed there.
If you find some files left in ‘site-packages’,
they are there because you installed them.

Packages installed from AUR and PyPI are your packages;
the system does not need them to be re-installed.
You can choose whether you want them or not.

WARNING:

  • sudo pip install’ installs packages in system python,
    and some experts strongly advise not to do that,
    because pip packages cannot be guaranteed to be safe.

    Bear in mind that:

    • AUR packages always install in system python,
      and they also cannot be guaranteed to be safe;

    • malware installed in user space can steal files,
      and passwords that are saved by browsers;
      root privileges are not required for these exploits.

    Carefully evaluate each package on AUR or PyPI before installing,
    as malware has been found (and removed) on both systems.

A. Do the system upgrade, with special care for AUR Python packages

Note:

  • In this document, ‘pkg_name’ is a place-holder.
    When copying a command, replace ‘pkg_name’ with an actual package name.

  • ‘PyPI’ is the “Python Package Index”, where pip gets packages from.

First finish studying the issues in the Manjaro update post,
and make whatever preparations are required for fixing those issues.

Then:

  1. Upgrade the system without AUR:
    pamac upgrade
    If there’s a problem, do not proceed to the next step.
    Deal with the problem and repeat ‘pamac upgrade
    until the upgrade completes successfully
    or says “there is nothing to do”.

  2. Upgrade your AUR Python packages:
    pamac build $(pacman -Qoq /usr/lib/python3.9)
    That’s “3.9”, to list the old packages that we have not updated yet.

    • If a package fails to upgrade, remove it:
      pamac remove -un pkg_name
      then do step 2 again.
      If you don’t remove it, it will probably fail again in step 3.
      (Make a note of the name, if you want to try re-installing it later.)
  3. Upgrade the rest of your (non-Python) AUR packages:
    pamac upgrade --aur

    • If a package fails, do ‘pamac build pkg_name’.
      If it still won’t install, remove it: ‘pamac remove -un pkg_name
      then do step 3 again.

    • If you’d like to see a list of the remaining not-updated packages:
      pamac checkupdates --aur

  4. Make sure pip is installed as a system package:
    sudo pacman -S python-pip
    If it says “up to date – reinstalling”, it’s fine; enter ‘n’ to abort.
    Otherwise enter ‘y’ to install it.

B. Get a list of other Python packages

Caution:
The first time in a linux session that you run pip, it may try to open a wallet and/or keyring.
If the wallet dialog does not come to the front, it will seem like pip has hung;
remember to click the wallet button in the task bar.

The next command gets a list of every package installed in your user site-packages
(assuming it has a ‘dist-info’ or ‘egg-info’ folder).

pip list --path $HOME/.local/lib/python3.9/site-packages/

If there are too many to remember easily, save them to a file:
pip list --path $HOME/.local/lib/python3.9/site-packages/ > pip_list.txt

If a package name is cryptic and you don’t know what it is,
read the ‘METADATA’ file in its ‘dist-info’ folder.
Alternatively, search for it at pypi.org.

If you might have previously installed a package in system python,
also do ‘pip list --path /usr/lib/python3.9/site-packages/
and add whatever it finds to your list, to be considered for re-installation.

C. Decide which source to use

The name of a package in the Arch/Manjaro repo may be different from the PyPI name,
e.g. “commonmark” in PyPI is “python-commonmark” in the Arch and Manjaro repos.

When you install a package, you need to use the name that it has in the repo from which you are installing.

To get the right package name, do ‘pamac search --aur pkg_name

If you want more info about a package, search for it here:

It’s better to install a package from official Manjaro repos, if it exists there,
because it’s more likely to be safe,
is versioned to match other components,
and will be upgraded automatically next time.

With AUR and PyPI, neither repo is intrinsically better than the other.
On each repo, there is huge variation of quality between packages.

If a package is available from both AUR and PyPI,
consider the quality of that package at each place.
You can see more info about a package by searching for it here:

Note:

  • With the AUR, each package has to be downloaded again and rebuilt.
    I tried re-installing one without rebuilding, and it went into the old site-packages.

  • With pip, I was surprised to see it quickly pull files out of its cache
    and install them into site-packages of the new Python.
    (.py files can work with the new version; only .pyc files are version-specific.)

D. Re-install Python packages from your pip list

You can leave out any that you don’t want to re-install.

For each package that you want to install again,
use only one of the following commands.
(If one fails, you can try another).

Remember to use the right ‘pkg_name’ for the source.

  • Manjaro: pamac install pkg_name

  • AUR:       pamac build pkg_name

  • PyPI:      pip install --user pkg_name


Short version

This is only a check list.
Before you use it, you should be familiar with the details provided in the full version above.

Stage 1

  1. pamac upgrade

  2. pamac build $(pacman -Qoq /usr/lib/python3.9)

    • If a package fails to upgrade: ‘pamac remove -un pkg_name
      Then do step 2 again.
  3. pamac upgrade --aur

    • If a package failed: ‘pamac build pkg_name’.
      If it still won’t install: ‘pamac remove -un pkg_name’.
      Then do step 3 again.

    • to list all remaining not-updated packages:
      pamac checkupdates --aur

  4. sudo pacman -S python-pip
    If it says “up to date – reinstalling”, enter ‘n’ to abort.
    Otherwise enter ‘y’ to install it.

Stage 2

pip list --path $HOME/.local/lib/python3.9/site-packages/ > pip_list.txt

Look for alternative sources for re-installation
pamac search --aur pkg_name

Re-install from a Manjaro repo if possible
(unless you have a specific need for another source).

If a package is available from both AUR and PyPI,
consider the quality of that package at each place:

Evaluate packages carefully on AUR and PyPI,
as malware has been found on both systems.

Stage 3

Use only one of the following commands (unless one fails) for each package.
Use the right ‘pkg_name’ for the source.

  • Manjaro:   pamac install pkg_name

  • AUR:         pamac build pkg_name

  • PyPI:        pip install --user pkg_name


12 Likes
[Unstable Update] 2021-12-13 - Python 3.10 Rebuilds
[Stable Update] 2022-02-14 - Kernels, Haskell, Cutefish 0.7, Deepin, Firefox, Maui 2.1.1, Systemd
Problems with python modules
[Stable Update] 2022-04-15 - Kernels, Mesa, Nvidia, Budgie, PipeWire, LIbreOffice, KDE Frameworks, Wine
[Testing Update] 2022-01-03 - OpenSearch 1.2.3, Nvidia 495.46, Wine 7.0-rc3, LibAdwaita 1.0, Mesa 21.3.3
[Testing Update] 2022-01-17 - OpenSearch, Kernels, KDE, Cinnamon, Pipewire, Mesa
[Testing Update] 2021-12-31 - Last Update of the year! Kernels, KiCad 6.0, Wine 7.0-rc2
Some packages that I can't update
[Stable Update] 2022-01-23 - OpenSearch, Kernels, KDE Software, Cinnamon, Pipewire, Mesa, Gnome, LibreOffice
[Testing Update] 2022-01-24 - OpenSearch, Kernels, VirtualBox, Systemd, Firefox, Linux-Firmware
[Stable Update] 2022-01-02 - Kernels, Systemd, KDE Frameworks, Mesa, Xorg-Server, Wine, Python 3.10
How to best manage python interpreters for stability (dev)?
[Stable Update] 2022-01-02 - Kernels, Systemd, KDE Frameworks, Mesa, Xorg-Server, Wine, Python 3.10
Python 3.9 to 3.10 validation and cleanup
[Stable Update] 2022-01-02 - Kernels, Systemd, KDE Frameworks, Mesa, Xorg-Server, Wine, Python 3.10
Python 3.9 to 3.10 validation and cleanup
Problems with python modules
[Testing Update] 2022-02-01 - Kernels, Pipewire 0.3.44, Qt5, Mesa 21.3.5, Wine 7.0
[Testing Update] 2022-02-03 - Kernels, LibreOffice 7.3.0, KDE Gear 21.12.2, Nvidia 510.47.03, Calamares
[Testing Update] 2022-02-04 - 4.x-Kernels, Pipewire 0.3.45, Linux-Firmware
[Stable Update] 2022-02-05 - Kernels, LibreOffice, KDE Gear, Nvidia, Calamares, VirtualBox, Pipewire, Mesa, Systemd
[Testing Update] 2022-02-06 - Kernels, Pamac, Cleanups
[Testing Update] 2022-02-21 - Kernels, Toolchain, Plasma 5.24.1, KDE FW 5.91.0, Mesa 21.3.6, Wine 7.2, Gnome 41.4, Linux-FW, Gstreamer 1.20.0, Nvidia, Pipewire
[Testing Update] 2022-02-24 - Kernels, Plasma 5.24.2, Firefox Beta, AMDVLK 2022.Q1.3
[Testing Update] 2022-02-25 - Kernel, Mesa 21.3.7, PulseAudio, LibxCrypt
[Testing Update] 2022-03-20 - Kernels, Haskell, Python, Gnome, PHP
[Stable Update] 2022-02-27 - Kernels, Mesa 21.3.7, Plasma 5.24.2, Frameworks 5.91, Pipewire 0.3.47, Toolchain, Gstreamer 1.20, Nvidia 510.54
[Testing Update] 2022-03-16 - GStreamer, Haskell, KDE Frameworks, Budgie, NetworkManager, Calamares
[Stable Update] 2022-03-14 - Kernels, KDE, LibreOffice, Kodi, Qt5, Mozilla, NetworkManager, Pipewire
[Testing Update] 2022-05-17 - Kernels, Toolchain, Nvidia, KDE, Qt, Maui, LibreOffice
[Testing Update] 2022-03-24 - Linux517, Nvidia, Palemoon (not)
[Testing Update] 2022-04-02 - Kernels, Mesa, Plasma 5.24.4, Deepin, LibreOffice, PipeWire, Wine 7.5
[Testing Update] 2022-04-04 - Mesa 21.3.8, Blender 3.1.2, Xorg-Stack, Haskell, Python
[Testing Update] 2022-04-25 - Kernels, Mesa, Gnome 42, KDE Apps & Frameworks, LxQt, Firefox, Thunderbird, Virtualbox
[Testing Update] 2022-05-19 - Kernels, Qt5, KDE-Git, Firefox Beta, Haskell
[Testing Update] 2022-05-18 - Kernels, Plasma
[Testing Update] 2022-05-21 - Mesa 22.0.4, Firefox 100.0.2, Qemu
[Testing Update] 2022-05-23 - KDE-Git, Haskell
[Testing Update] 2022-05-04 - Kernels, Plasma 5.24.5, Systemd, Gstreamer, Pipewire, Firefox, Thunderbird
[Testing Update] 2022-02-09 - Kernels, Haskell, Cutefish 0.7, Deepin, Firefox, Maui 2.1.1, Systemd
Error opening audacity "libSoundTouch.so.1"
[Testing Update] 2022-05-10 - Kernels, Mesa, Gnome 42.1, LibreOffice, Gstreamer, Qemu
[Testing Update] 2022-03-12 - Kernels, KDE, LibreOffice, Kodi, Qt5, Mozilla, NetworkManager, Pipewire
Update fails "unable to satisfy dependency"

Note that pip install defaults to --user since about v20.1, so specify --user in no longer needed, while never use sudo is still true

1 Like

Not entirely … pip list without any flags acts as if it is sudo or global pip.

Which is why the list action above should invoke the --user flag.
Though … it should probably get rid of the path portion as it should not be required … and there is the slight chance that the location is incorrect, whether due to configuration or otherwise.

1 Like

and then gets a permission error.

I’ve put “–user” back into that last ‘pip install’ command.
(I hadn’t taken it out of the other one.)

Hang on, you said ‘pip list’ …
It works right for me as it is, but not with --user :

$ pip list --user --path $HOME/.local/lib/python3.9/site-packages/
ERROR: Cannot combine '--path' with '--user' or '--local'

As I said … the path should not be there.
Simply …

pip list

will return as if it is sudo or global.

pip list --user

will return correctly --user installed pip packages, whatever the actual configured path.

After the upgrade it is required.
I simplified the procedure by cutting out the need to do anything before the upgrade.

But for the current version.

I’m telling them to list the old version,
after they’ve done the upgrade.

oh right … afterwards it makes sense.

But you can trim it slightly …

--path $HOME/.local/lib/python3.9/

[jh@SSD2 220106 17:06:26 zp0]$ pip list --path $HOME/.local/lib/python3.9/site-packages/Package            Version
------------------ -------
commonmark         0.9.1
filetype           1.0.7
font-line          3.1.1
fonttools          4.11.0
gdown              3.12.2
Genshi             0.7.5
html2docx          1.3.1
mpv                0.1
pymdown-extensions 8.1.1
PySocks            1.7.1
python-docx        0.8.10
python-uinput      0.11.2
rst2html5          2.0
tqdm               4.57.0
xxhash             2.0.2
[jh@SSD2 220106 17:06:33 zp0]$ pip list --path $HOME/.local/lib/python3.9/
[jh@SSD2 220106 17:06:38 zp0]$
(nothing)

Odd. works here.
…oh … probably wildcard expand.

$ pip list --path /home/kropo/.local/lib/python3.10/*
Package Version
------- -------
cowsay  4.0

But … at that point its probably better to just leave it instead of using *

Thanks.
I’m hoping it’s good enough now to leave as is for a while …

Why oh why are you recommending such opinionated stance? :rage:

There are reasons why someone installed those packages per user level instead of global level.

Please omit this recommendation and strive to preserve the old situation.
As in user level desired package should be installed per user level, global desired packages should be installed at global level.

For the rest, great tutorial :+1:t2:

It’s not really my opinion.
It’s in line with what seems to be general moderator policy
of telling people to do the safest thing.

It’s about official repos being safer than PyPI or the AUR.

And AUR being easier to upgrade than PyPI.

It’s easy to install something with pip and not know that it’s in the repos.
This advice is intended to get people familiar with the idea of checking the repos first.

The info is intended for relative newcomers.
Those who know enough, know that they can do things their own way. :slight_smile:

It’s common in Windows, but in Linux I haven’t noticed
anything about per-user app installs (expect with pip).
Can it be done with pacman, with the --root option?

1 Like

Thank you for this “tutorial”. It finally solved a few issues I had with Qtile and some extensions, as everything is written and configured in Python. I didn’t install all old packages and that caused problems. Also great to see that native pamac is used, instead of a third party program yay.

thanks for the tutorial!
supposing that I reinstalled all my Python packages via pacman, would there be any point if I deleted the contents of $HOME/.local/lib/python3.9/site-packages, now that we’re on 3.10?

[I’m assuming “point” means “problem”.]

No, no problem.
You can remove all of the ‘$HOME/.local/lib/python3.9/’ tree.

Also, my machine has no ‘/usr/lib/python3.9/’ tree now.
I vaguely remember removing it myself.
Maybe it would have been removed by the upgrade,
but there was a package in ‘/usr/lib/python3.9/site-packages/’
that the system had not installed, so it left it there.

Use the following commands to see what you’ve got :
ls -l /usr/bin/python*
ls -ld /usr/lib/python*

If there’s no python3.9 in /usr/bin/, there’s no point having python3.9 in /usr/lib/.
But if you want to be cautious, just rename it first,
then if something complains you can rename it back.

Btw, I thought it was obvious from the background info
that you could remove the old version, but I suppose
not everyone has time to think through it all… :slight_smile:

After a Python upgrade, Python and pip use only the new one, not the old one.

2 Likes

With “opinionated” I mean the terminology used in software development.
In short, the tutorial gives little room for alternative ways of handling Python packages which fit better to the person’s workflow.

You barely want previous per-user installed Python packages installed as system-wide packages. Especially when you’ve a multiuser system where each user prefer a different version of a Python package. Or an user is not allowed to use specific software.

Not every Python package is provided in the Manjaro repos nor in the AUR. Or do not immediately up-to-date when upstream has a new version.

For some packages like youtube-dl, gallery-dl and PixivUtil2, it is as good as required to upgrade the packages with pip (or a different package manager like asdf). Because upstream delivers that many frequent new releases that Manjaro repos nor AUR can not caught up the update speed.

1 Like

I agree with you on every point.

Later, I decided that for myself, I prefer pip/PyPI (with ‘--user’) rather than AUR.
Upgrading is actually easier than with AUR.

After you first wrote about it, some of my edits lifted PyPI to equal level with AUR:

If a package is available from both AUR and PyPI,
consider the quality of that package at each place …
Evaluate packages carefully on AUR and PyPI,
as malware has been found on both systems.

When I first wrote the guide, I was new to the idea of using the AUR for Python packages, and simply followed recommendations about that from experienced Arch/Manjaro users. Now I prefer PyPI for most packages.

To accomodate all your points, the whole thing would have to be re-written, or replaced, and I don’t really have time now.

A lot of users who know nothing about Python installations, versions, or package updating, were getting confused when Python did a major version bump.
All I tried to do was make a simple step-by-step guide for those users,
and I didn’t expect anyone else would look at it.

People with more complex needs would know better than to rely on this little guide, wouldn’t they? :slight_smile:

You don’t have to edit the whole thing, just add a link to @RoestVrijStaal’s message to the end of your first post so people could follow it and learn more.

I’ve added it at the top,
along with a note about preferring pip over AUR
(the advantages of pip were already mentioned in the long version).

Last Christmas I tried to write this post in the wiki,
but it disappeared, so I wrote it here instead.
Today I discovered that my post is in the wiki, though now out of date,
so I’ll have to do some clean up when I have time … :slight_smile:

1 Like