About Python versions, directories, and upgrades

Contributions Tutorials
1

Python versions, directories, and upgrades

This was originally written as the intro to a set of instructions,
and it has not been completely rewritten yet, so you might see
some references to things that don’t exist now.

Here is the new version of the instructions.

A - Brief info

A1 - Python

Once a year, a Manjaro system update
installs a new version of the Python interpreter.

In the “official” repositories of Manjaro and Arch,
all Python packages are updated to match the new interpreter.
You get these in the same update that brings the new interpreter,
and they continue to work.

Python packages from other sources (AUR, PyPI, Github, etc)
are not updated by Manjaro, and you need to re-install them yourself.

(If you write programs in Python, also see
How to avoid common pitfalls as a developer.)

A2 - pamac and AUR helpers

The real name of the GUI “Add/Remove Software” is ‘pamac’.
‘pamac’ is also a CLI command.
For more info see the Pamac wiki page.

If you use pamac to get AUR packages, then pamac is your “AUR helper”.
There are many other AUR helpers, eg pikaur, yay, paru.

Generally, use only one AUR Helper, as each keeps its own cache and data,
so, while following this guide, use the same AUR helper that you have used previously.

Here’s a wiki page about the AUR: Arch User Repository.

A3 - pip and PyPI

‘PyPI’ is the ‘Python Package Index’, a free online store of Python packages.

The command ‘pip’ is the ‘package installer for python’.
Its default source of packages is the PyPI.

B - Detailed info

B1 - Python

B1a - versions and directories

Python’s major version is 3, for the forseeable future.

Its minor version updates once a year.
Although source code can be compatible across minor version changes,
Python bytecode is minor-version-specific, and most packages contain bytecode,
so Python packages have to be recompiled every year.

Micro version changes do not affect compatibility.
Their release timing is irregular.

In the following description, ‘X’ represents a specific minor version number.

‘/usr/bin/python’ is a symlink.
It points to the ‘system python’ starter ‘/usr/bin/python3.X’.
This is a small file that loads ‘/usr/lib/libpython3.X*.so’, the interpreter core.
(The * represents possible smaller file-version numbers.)
When running, the interpreter can import modules from directory trees
under ‘/usr/lib/python3.X/’ and ‘$HOME/.local/lib/python3.X/’.

Python comes with a lot of modules under ‘/usr/lib/python3.X/’.
Under that there is a special folder named ‘site-packages’,
where the OS’s package manager can install extra Python packages.

The only folder that Python makes in user space is
‘$HOME/.local/lib/python3.X/site-packages/’, which it leaves empty.
Here the user may install python packages without using the OS’s package manager.

B1b - “System Python”

System Python is an installation of Python that is

  • installed by the OS (not requested by any user)

  • used by the OS to run python system scripts
    (in the same way as it runs shell and perl scripts)

“System Python” seems to be unique to Linux;
FreeBSD has never used Python this way;
macOS did very briefly, but doesn’t now.

B1c - Side-by-side Python versions

Version numbers are present throughout Python’s directories,
making it possible for different versions to exist side-by-side.
Each version has a complete set of folder trees under its own version number.

The symlink ‘/usr/bin/python’ points to the version we call ‘system python’,
allowing it to be started by the simple command ‘python’, while
other versions have to be started with a version number, such as ‘python3.9’.

Although the Manjaro update removes the old system python
before it installs a new version as system python,
you can install the old version again from the AUR, at any time;
it just won’t be the ‘system python’ then.

If a package that you need will not run with the new Python,
you can install any older Python from AUR, and install the package in it.
You will have to use a command like ‘python3.10’, instead of ‘python’,
because ‘/usr/bin/python’ will point to the new ‘system python’.

Many people say to use pyenv to have different versions of python,
but pyenv is for developers, and for most users it’s too complex and not necessary.
It’s easier to install an older version from the AUR.

B1d - pip, Python’s package manager

Although pip can be complex, as you can see in the pip documentation,
it cannot manage dependency conflicts in the same way that pacman can.

If you have a lot in your user-space site-packages, you can easily have dependency conflicts.
A good solution then is ‘venv’, as it gives you another site-packages directory in user space,
to which you can move some packages, to separate those that conflict.

Pacman gets packages from the curated repository here,
which is carefully managed to avoid dependency conflicts,
while pip installs packages from anywhere.

By default pip installs from the ‘Python Package Index’,
but it can also get files from github etc, or a local repo.

Pip does not try to distinguish between OS-installed packages
and packages that it installed itself.

A good reason to prefer system/AUR installation over pip
is that a package that is an app (eg pikaur) needs to have
a starter command placed in /usr/bin/ and/or the DE menus,
and pip does not do this. The result is that you have to
start the app with a command like ‘python -m pikaur’.
Pip installation is fine for modules that are imported by other Python programs.

2023-06-21: the new “python-externally-managed” feature

Linux distributions that have a “system python”
can now mark it as “externally-managed”
and then pip will not attempt to install packages into it.

It completely prevents ‘sudo pip install ...’
and even discourages ‘pip install --user’, advising you to use venv instead.

You can still install with ‘--user’ if you add ‘--break-system-packages’.
This simply lets ‘--user’ works as before.
The scary wording is only to warn you that version conflicts are possible.

Such version conflicts would affect only you.
A system process running python cannot be affected by
anything that you install in your “/home/…/site-packages/”
as it never tries to import from there.

For a python process running in your own user account (not root),
“/home/…/site-packages/” shadows “/usr/…/site-packages/”.
How conflicts may arise and how you might resolve them
are described in Upgrading with pip: Version conflicts.

Here are the recommended actions to install a Python package named ‘xyz’,
in order of preference, depending on where you can get the package, and its nature:

  1. sudo pacman -S python-xyz

  2. your-aur-helper -S python-xyz

  3. pipx, for applications written in Python:

    • sudo pacman -S python-pipx # once only

    • pipx install xyz # for each application

    As long as ~/.local/bin/ is on your PATH,
    you can start each app in the usual way.

    pipx puts each app in its own venv (under ~/.local/pipx/venvs/)
    so there is no sharing of libraries, hence no risk of version conflict.

  4. venv, for Python libraries (modules used by other Python programs)

    This might be useful only if you are writing programs yourself,
    because modules in a venv are accessible only from a process of
    python that was started in the same venv, by ‘path/to/the_env/bin/python’.

    1. create a virtual environment
      in which you may install many packages
      (replace ‘an_env’ with your own choice of name)

      python -m venv path/to/an_env

    2. then you can invoke python and pip on these paths

      path/to/an_env/bin/python
      path/to/an_env/bin/pip

    3. e.g. to install package xyz

      path/to/an_env/bin/pip install xyz

Why “python-externally-managed” has been adopted

I wrote the following before I knew about externally-managed,
and it has removed the problems described here.

‘pip install’ would happily overwrite a system package of the same name.

Also, if you point pip at system python’s site-packages folder
and tell it to upgrade packages there (which pacman installed),
it will upgrade all of them to whatever it finds on PyPI.
That could be disastrous for system services that use python.
(Note that pacman upgrades only packages that it installed itself.)

So, it’s obvious that pip must not be allowed
to do bulk upgrades in system site-packages,
and this is a major reason why you should not
allow pip to install packages there.

If you let pip install only in user-space site-packages,
you can safely let pip do all your bulk upgrades in user-space.

The way to achieve this is :
never do ‘sudo pip install some-pkg-name’,
instead do ‘pip install --user some-pkg-name’.

B2 - Pacman and AUR

The AUR system passes each compiled package to pacman to install.

Pacman keeps track of which packages are ‘native’ (from system repos)
and which are ‘foreign’ (from AUR, or manually built with makepkg).

Packages installed any other way (such as by pip) are totally alien:
pacman has no record of them, and its Qo command does not list them.

Although anyone can add a package to the AUR, each package is finally installed by pacman,
and contains info about potential conflicts so that pacman can avoid them.

B3 - System upgrade with new Python version

When Manjaro updates the Python interpreter to a new minor version,
it also updates all of the Python packages in its system repos,
so when you do the system update, those packages are updated at the same time.

All Python packages from system repos are
removed from /usr/lib/python3.{OLD}/site-packages/ and
re-installed in /usr/lib/python3.{NEW}/site-packages/,
and they continue to work as expected.

Python packages from AUR, PyPI, or anywhere else,
did not get updated, and are still in the OLD /site-packages/.
The old Python interpreter has been removed,
so the code in these packages cannot function,
and any program that depends on them cannot run.

To get them running again, they need to be
compiled for the new interpreter and then
installed into the new site-packages directory.
(These 2 steps are usually done by a single AUR or pip command.)

C - Some notes about what the commands do

C1 - bash syntax

‘command2 $(command1)’ is a cleaner way of writing ‘command1 | command2 -’

‘command $(< textfile)’ is a clean way of sending textfile to command.

‘command > textfile’ you probably know: redirect the output of command to textfile.

C2 - pacman

‘pacman -Qo path’ shows a list of names of packages that “own” path.
In our case, “own” means they are installed there.

The small q, as in ‘pacman -Qoq’, leaves out some info so you get only the names.

‘pacman -Qn’ filters for ‘native’ packages, i.e. those that are from system repositories.

‘pacman -Qm’ filters for ‘foreign’ packages, which were installed by pacman,
but are not from system repositories; i.e. those from the AUR, or others manually built with makepkg.

Note that a list from pacman -Q cannot include packages that pacman did not install,
such as those wrongly installed with ‘sudo pip install’.

‘pacman -Rsu $(< textfile)’ uninstalls each package listed in textfile;
the ‘s’ makes it recurse into the package’s depency tree,
but ‘u’ makes it not remove anything needed by another package.

C3 - AUR helpers

pikaur and paru work as expected.
They rebuild packages and remove them from the OLD site-packages dir.
with a simple --rebuild or --rebuild yes

yay currently needs its cache cleaned first, to force rebuilding
and it does not remove old packages from sp_SYS_OLD.
see (Rebuild option is ignored · Issue #2153 · Jguer/yay · GitHub)
--rebuild was fixed recently in git, but is not in release 12.0.5
--answerclean --rebuildall --rebuildtree don’t work at present.
What it was supposed to do:
“–answerclean All” should cause each package to be cleaned from the cache,
“–rebuildall” should cause each package to be rebuilt from source.

pamac has no ‘rebuild’ option,
so to force rebuilding, we clean its cache.

C4 - pip

‘pip freeze’ gets a clean list of package names,
which can then be fed directly to ‘pip install’:
‘pip freeze --path path/to/site-packages > textfile’
‘pip install --user -r textfile’

Note that ‘pip uninstall’ cannot work on the OLD site-packages
unless we use it before doing the system update to the NEW python.
‘pip uninstall’ has no option to point it to a specific directory,
and a requirements file cannot have URLs for the installed path.

To list ALL packages

You can use ‘pip list --path some-directory’ to get
a human-readable list of every package in that directory.

Although this includes many packages that pip did not install,
I’m not sure that it always gets everything, so I use the ls command as well:
‘ls -AF1 path | sort -f’ gives a list similar to pip’s, sorted in the same order.
(‘ls -AF1’ lists only names, one per line, and ‘sort -f’ sorts case-insensitive.)

Not every item in site-packages is an actual package,
but at least it should have been put there as part of a package.
If you want to see what files belong to a package,
read the RECORD file in its dist-info folder.

C5 - looping

Each of those managers can take a list of package names,
but if the attempt to install one package throws errors,
the manager will abort and not process the rest of the list
(if what I’ve read is true).

To make it keep going and process every package,
we can loop through the list in bash, as in this example:
for line in $(< AUR_pkgs_1.txt); do pikaur -S $line --rebuild; done

But my AUR helper (pikaur) writes so much text on the terminal
that if one package fails and the processing does not stop,
I might not notice that one failed, and anyway, it would be
hard to find the details of the one that failed.
So in the directions, I’ve written the non-looping method.

See also: re-installing packages around a system python upgrade.

2 Likes