[HowTo] Upgrade many python packages with pip

Contributions Tutorials
1

Upgrade Python packages with pip

See also: re-installing packages around a system python upgrade.

This is not only for the annual Python upgrade.
It may be better to do it more often, e.g. whenever you do a big Manjaro update.

Caution

  1. Do not run pip as root or with sudo in Linux.
    Always include the --user option (with commands that have it).

    Reasons:

    • to avoid over-writing a system package with an incompatible version

    • to keep your pip packages separate, so that you can upgrade them with pip
      (pip does not know that it should not touch system packages,
      so you cannot upgrade pip packages in bulk in a system directory)

  2. Pip may tell you it has a new version and to run ‘pip install --upgrade pip’,
    but do not run this in Manjaro, as it would over-ride “system pip”.
    (Running it with --user might not be harmful.)

Update 2023-06-21

Beginning with the next Manjaro Stable update (it’s already in Testing),
PEP 668 – Marking Python base environments as “externally managed”
makes it impossible to do ‘sudo pip install’,
and requires an extra option with ‘pip install --user’.

There’s an explanation in background information, part B1d.

Commands

  1. Make sure you’re running the right pip:

    • pip -V

    If, where it says “Python 3.N”, N is not the correct version:

    • rm -vi ~/.local/bin/pip ~/.local/bin/pip3
    • sudo pacman -S python-pip
    • pip -V

  2. List the packages that are out-of-date:

    • pip list --user --outdated --not-required > pip_pkgs_outdated.txt

    (Note: ‘--outdated’ with ‘-not-required’ is less likely to introduce conflicts.)

  3. Reformat the list (this removes column headers and version info)

    _file1=pip_pkgs_outdated.txt
    _file2=pip_pkgs_to_upgrade.txt
    cat ${_file1} | sed 1,2d | awk -F " +" ‘{ print $1 }’ > ${_file2}

    In ‘pip_pkgs_to_upgrade’ you may comment out (with #)
    any that you don’t want to upgrade right now.

  4. Do the upgrade

    • pip install --user --upgrade --break-system-packages -r pip_pkgs_to_upgrade.txt

  5. Dependency version conflicts

    ‘pip check’ reports version conflicts; that’s it’s only function.
    ‘pip install’ also reports conflicts when it has finished installing.

    They may give false alarms, because you can have different versions
    of the same package group (e.g. pyside6) in system and user site-packages
    and in practice they never interact, but pip does not consider that possibility.
    More details below.

  6. upgrade can also downgrade a package, if you specify a lower version:

    • pip install --user --upgrade --break-system-packages 'pkg-name <= version.number'

Version conflicts

Check for version conflicts with ‘pip check’.

Pip’s conflict checking is not perfect. On Linux (the only OS that has ‘system python’)
it does not adequately consider the separation of system and user python processes.

PySide6 is a good example.
In June 2023, my system has installed PySide6 6.5.0,
but it’s a cut-down package that might not have all modules I need,
so I installed PySide6 from the PyPI into my user space, and it’s version 6.5.1.1.
‘pip check’ says that
pyside6 6.5.0 has “requirement shiboken6==6.5.0,
but you have shiboken6 6.5.1.1”,
ignoring the fact that 6.5.0 will be started only by the system,
and shiboken6 6.5.0 is present in system space!
shiboken6 6.5.1.1 is in user space, and the system will never try to import it.

‘sudo pip check’ says “No broken requirements found”.

Version conflicts between user modules and system modules are possible,
but you can avoid them if you remember these principles:

  • a python process started by the system (as root)
    will never conflict with your user-space,
    because root never tries to import from user-space

  • python run by you will always try to import first from user space,
    and will try to import from system-python
    only if it does not find the module in user space

  • so if a process started by you tries to import a module from system python,
    and the version is wrong in system python,
    then simply install the required package into user-space

    pip install --user --break-system-packages pkgname

  • the solution above might cascade more dependency upgrades in user space,
    so consider if downgrading a package in user-space could be a simpler solution.

    pip install --user --upgrade --break-system-packages 'pkgname<=lower-version'

If conflict resolution is not possible

Try ‘venv’.

Each virtual environment has another ‘site-packages’ folder in user-space,
but it’s not in Python’s usual import path, so it cannot conflict.

If you can strategically move some groups of packages into a venv,
such that the conflicting packages are no longer in the same space,
this will resolve the version conflict.

But it won’t help if you need to run conflicting packages in the same program.
In this case you must try to find versions of them that can work together,
or submit bug reports to help get the current versions into a compatible state.

Many people say to use pip only in virtual environments,
but I haven’t found this to be necessary as an ordinary user;
virtual environments are mainly for programmers.

More details

The official pip documentation is at pip.pypa.io/en/stable/cli

System pip

You could say that Linux has “system pip” as well as “system python”.

System pip is installed from the system repo, by ‘sudo pacman -S python-pip’.

If you have another version of python installed from AUR,
which you access by running, for example, ‘python3.9’,
then to access its pip you run ‘pip3.9’, or ‘python3.9 -m pip’,
and this will act on only python3.9’s directories.
(You may first need to install it with ‘python3.9 -m ensurepip’)

‘pip install –user --upgrade pip’ may not do any harm, but it’s not needed.
It installs a second pip in user-space, which your pip commands will run.
Running ‘sudo pacman -S python-pip’ will not change this one; if you want it gone,
you need to delete it with ‘rm -vi ~/.local/bin/pip ~/.local/bin/pip3’;
then the plain ‘pip’ command will run from the system python tree.

Command options

  1. ‘--outdated’ lists only packages that have new versions available.

  2. ‘--not-required’ seems to mean:
    “exclude any package that is not required”,
    where ‘required’ means any outdated package
    that is at the top level (not brought in as a dependency),
    or that is a dependency required by such a top-level package.

    In other words, although a package is outdated, don’t list it if it’s
    a dependency of only an un-listed (not outdated) higher-level package.

    This reduces the risk of dependency version conflicts.

  3. The ‘-r’ in the ‘install’ command means ‘requirements’.
    The file ‘pip_pkgs_to_upgrade.txt’ is a requirements file.

  4. ‘--upgrade’ can actually downgrade if you specify a lower version.
    Note the quotes around ‘pkgname < version-number’,
    which prevent the shell interpreting the less-than symbol.

  5. With install, ‘--dry-run’ is helpful if you need to be cautious.
    It shows what would be done, actually downloads packages, but does not install any.
    Then install without ‘--dry-run’ uses the downloaded packages.

The ‘requirements’ file

There seems to be no way that the ‘list’ command with ‘outdated’
can give output that is formatted as a requirements file.

The default output, with columns, is the best.
Here’s an example:

	Package            Version   Latest   Type
	------------------ --------- -------- -----
	PySide6-Essentials 6.5.0     6.5.1    wheel
	regex              2023.3.23 2023.5.5 wheel

So that we can use it as input for install --upgrade,
we use sed and awk to reduce it to this format:

	PySide6-Essentials
	regex

‘sed 1,2d’ removes the first 2 lines;
‘awk -F " +"’ splits the line at any group of spaces;
“'{ print $1 }'” writes the first group to the output.

A simple example of conflict resolution

My pip list with -not-required did not included tenacity.
I ran ‘pip install --user --upgrade --break-system-packages -r my_list’,
after which ‘pip check’ showed no conflicts.

But without -not-required, tenacity was in the list,
so later I ran ‘pip install --user --upgrade tenacity’ to see what would happen.

pip upgraded tenacity, then told me:

mega-py 1.0.8 requires tenacity<6.0.0,>=5.1.5,
but you have tenacity 8.2.2 which is incompatible.

Things were OK before this upgrade,
so it was obvious that tenacity did not need to be upgraded,
so I downgraded it with
pip install --user --upgrade --break-system-packages 'tenacity < 6.0.0'
(Note the quote marks to prevent bash interpreting the less-than symbol.)

pip then reverted to the previously-installed version,
which was still in its cache:

…
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
…
Successfully uninstalled tenacity-8.2.2
Successfully installed tenacity-5.1.5

See also: re-installing packages around a system python upgrade

2 Likes