The point is, don’t mix system and user Python packages. Hence why a virtual environment is best.
I’ve found a difference.
But it makes no difference to how they work once installed.
Each package has a dir with a name ending with .dist-info or .egg-info.
Some of the .dist-info dirs contain a file named INSTALLER, which contains only one word.
Every package in my ~/.local/lib/.../site-packages/ has an INSTALLER file that says ‘pip’.
Most packages in /usr/lib/python3.10/site-packages/ do not have such a file,
but I have some from PyQt6 that do, and it says ‘pyqtbuild’, while some from PyQt5 say ‘sip-build’.
Arch, AUR, and Manjaro, do not identify themselves in this way.
Summary
Have done a major re-write.
This summary gives a clean overview of what the actions do now.
Where ‘pamac install’ is done, it’s in case
any pkg has been promoted to the Manjaro repo.
pip packages are not uninstalled.
A pip pkg is cached as python source.
pip will fetch a new version if there is one, but if not,
the existing code in the local cache is re-compiled for the new python.
I saw it happen last year – very neat, very quick.
(AUR packages had to re-download and recompile.)
List files need to be adjusted between steps;
here I mention only the main adjustments.
1: remove old AUR pkgs
-
pacman -Qoq /usr/lib/python${old} | pacman -Qmq - > pacmanQom_list.txt -
pamac remove -on "$(< pacmanQom_list.txt)" -
pamac clean --uninstalled --build-files --keep 0
2: re-install pip pkgs
-
pip list --path /usr/lib/python${old}/site-packages/ > pip_list_system.txt -
pip list --path $HOME/.local/lib/python${old}/site-packages/ > pip_list_user.txt -
copy into pip_new.txt the names of the pkgs you want
(including from pacmanQom_list.txt if you wish) -
pamac install --upgrade "$(< pip_new.txt)"
(then in the list, comment out any that were installed by this) -
pip install --user -r pip_new.txt
3: install new AUR pkgs
-
in pacmanQom_list.txt, comment out any you don’t want
-
pamac install --upgrade "$(< pacmanQom_list.txt)"
(then in the list, comment out any that were installed by this) -
pamac build "$(< pacmanQom_list.txt)"
I see some issues with this tutorial, I don’t see why we would even use the AUR to Install python packages when we can install them using pip and that way bypass the AUR middle man of inspecting every package before installation, installing packages as root or from the AUR is a expert topic and although I do sudo pip sometimes I would not advise that for common users as it will cause issues that they would have to deal with and break their installation, there is already many support requests in the forum of users that broke functionality by using this methods.
I agree with you, but … did you read the previous conversation in this topic?
Consider these points:
- people who are mainly into shell and pacman, and know little about python, have a hard time believing that it’s possible to replace AUR packages with pip packages;
- to please everyone, it’s best to allow the user to choose.
This process only re-installs AUR packages that they had previously installed;
so it happens only if they had previously chosen to do that ‘expert’ thing.
Going off-topic a bit –
Yesterday I began wondering why Linux distros even provide python packages in their repos. They could provide the interpreter and core modules, and let pip do the rest. Maybe provide a PyPI mirror and configure their pip to use it.
It’s very strange the way Linux hard-codes a path into the interpreter, so you can’t copy it to some other location and use it independently; to do that you have to compile your own.
On windowz you can put several versions of python interpreter in different places, and use each of them independently, at the same time. You can do that on Linux if you compile each interpreter with the right flags so that it’s not bound to a specific path, but it would be easier if you could just grab a binary of any version from the repo.
For “system python”, they could make a minimal python, just enough to cover what the system scripts do, and lock it away where normal users would never touch it. Then make it easy for users to install any number of versions of python, whatever they need – with the latest being the default, of course.
So instead of a single package manager, I’d do pacman -S some_package and got error: missing python-xyz. And then do pip install xyz and got error: missing npm-abc, and then do npm install abc … so much fun. 
I’d argue that this is one of the main things distros are about. To put together 1000s different parts and try to make them easy to manage.
Well, all packages are there for a reason. It’s not like you have 100 python packages installed just in case you might need them in future.
1 Like
Any reason why pacman could not call pip?
The idea is not a simple quick-fix thing,
but it could be a valid useful idea in the longer term.
(we’re off topic, and this could go on for days … 
)
Thanks for that.
Didn’t know about the Qm part, but after trying it I put it in the tut.
-Qm lists foreign (aka AUR) packages. This is just an example to check what’s from AUR before you do the update. Or you can check what’s left in afterwards (and rebuild it in one go, with command @Yochanan posted).
About a year ago, someone persuaded me that PyPI packages were equal to or better than AUR packages, and it’s taken me until now to realize it’s not true for all packages.
If the python package is an app in its own right, such as TortoiseHg or Pikaur :
-
the AUR package puts an executable in /usr/bin/,
and places desktop files for GUI menus -
pip doesn’t do that, so you can’t run it normally,
you have to do ‘python -m appname’
So pip is fine for modules that are only imported by other python programs,
but not for standalone python apps.
As for my long complex post (that I’ve deleted):
I wasn’t completely clear about the purpose of it.
A big part of my thinking was to provide some general info about Python in Linux/Manjaro, but this should have a wiki article or tutorial post of its own.
The possibilty of switching between AUR and pip is also a separate subject.
The advice provided when there’s a Python version upgrade should deal only with the upgrade.
But this on its own
‘yay -S --rebuildall $(pacman -Qoq /usr/lib/python3.9) --answerclean All’
is not enough, because noobs use only pamac, some people use pikaur etc,
and it doesn’t mention PyPI/pip.
Covering all AUR helpers and PyPI/pip might be too much to put straight into the update announcement, so an article on this topic could still be useful.
EDIT: new version is up.
I guess, because if the package at AUR has not changed, upgrade won’t have a trigger to do anything.
Rebuilding will use the existing source code,
but compile it with the current (new) Python interpreter,
and put it in the current (new) site-packages dir,
which is exactly what we want.
So rebuild is the way for yay and pikaur,
but pamac does not have a rebuild option.
Will 'pamac build' make it rebuild a package that has not changed?
https://wiki.manjaro.org/index.php/Pamac doesn’t say, and doesn’t link to a man page.
I’ve looked through the local man page and don’t see an option that would force rebuild.
If we do ‘pamac remove’ then ‘pamac build’, would that force a rebuild?
Or might it use an existing built package from cache?
‘pamac remove’ then
‘pamac clean --keep 0 --uninstalled --build-files’ then
‘pamac build’ should do it,
but would also require downloading again.
Is it really necessary?
EDIT: for now i’m assuming pamac needs to remove and clean before building,
and have re-written the main post accordingly. UTC 22-12-05 11:25