What? AUR packages have nothing to do with packages installed via Pip.
This is what we did last year for Python 3.10. I don’t see any reason to include anything different. If users install packages with Pip, then that’s up to them to manage them. Nothing to do with Manjaro updates.
It seems we were missing the --rebuildall Yay flag, so I think it should be:
yay -S --rebuildall $(pacman -Qoq /usr/lib/python3.9) --answerclean All
This is a hot mess.
This topic is meant to replace the one I wrote last year, which was far more messy, I thought, and also covered pip. (Look in the edit history to see what was there.)
@linux-aarhus gave this new one a 
flag – now I wonder if he had read all the details. 
Anyway, this is now a wiki post that others can work on, and I would rather drop out than get into debates, as I don’t really have that much time.
But this is for noobs/newbies (I hate those words 
) who do not know how to (nor even that they need to) upgrade the python packages that they previously installed, wherever they got them from.
There were a lot of questions in the update announcements about python packages not working, so I wrote that post last year to help them out.
It was messy, so a few weeks ago I tried to write a neater version.
As far as those noobs are concerned, they ( don’t care / are not aware of ) whether they got a package from AUR or PyPI, and I thought it would simplify their situation if they use only pip and avoid AUR for python packages. (Should add a note advising them to install only via pip in future.)
Some people complained about directions to use pacman and yay, saying we should advise to use homegrown pamac, so I did that, even though I use pacman and pikaur for myself.
Conclusion: for noobs it would work as is; the objections seem to come from experts who don’t realize it’s for noobs.
Don’t get me wrong, I appreciate the time effort you’ve put into this in order to help new users.
Honestly, I haven’t had time to sit down and go through the whole thing. At this point, we have more time than we had originally thought before Python 3.11 lands in the repos.
Again, system Python packages installed from the repos and the AUR have NOTHING to do with packages installed via pip install --user.
These are not what this is about,
because they get re-installed into the new python by the system update,
so the noob user has no problem with them.
(I guess this happens simply because all python packages in official repos have been upgraded to match the new interpreter, and each such package removes its old version.)
Packages from the AUR and those installed via pip install (--user or not) are linked together because these are the ones that stop working after the python interpreter upgrade, and this is the problem the noobs need help with.
As far as a noob is concerned, there is no need to know whether a package was installed from AUR or by pip, and it’s simpler to just re-install them all with pip (with --user).
I’m not aware of any advantage to ever installing any Python package from AUR, if you can get it from PyPI…
My comment was on the draft and I have read the details and I have the utmost respect for the hard work you put into describing the issues.
And the details reflect correctly how system python is organised in an Arch based system.
The listings you describe - I have not tested - but looking at your efforts as a whole - I assume you have tested them.
It has been repeated over and over - never use sudo pip pkgname on an Arch based system - but there is always some who does it - I know I had to learn it the hard way.
As with any system sync - it is recommended to create a list of your custom packages and verify if and how they are effected by the sync.
When you sync your system and the system python changes and you have not excersised your due diligence thus getting into troubles then the explanations you have provided will give the reader a deeper understanding why it is such a bad decision to use pip and sudo in the same command.
I stand by my respect for your efforts
I have seen a lot of comments blindly recommending a pamac sync command which includes both downgrade, aur and development in the same command which is insanely stupid in my opinion
pamac update --enable-downgrade --aur --devel
Thus the sentence
didn’t raise any flags.
In fact I expect a lot of readers has copy-pasted that command into an alias - and then they don’t understand why all hell breaks loose.
On a sidenote - I don’t want to be that grumpy knows-it-all-idiot who aggressively correct other members - so I stay positive and commends contributions that elevates the general information level in a positive contructive manner.
Some of the commands are the same as were in the other post last year.
I researched the new commands carefully,
but don’t have time to set up a rig for testing,
probably not even when py 3.11 arrives in unstable/testing …
FIrst of all, I second that.
There is one big advantage - using ONE package manager.
And in this regard, if you want to be less confusing, you should avoid mixing pamac and pacman’s commands, eg. pamac remove $(pacman -Qoq /usr/lib/python${old}).
I think it’s the other way around.
This is not so much about HowTo but more about users who don’t want to learn how to maintain their systems. If they don’t read all of this on 1000 other places, they won’t read this either. They want all commands together for a quick no-think copy/paste. 
Of course there is! AUR packages are installed to the system just like repo packages.
AUR packages cannot be reinstalled with Pip. Pip packages cannot be reinstalled via the AUR.
You’re mixing it all together and will confuse anyone who has AUR packages but no Pip packages or vice versa.
I figure that this tutorial will have to include the option to re-install some from AUR, as it will otherwise be seen as restricting the user’s choice. So we don’t need to argue this point any further.
But, for the record: they are all Python packages. AUR and PyPI are simply 2 ways of getting the same Python package. After it’s installed, as far as Python is concerned, there’s no difference. If you examine the files in a site-packages dir, you can’t see any difference.
In the list of packages to re-install, those got by ‘pacman -Qo’ have the name prefixed by python-, but by simply removing that, you get the name that works with pip. And the files finally installed in site-packages, even by the AUR package, have that shorter name.
Sometimes there’s a real difference, eg PySide6 from AUR is minimal, runtime-only, while the PyPI package includes QtDesigner. But in this case, the names aren’t quite the same, so you couldn’t accidentally substitute one with the other.
Repo and AUR packages install to the system location: /usr/lib/python${ver}/site-packages/
Installing packages with pip install --user install to the user location: ~/.local/lib/python${ver}/site-packages/
See the difference now?
Who? pyside6 is in the extra repo.
There are nearly 2000 Python packages in the Manjaro repos,
so it’s definitely the place to look first.
/usr/lib/python${ver}/site-packages/ and
~/.local/lib/python${ver}/site-packages/
are both on python’s internal path,
so when python does ‘import pkg_name’, both locations are searched,
and the import is successful, whichever one pkg_name happens to be in.
Of course. Especially since pyside6 is not in the AUR?
You did see @linux-aarhus’s helpful tutorial post he linked earlier, right?
I have now.
Have been writing python scripts for about 20 years.
Have tried virtual envs, and decided I don’t need them, as I’m making only one small app now, and it’s quite safe running on system python. Any extra packages needed, I install via pip --user, if they’re not in the repo.
A year ago I compiled python from source, to get an interpreter that would be totally independent of system python (v envs are not). Haven’t bothered to keep it up-to-date.
I’m past retirement age, and there are not enough good years left to waste time messing about with things I don’t need.
The point is, don’t mix system and user Python packages. Hence why a virtual environment is best.
I’ve found a difference.
But it makes no difference to how they work once installed.
Each package has a dir with a name ending with .dist-info or .egg-info.
Some of the .dist-info dirs contain a file named INSTALLER, which contains only one word.
Every package in my ~/.local/lib/.../site-packages/ has an INSTALLER file that says ‘pip’.
Most packages in /usr/lib/python3.10/site-packages/ do not have such a file,
but I have some from PyQt6 that do, and it says ‘pyqtbuild’, while some from PyQt5 say ‘sip-build’.
Arch, AUR, and Manjaro, do not identify themselves in this way.
Have done a major re-write.
This summary gives a clean overview of what the actions do now.
Where ‘pamac install’ is done, it’s in case
any pkg has been promoted to the Manjaro repo.
pip packages are not uninstalled.
A pip pkg is cached as python source.
pip will fetch a new version if there is one, but if not,
the existing code in the local cache is re-compiled for the new python.
I saw it happen last year – very neat, very quick.
(AUR packages had to re-download and recompile.)
List files need to be adjusted between steps;
here I mention only the main adjustments.
pacman -Qoq /usr/lib/python${old} | pacman -Qmq - > pacmanQom_list.txt
pamac remove -on "$(< pacmanQom_list.txt)"
pamac clean --uninstalled --build-files --keep 0
pip list --path /usr/lib/python${old}/site-packages/ > pip_list_system.txt
pip list --path $HOME/.local/lib/python${old}/site-packages/ > pip_list_user.txt
copy into pip_new.txt the names of the pkgs you want
(including from pacmanQom_list.txt if you wish)
pamac install --upgrade "$(< pip_new.txt)"
(then in the list, comment out any that were installed by this)
pip install --user -r pip_new.txt
in pacmanQom_list.txt, comment out any you don’t want
pamac install --upgrade "$(< pacmanQom_list.txt)"
(then in the list, comment out any that were installed by this)
pamac build "$(< pacmanQom_list.txt)"
I see some issues with this tutorial, I don’t see why we would even use the AUR to Install python packages when we can install them using pip and that way bypass the AUR middle man of inspecting every package before installation, installing packages as root or from the AUR is a expert topic and although I do sudo pip sometimes I would not advise that for common users as it will cause issues that they would have to deal with and break their installation, there is already many support requests in the forum of users that broke functionality by using this methods.
I agree with you, but … did you read the previous conversation in this topic?
Consider these points:
This process only re-installs AUR packages that they had previously installed;
so it happens only if they had previously chosen to do that ‘expert’ thing.
Going off-topic a bit –
Yesterday I began wondering why Linux distros even provide python packages in their repos. They could provide the interpreter and core modules, and let pip do the rest. Maybe provide a PyPI mirror and configure their pip to use it.
It’s very strange the way Linux hard-codes a path into the interpreter, so you can’t copy it to some other location and use it independently; to do that you have to compile your own.
On windowz you can put several versions of python interpreter in different places, and use each of them independently, at the same time. You can do that on Linux if you compile each interpreter with the right flags so that it’s not bound to a specific path, but it would be easier if you could just grab a binary of any version from the repo.
For “system python”, they could make a minimal python, just enough to cover what the system scripts do, and lock it away where normal users would never touch it. Then make it easy for users to install any number of versions of python, whatever they need – with the latest being the default, of course.