[HowTo] DRAFT: Update your Python packages after a Python version upgrade

UTC 2022-12-15 02:15
Have made a fresh start.
Created a new topic, as most replies below are not relevant to the new version.

3 Likes

Why is that already linked to the troubleshoots? As I see it python3 wonā€™t get any updates in 2022 anymore and will stay on 3.10 based on that: Bootstrapping Python - Arch-dev-public - lists.archlinux.org. You can check in IRC what the plans at Arch are to have a rough roadmap ā€¦

This sounds to me a lot more complicated than it needs to be. You need to rebuild all dependant AUR packages as usual. And those will be:

pacman -Qoq /usr/lib/python3.10 | pacman -Qm -

It will list exactly whatā€™s needed, always. Itā€™s not pacmanā€™s fault if people install things with ā€˜sudo pipā€™.

To get some expert eyeballs on it, to make sure itā€™s correct before the noobs need it.
A few have seen it now, so itā€™s OK to remove it until 3.11 arrives.

I donā€™t have enough brain-power to get involved at that level. :slight_smile:

Way ahead of you. :stuck_out_tongue_winking_eye:

1 Like

Itā€™s simpler than last yearā€™s version.
The main simplification Iā€™ve made is remove the ā€œAUR or pipā€ question:
AUR python packages from v3.10 will be re-installed to 3.11 via pip install --user.
(I doubt thereā€™s any package in AUR thatā€™s not in PyPI, but I covered the possibility just in case there is.)

Directions for noobs always seem complicated.
Simple instructions for experienced users are meaningless to noobs.

Iā€™m not an expert, and the intended audience is people who know even less than I do.
These directions should work for them right now.
Later, when they have more experience, they can figure out their own preferred way.

What? AUR packages have nothing to do with packages installed via Pip.

This is what we did last year for Python 3.10. I donā€™t see any reason to include anything different. If users install packages with Pip, then thatā€™s up to them to manage them. Nothing to do with Manjaro updates.

It seems we were missing the --rebuildall Yay flag, so I think it should be:

yay -S --rebuildall $(pacman -Qoq /usr/lib/python3.9) --answerclean All

This is a hot mess.

This topic is meant to replace the one I wrote last year, which was far more messy, I thought, and also covered pip. (Look in the edit history to see what was there.)

@linux-aarhus gave this new one a :100: flag ā€“ now I wonder if he had read all the details. :slight_smile:

Anyway, this is now a wiki post that others can work on, and I would rather drop out than get into debates, as I donā€™t really have that much time.

But this is for noobs/newbies (I hate those words :wink: ) who do not know how to (nor even that they need to) upgrade the python packages that they previously installed, wherever they got them from.

There were a lot of questions in the update announcements about python packages not working, so I wrote that post last year to help them out.
It was messy, so a few weeks ago I tried to write a neater version.

As far as those noobs are concerned, they ( donā€™t care / are not aware of ) whether they got a package from AUR or PyPI, and I thought it would simplify their situation if they use only pip and avoid AUR for python packages. (Should add a note advising them to install only via pip in future.)

Some people complained about directions to use pacman and yay, saying we should advise to use homegrown pamac, so I did that, even though I use pacman and pikaur for myself.

Conclusion: for noobs it would work as is; the objections seem to come from experts who donā€™t realize itā€™s for noobs. :slight_smile:

Donā€™t get me wrong, I appreciate the time effort youā€™ve put into this in order to help new users.

Honestly, I havenā€™t had time to sit down and go through the whole thing. At this point, we have more time than we had originally thought before Python 3.11 lands in the repos.

Again, system Python packages installed from the repos and the AUR have NOTHING to do with packages installed via pip install --user.

These are not what this is about,
because they get re-installed into the new python by the system update,
so the noob user has no problem with them.
(I guess this happens simply because all python packages in official repos have been upgraded to match the new interpreter, and each such package removes its old version.)

Packages from the AUR and those installed via pip install (--user or not) are linked together because these are the ones that stop working after the python interpreter upgrade, and this is the problem the noobs need help with.

As far as a noob is concerned, there is no need to know whether a package was installed from AUR or by pip, and itā€™s simpler to just re-install them all with pip (with --user).

Iā€™m not aware of any advantage to ever installing any Python package from AUR, if you can get it from PyPIā€¦

My comment was on the draft and I have read the details and I have the utmost respect for the hard work you put into describing the issues.

And the details reflect correctly how system python is organised in an Arch based system.

The listings you describe - I have not tested - but looking at your efforts as a whole - I assume you have tested them.

It has been repeated over and over - never use sudo pip pkgname on an Arch based system - but there is always some who does it - I know I had to learn it the hard way.

As with any system sync - it is recommended to create a list of your custom packages and verify if and how they are effected by the sync.

When you sync your system and the system python changes and you have not excersised your due diligence thus getting into troubles then the explanations you have provided will give the reader a deeper understanding why it is such a bad decision to use pip and sudo in the same command.

I stand by my respect for your efforts :100:

I have seen a lot of comments blindly recommending a pamac sync command which includes both downgrade, aur and development in the same command which is insanely stupid in my opinion

pamac update --enable-downgrade --aur --devel

Thus the sentence

didnā€™t raise any flags.

In fact I expect a lot of readers has copy-pasted that command into an alias - and then they donā€™t understand why all hell breaks loose.

On a sidenote - I donā€™t want to be that grumpy knows-it-all-idiot who aggressively correct other members - so I stay positive and commends contributions that elevates the general information level in a positive contructive manner.

2 Likes

Some of the commands are the same as were in the other post last year.

I researched the new commands carefully,
but donā€™t have time to set up a rig for testing,
probably not even when py 3.11 arrives in unstable/testing ā€¦

FIrst of all, I second that.

There is one big advantage - using ONE package manager.

And in this regard, if you want to be less confusing, you should avoid mixing pamac and pacmanā€™s commands, eg. pamac remove $(pacman -Qoq /usr/lib/python${old}).

I think itā€™s the other way around.

This is not so much about HowTo but more about users who donā€™t want to learn how to maintain their systems. If they donā€™t read all of this on 1000 other places, they wonā€™t read this either. They want all commands together for a quick no-think copy/paste. :stuck_out_tongue:

1 Like

Of course there is! AUR packages are installed to the system just like repo packages.

AUR packages cannot be reinstalled with Pip. Pip packages cannot be reinstalled via the AUR.

Youā€™re mixing it all together and will confuse anyone who has AUR packages but no Pip packages or vice versa.

I figure that this tutorial will have to include the option to re-install some from AUR, as it will otherwise be seen as restricting the userā€™s choice. So we donā€™t need to argue this point any further. :slight_smile:

But, for the record: they are all Python packages. AUR and PyPI are simply 2 ways of getting the same Python package. After itā€™s installed, as far as Python is concerned, thereā€™s no difference. If you examine the files in a site-packages dir, you canā€™t see any difference.

In the list of packages to re-install, those got by ā€˜pacman -Qoā€™ have the name prefixed by python-, but by simply removing that, you get the name that works with pip. And the files finally installed in site-packages, even by the AUR package, have that shorter name.

Sometimes thereā€™s a real difference, eg PySide6 from AUR is minimal, runtime-only, while the PyPI package includes QtDesigner. But in this case, the names arenā€™t quite the same, so you couldnā€™t accidentally substitute one with the other.

Repo and AUR packages install to the system location: /usr/lib/python${ver}/site-packages/

Installing packages with pip install --user install to the user location: ~/.local/lib/python${ver}/site-packages/

See the difference now?

Who? pyside6 is in the extra repo.

There are nearly 2000 Python packages in the Manjaro repos,
so itā€™s definitely the place to look first.

/usr/lib/python${ver}/site-packages/ and
~/.local/lib/python${ver}/site-packages/
are both on pythonā€™s internal path,
so when python does ā€˜import pkg_nameā€™, both locations are searched,
and the import is successful, whichever one pkg_name happens to be in.

Of course. Especially since pyside6 is not in the AUR? :wink:

You did see @linux-aarhusā€™s helpful tutorial post he linked earlier, right? :wink:

I have now. :slight_smile:

Have been writing python scripts for about 20 years.

Have tried virtual envs, and decided I donā€™t need them, as Iā€™m making only one small app now, and itā€™s quite safe running on system python. Any extra packages needed, I install via pip --user, if theyā€™re not in the repo.

A year ago I compiled python from source, to get an interpreter that would be totally independent of system python (v envs are not). Havenā€™t bothered to keep it up-to-date.

Iā€™m past retirement age, and there are not enough good years left to waste time messing about with things I donā€™t need. :slight_smile: