Python 3.8: preparing for the switch (upgrade and clean up)

Hi, while I’m waiting to apply the [Stable Update] 2019-12-10 which also and especially brings the update of python to 3.8, I’m investigating on packages/applications which depends on Python.

For now I understood that the packages installed/builded from AUR (those which depends on Python) must be rebuilded, but I'm puzzled for those installed via pip or third-party (eg some that I compiled from some github’s repository).

On my system, eg, I have two folders of two different Python version: /usr/lib/python2.7/site-packages/ and /usr/lib/python3.7/site-packages/: On both folders there are some duplicate things (some are identical which doesn't indicate a version), but some with different version, eg: in the directory /usr/lib/python3.7/site-packages/ there are numpy and
numpy-1.17.3-py3.7.egg-info folders; in the directory /usr/lib/python2.7/site-packages/ there are numpy and numpy-1.16.4-py2.7.egg-info. So I used the command pacman -Qo:

[dave@probook] $ pacman -Qo '/usr/lib/python2.7/site-packages/numpy-1.16.4-py2.7.egg-info' 
/usr/lib/python2.7/site-packages/numpy-1.16.4-py2.7.egg-info/ is owned by python2-numpy 1.16.4-2
[dave@probook] $ pacman -Qo '/usr/lib/python3.7/site-packages/numpy-1.17.3-py3.7.egg-info' 
/usr/lib/python3.7/site-packages/numpy-1.17.3-py3.7.egg-info/ is owned by python-numpy 1.17.3-1

Using pacman -Qi I can see which packages needs python2-numpy and python-numpy; python2-numpy is an optional dependency for pygtk, which is a dependency for a lot of “important” apps for me, like gimp and nmap; is also required by avahi which was installed by default on Manjaro. Python-numpy is needed also for other applications which I installed from Pamac, some from the official repo and something (eg some deps) from AUR.

But apart this example, I ask to myself if the "site-packages" folders are containing some leftover or unneeded stuff, and what will happen with the upgrade to Python 3.8: I guess that upgraded applications/packages which have been rebuilt with python 3.8 will have upgraded needed stuff in the /usr/lib/python3.8/site-packages and maybe will left something in the /usr/lib/python3.7/site-packages folder. Am I right?

I admit that I am a little bit intimidated of the fact that with the upgrade to python 3.8 I will have something which stop to works and also a lot of leftovers.
There is some way/approach to find these leftovers and check if something is broken and in need of manual upgrade (eg stuff installed via pip)?

I'm sorry for all these questions and preoccupations, I admit to be a newbie abouth these specific things.

EDIT: I also discovered that python2 will be deprecated soon:

Anything you manually compiled on your system (including packages from the AUR, pip, etc) will need to be rebuilt after the update to continue to function. There may be specific packages that have issues with 3.8 even after being rebuilt, tho I haven't run into any.

When packages are updated/rebuilt, most of the files/folders will disappear from /usr/lib/python3.7/site-packages and will appear in /usr/lib/python3.8/site-packages instead. There may be some files/folders left in the 3.7 folder even if they aren't used; you'll just have to check the individual packages (which shouldn't be many after the main update).

There are some default packages that still require python 2. They'll need to be updated upstream or replaced before python 2 can be removed. This is something to keep an eye on in future updates.

2 Likes

Hello, I have not manually compiled anything on my system, would it be fairly safe to say I can ignore the Python 3.8 warnings in the Dec. 10 update thread? I am not aware of using Python if I do even use it. I searched package manager for "python" and saw a few things installed out of a very long list. thanks

At the very least it should be safe to upgrade - any manual rebuilding/recompiling would be done after the upgrade and after a reboot anyway. That's when you'd check what's left in /usr/lib/python3.7/site-packages

OK, I'll be backing up my system before doing this one anyway, but if I've never compiled anything to begin with [as a noob I haven't crossed that bridge yet] then am I correct in thinking Python 3.8 should be a non-issue on my systems? thanks

You may not know if you manually compiled something. Most packages in the AUR are compiled on your computer when you go to install them (that's why in Pamac the button to install an AUR package is "Build" as opposed to "Install" for packages in the official repos). Packages from the AUR that were compiled against the older version of python will need to be rebuilt (read: re-installed).

Overall it should not be an issue. Again, any rebuilds would be done after the bulk of the update is already complete.

Note: not all AUR packages rely on python. The number of packages needing a rebuild depends on what you have installed

ah! excellent, thanks, I was unaware of this, the missing piece in my thought process :slight_smile: thanks for the explanation
edit: as a noob when I see the word "compiled" I automatically think of having to type cryptic lines in a file that I must first know how to find somewhere. That's what I thought I meant when I was saying I "never compiled anything"
edit:

heh, didn't catch that earlier, sinking in a bit more now :slight_smile: Not quite so scary!

1 Like

Understandable. Sometimes I forget not everyone knows how incredibly easy it is to compile things on Arch-based systems (to the point of not even realizing it). Glad that's straightened out

1 Like

A great "example" is GIMP: it depends on pygtk; pygtk depends on python2-cairo, python2-gobject2 e python2-numpy; those three depends on Pyhton2.
I ask to myself why a Project like GIMP didn't switch already in past to Python3.

EDIT:
Seems that GIMP from AUR doesn't rely anymore on Python2: gimp-git depends on python-gobject (instead of python-gobject2) which depends on Python3.

This little post, dug up via your mention in the latest stable thread, was a huge illumination for this noob coming up on one year on Linux desktop with Manjaro. Thank you!

1 Like

After backing up I just updated my less critical laptop, rebooted fine. I didn't think to check beforehand but looking now in usr/lib I see there is no python 3.7 folder at all [to see if it's empty] but there is now a python 3.8 folder. This computer is a much newer install, would it be safe to say python 3.7 was there before, has been deleted and replaced by 3.8, leaving no python concerns? The screen shot below is of my non-updated critical laptop which is a much older install and probably has many more bits and bobs installed that I may not be aware of:
image
Both machines have the 2.7 python folder so I guess that is the thing to be aware of for future updates like you mentioned.

Correct. Packages now using 3.8 would have previously been using 3.7, and python 2.7 is still in use. The newer install didn't have any leftover packages still using 3.7, so the folder was no longer there.

And if this folder will be still present will meaning that there are leftovers indicating that something hasn't been updated and require a manual update? (eg using pip install)

Yes, at which point you can use
pacman -Qo folder_or_file
to determine what package owns the file (this only works if the file/folder is from a package that was installed with pacman or pamac)

It is possible that not all files will be removed. If they're still there, no package owns them, and you don't notice any issues, they can likely be ignored.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Forum kindly sponsored by