Python 3.9 to 3.10 validation and cleanup

[Daniel-I]

Today’s new stable update went well, and as this is my first time experiencing a majaor python update under any GNU/Linux distro, I’d like to make sure all is well within/under python.

I took the following command results (post update) as a good sign:

$ pacman -Qoq /usr/lib/python3.9
error: No package owns /usr/lib/python3.9

… I take it to mean that nothing is left left relying on python 3.9 (as far as what was installed via pamac/pacman) post update.

Also, the only thing left in pamac’s foreign/AUR list that listed python as a dependency was zenstates-git… and as I never use it, I’ve uninstalled it.

I’m not 100% sure how to interpret the pip freeze commands before the update though… and I also don’t recall installing much if anything via pip (will ensure I document pip installs in the future).

Python 3.9 sudo pip list

apparmor==3.0.3
appdirs==1.4.4
application-utility==1.3.2
btrfsutil==5.15.1
CacheControl==0.12.6
ceph==1.0.0
ceph-volume==1.0.0
cephfs==2.0.0
cephfs-shell==0.0.1
certifi==2021.10.8
cffi==1.15.0
chardet==4.0.0
colorama==0.4.4
contextlib2==0.6.0.post1
cryptography==35.0.0
cupshelpers==1.0
distlib==0.3.3
distro==1.6.0
dnspython==2.1.0
docopt==0.6.2
entrypoints==0.3
gufw==21.4.0
html5lib==1.1
idna==3.3
importlib-metadata==4.8.1
iotop==0.6
jeepney==0.7.1
keyring==23.0.1
keyutils==0.6
lensfun==0.3.95
LibAppArmor==3.0.3
louis==3.20.0
lxml==4.6.4
Markdown==3.3.6
meld==3.20.4
more-itertools==8.10.0
msgpack==1.0.2
mutagen==1.45.1
netsnmp-python==1.0a1
npyscreen==4.10.5
opendht==2.3.0
ordered-set==4.0.2
packaging==20.9
pacman-mirrors==4.23.2
pep517==0.12.0
Pillow==8.4.0
ply==3.11
progress==1.6
pwquality==1.4.4
pycairo==1.20.1
pycparser==2.21
pycryptodome==3.11.0
pycryptodomex==3.10.1
pycups==2.0.1
pycurl==7.44.1
PyGObject==3.42.0
pyOpenSSL==21.0.0
pyparsing==2.4.7
PyQt5==5.15.6
PyQt5-sip==12.9.0
pysmbc==1.0.23
PyYAML==5.4.1
rados==2.0.0
rbd==2.0.0
reportlab==3.6.2
requests==2.26.0
resolvelib==0.5.5
retrying==1.3.3
rgw==2.0.0
SecretStorage==3.3.1
six==1.16.0
speedtest-cli==2.1.3
TBB==0.2
team==1.0
toml==0.10.2
tomli==1.2.2
udiskie==2.3.3
ufw==0.36
urllib3==1.26.7
webencodings==0.5.1
websockets==9.1
youtube-dl==2021.6.6
yt-dlp==2021.11.10.1
zipp==3.6.0

Python 3.9 user pip list

aiosmb==0.2.50
aiowinreg==0.0.7
asn1crypto==1.4.0
asysocks==0.1.2
minikerberos==0.2.14
msldap==0.3.30
oscrypto==1.2.1
portio==0.5
prompt-toolkit==3.0.24
tqdm==4.62.3
wcwidth==0.2.5
winacl==0.1.2
winsspi==0.0.9

Post update… my “user pip list” is empty… and (according to meld) my “sudo pip list” has the same “named” entries, although as expected some have had version updates.

So what do I do with my “3.9 user pip list”? portio sounds like something I might have pursued, but really I’m pretty sure I didn’t install these… and not really knowing what they are or what they were used for, I don’t think I’ll miss them. In this case, can I just delete my HOME folder “leftovers”?

I also found some python 3.9 leftovers in /usr/lib/… can I delete this too?

[cscs]

@Daniel-I

I take it to mean that nothing is left left relying on python 3.9

Yup. Nothing like a repo package anyway…
But that should have meant ythe directory was removed.
(on my system I have never removed an old python directory, like 2.7 etc)
So … it is possible something is relying on it …

you didnt …

@Daniel-I

sudo pip list

Do that, for anything but list, right?
Do not use sudo with pip

@Daniel-I

So what do I do with my “3.9 user pip list”?

So you have not removed 3.9 yet?
In any case … you can freely remove (through pip and/or manually deleting directories) old python version stuff or anything you dont want.

So…

@Daniel-I

can I just delete my HOME folder “leftovers”?

yes.

@Daniel-I

I also found some python 3.9 leftovers in /usr/lib/… can I delete this too?

Now this is odd. I am inclined to say yes, as nothing owns it … but I still wonder why it exists.
(as noted above these directories usually are removed along with the python package)

[Daniel-I]

@cscs

you didnt …

@Daniel-I

sudo pip list

Do that, for anything but list, right?
Do not use sudo with pip

Ah, my mistake… I should have included the full commands I ran from the stable “wiki” post…

@philm

Do ‘pip freeze’ BEFORE the update

If you might have installed something via pip,
get this info before the python upgrade,
as there’s no easy way to get it afterwards.

sudo pip freeze > pip_list_sudo.txt
pip freeze --user > pip_list_user.txt
/path/to/<venv_name>/python -m pip freeze -l > pip_list_<venv_name>.txt

After the upgrade, although they say to do something like
sudo pip install -r pip_list_sudo.txt,
look at the lists and consider whether you still want them all.

It’s generally not a good idea to install with pip in the system python,
so look for packages you could install from AUR instead,
and if you still want some pip installs, do: pip install --user

Also, the list from ‘pip freeze’ includes version numbers,
so you might want to manually install afresh to get current versions.

pacman -Qoq /usr/lib/python3.9 is quite different.
If you use it before the update, it will list many packages
that pacman will soon re-install for python 3.10,
so use it after the update, when it will get only AUR packages.
To use it with pikaur:
pikaur -S $(pacman -Qoq /usr/lib/python3.9) --rebuild

… so one of my commands was run with sudo as recommended

• $ sudo pip freeze > pip_list_sudo.txt
• $ pip freeze --user > pip_list_user.txt

@cscs

@Daniel-I

can I just delete my HOME folder “leftovers”?

yes.

That’s good news, I was hoping that was the case!

@cscs

@Daniel-I

I also found some python 3.9 leftovers in /usr/lib/… can I delete this too?

Now this is odd. I am inclined to say yes, as nothing owns it … but I still wonder why it exists.
(as noted above these directories usually are removed along with the python package)

The only time I’ve experienced something like this (under Windows) was when a file was locked or in use or not part of the uninstall script… and usually folders refuse to delete so long as there are any files inside. But as you can see from the folder that the vast majority of the /usr/lib/python3.9 folder was removed… with the exception of one file that prevented its parent folders from being removed.

[cscs]

@philm

sudo pip freeze > pip_list_sudo.txt

Honestly its just kinda silly to do that.
First because, again, users should not invoke sudo with pip. So there should be no ‘sudo pip’ packages.
But second because … that list will be largely useless because of listing all sorts of system packages that rely on python but are from the repos and not pip (example: ufw,gufw) …

The line afterwards …

@philm

After the upgrade, although they say to do something like
sudo pip install -r pip_list_sudo.txt,

Is downright dangerous.

[Daniel-I]

I always appreciate your feedback @cscs . Being a relatively new to gnu/linux user still… I always make sure to post my questions if something looks questionable. Whether that be a command or the end result of a command.

But as the commands I ran were just generating lists, I wasn’t too concerned about them as it was my intention to follow one sage piece of advise…

@philm

look at the lists and consider whether you still want them all.

… i.e. no plan to run sudo pip install -r pip_list_sudo.txt in my case.

[cscs]

The instruction there really should be avoided. Only the ‘user’ list should be needed or used.
And then only if you want those python packages.

I am currently trying to get the announcement ‘wiki’ section revised…

[Kevin77]

Broke my system following this piece of advice in the Announcement post.

@philm

sudo pip freeze > pip_list_sudo.txt
pip freeze --user > pip_list_user.txt
/path/to/<venv_name>/python -m pip freeze -l > pip_list_<venv_name>.txt

@philm

After the upgrade, although they say to do something like
sudo pip install -r pip_list_sudo.txt,

[Daniel-I]

Yikes! do you feel it was the “reinstall” via sudo pip install -r pip_list_sudo.txt that broke things for you @Kevin77 ?

In any case, you might want to add a warning in the stable thread for others.

[cscs]

I would expect it to break things. Should be fixed now.

[cscs]

Its the only thing that could. And it has an almost certainty that it would.
Sorry to you @Kevin77

[Kevin77]

@Daniel-I

Yikes! do you feel it was the “reinstall” via sudo pip install -r pip_list_sudo.txt that broke things for you @Kevin77 ?

Yes. Ha ha. No big deal. Timeshift to the rescue.

@cscs

Sorry to you @Kevin77

Not a problem. All is well and good now

[Daniel-I]

Okay… so just to circle back to my original question/resolution… I won’t miss anything listed (pre-update) via python 3.9 pip freeze --user > pip_list_user.txt so I can feel good about $ rm -rdv /home/<userid>/.local/lib/python3.9?

And although it’s odd one file was left in /usr/lib/python3.9… I can remedy that with $ sudo rm -rdv /urs/lib/python3.9?

[ishaan2479]

@Kevin77

Yes. Ha ha. No big deal. Timeshift to the rescue.

that’s a really positive attitude i switched to testing branch 2 days ago so i had got this update but came to know about this pip stuff just know (missed the testing announcement earlier). i have no aur package that uses python. the system python packages were automatically upgraded and i manually upgraded the ones installed via pip. important thing is nothing is broken (yet)

[cscs]

All Stable and Testing Announcement threads with similar entries have been edited.

[stephane]

the question about ‘pip’ results was raised in testing

[Testing Update] 2021-12-28 - Systemd 250, AMDVlk 2021.Q4.3, LibGLVND 1.4.0

Yep, this happened exactly in this post even though it’s in this post lol, some people don’t read Edit: Past post, but still applies

[Daniel-I]

I thought I’d share what I’ve written down about my understanding of the whole python update/readiness process… but as I wasn’t able to figure out how to upload an ODT/PDF, maybe a screenshot would be appreciated.

Major Python Version Updates.odt

Note: Click the image once to open in an “image box”, then optionally click once more to maximize for optimal reading

[j77h]

@philm

After the upgrade, although they say to do something like
sudo pip install -r pip_list_sudo.txt,

The next line said words to the effect of “do not do it”.

[Daniel-I]

I’m not sure who “they” is, but the capture of what was written was a “before you do” consideration, not “do not do” warning…

@philm

After the upgrade, although they say to do something like
sudo pip install -r pip_list_sudo.txt,
look at the lists and consider whether you still want them all.

I agree it was definitely a statement to make one stop and think… but I wouldn’t like it to a cautionary tale about dangers ahead.

Thankfully I didn’t execute that command and now that the wiki posts have been updated others shouldn’t be tempted to run it going forward either

[cscs]

Thats not really what that line says. Its closer to saying ‘do this but with caution’.
Thing is … there is no real case for doing that … it only produces false results and then 'sudo pip install’s everything on that list - which would be bad for any pip package on any day, but even worse now that you would be attempting to overwrite repo packages as well.
sudo pip may make sense only in the case that someone properly set up a virtual environment.
But that wasnt mentioned anywhere in those instructions so I have to conclude it was simply written out of a fundamental misunderstanding.
Ultimately those notes should not have been added in that form at all.

[j77h]

not sure who “they” is

The pip documentation, for one.

As for “the next line”, I should have said “the next few lines”.
Depends when/where it was copied from my original version;
I did strengthen the “don’t do it” message later on, IIRC.

I wrote it in the 2021-12-28 testing announcement, and expected problems to be picked up there.
Was working it out for myself, as I thought I might have installed some in system python by mistake.

Later in the process I found that pip freeze picks up packages that pip did not install, as you can read here

Years ago I followed the advice on pip.pypa.io and many other places, simply “pip install package”, which must be intended for Windows users, as it fails in Linux with a permission error. When it failed I simply used sudo, didn’t know it was dangerous. And last week I was forgetting that there had been malware on PyPI.