Python 3.9 to 3.10 validation and cleanup

Today’s new stable update went well, and as this is my first time experiencing a majaor python update under any GNU/Linux distro, I’d like to make sure all is well within/under python.

I took the following command results (post update) as a good sign:

$ pacman -Qoq /usr/lib/python3.9
error: No package owns /usr/lib/python3.9

… I take it to mean that nothing is left left relying on python 3.9 (as far as what was installed via pamac/pacman) post update.

Also, the only thing left in pamac’s foreign/AUR list that listed python as a dependency was zenstates-git… and as I never use it, I’ve uninstalled it.

I’m not 100% sure how to interpret the pip freeze commands before the update though… and I also don’t recall installing much if anything via pip (will ensure I document pip installs in the future).

Python 3.9 sudo pip list

apparmor==3.0.3
appdirs==1.4.4
application-utility==1.3.2
btrfsutil==5.15.1
CacheControl==0.12.6
ceph==1.0.0
ceph-volume==1.0.0
cephfs==2.0.0
cephfs-shell==0.0.1
certifi==2021.10.8
cffi==1.15.0
chardet==4.0.0
colorama==0.4.4
contextlib2==0.6.0.post1
cryptography==35.0.0
cupshelpers==1.0
distlib==0.3.3
distro==1.6.0
dnspython==2.1.0
docopt==0.6.2
entrypoints==0.3
gufw==21.4.0
html5lib==1.1
idna==3.3
importlib-metadata==4.8.1
iotop==0.6
jeepney==0.7.1
keyring==23.0.1
keyutils==0.6
lensfun==0.3.95
LibAppArmor==3.0.3
louis==3.20.0
lxml==4.6.4
Markdown==3.3.6
meld==3.20.4
more-itertools==8.10.0
msgpack==1.0.2
mutagen==1.45.1
netsnmp-python==1.0a1
npyscreen==4.10.5
opendht==2.3.0
ordered-set==4.0.2
packaging==20.9
pacman-mirrors==4.23.2
pep517==0.12.0
Pillow==8.4.0
ply==3.11
progress==1.6
pwquality==1.4.4
pycairo==1.20.1
pycparser==2.21
pycryptodome==3.11.0
pycryptodomex==3.10.1
pycups==2.0.1
pycurl==7.44.1
PyGObject==3.42.0
pyOpenSSL==21.0.0
pyparsing==2.4.7
PyQt5==5.15.6
PyQt5-sip==12.9.0
pysmbc==1.0.23
PyYAML==5.4.1
rados==2.0.0
rbd==2.0.0
reportlab==3.6.2
requests==2.26.0
resolvelib==0.5.5
retrying==1.3.3
rgw==2.0.0
SecretStorage==3.3.1
six==1.16.0
speedtest-cli==2.1.3
TBB==0.2
team==1.0
toml==0.10.2
tomli==1.2.2
udiskie==2.3.3
ufw==0.36
urllib3==1.26.7
webencodings==0.5.1
websockets==9.1
youtube-dl==2021.6.6
yt-dlp==2021.11.10.1
zipp==3.6.0

Python 3.9 user pip list
aiosmb==0.2.50
aiowinreg==0.0.7
asn1crypto==1.4.0
asysocks==0.1.2
minikerberos==0.2.14
msldap==0.3.30
oscrypto==1.2.1
portio==0.5
prompt-toolkit==3.0.24
tqdm==4.62.3
wcwidth==0.2.5
winacl==0.1.2
winsspi==0.0.9

Post update… my “user pip list” is empty… and (according to meld) my “sudo pip list” has the same “named” entries, although as expected some have had version updates.

So what do I do with my “3.9 user pip list”? portio sounds like something I might have pursued, but really I’m pretty sure I didn’t install these… and not really knowing what they are or what they were used for, I don’t think I’ll miss them. In this case, can I just delete my HOME folder “leftovers”?

I also found some python 3.9 leftovers in /usr/lib/… can I delete this too?
Screenshot_20220102_111813

1 Like

Yup. Nothing like a repo package anyway…
But that should have meant ythe directory was removed.
(on my system I have never removed an old python directory, like 2.7 etc)
So … it is possible something is relying on it …

you didnt …

Do that, for anything but list, right?
Do not use sudo with pip :pray:

So you have not removed 3.9 yet?
In any case … you can freely remove (through pip and/or manually deleting directories) old python version stuff or anything you dont want.

So…

yes.

Now this is odd. I am inclined to say yes, as nothing owns it … but I still wonder why it exists.
(as noted above these directories usually are removed along with the python package)

1 Like

Ah, my mistake… I should have included the full commands I ran from the stable “wiki” post…

… so one of my commands was run with sudo as recommended

  • $ sudo pip freeze > pip_list_sudo.txt
  • $ pip freeze --user > pip_list_user.txt

That’s good news, I was hoping that was the case!

The only time I’ve experienced something like this (under Windows) was when a file was locked or in use or not part of the uninstall script… and usually folders refuse to delete so long as there are any files inside. But as you can see from the folder that the vast majority of the /usr/lib/python3.9 folder was removed… with the exception of one file that prevented its parent folders from being removed.

Honestly its just kinda silly to do that.
First because, again, users should not invoke sudo with pip. So there should be no ‘sudo pip’ packages.
But second because … that list will be largely useless because of listing all sorts of system packages that rely on python but are from the repos and not pip (example: ufw,gufw) …

The line afterwards …

Is downright dangerous.

I always appreciate your feedback @cscs . Being a relatively new to gnu/linux user still… I always make sure to post my questions if something looks questionable. Whether that be a command or the end result of a command.

But as the commands I ran were just generating lists, I wasn’t too concerned about them as it was my intention to follow one sage piece of advise…

… i.e. no plan to run sudo pip install -r pip_list_sudo.txt in my case.

The instruction there really should be avoided. Only the ‘user’ list should be needed or used.
And then only if you want those python packages.

I am currently trying to get the announcement ‘wiki’ section revised…

2 Likes

Broke my system following this piece of advice in the Announcement post.

Yikes! do you feel it was the “reinstall” via sudo pip install -r pip_list_sudo.txt that broke things for you @Kevin77 ?

In any case, you might want to add a warning in the stable thread for others.

I would expect it to break things.:persevere: Should be fixed now.

1 Like

Its the only thing that could. And it has an almost certainty that it would.
Sorry to you @Kevin77

1 Like

Yes. Ha ha. No big deal. Timeshift to the rescue.

Not a problem. All is well and good now :+1:

1 Like

Okay… so just to circle back to my original question/resolution… I won’t miss anything listed (pre-update) via python 3.9 pip freeze --user > pip_list_user.txt so I can feel good about $ rm -rdv /home/<userid>/.local/lib/python3.9?

And although it’s odd one file was left in /usr/lib/python3.9… I can remedy that with $ sudo rm -rdv /urs/lib/python3.9?

that’s a really positive attitude :beers: i switched to testing branch 2 days ago so i had got this update but came to know about this pip stuff just know (missed the testing announcement earlier). i have no aur package that uses python. the system python packages were automatically upgraded and i manually upgraded the ones installed via pip. important thing is nothing is broken (yet) :wink:

1 Like

All Stable and Testing Announcement threads with similar entries have been edited. :safety_vest:

2 Likes

the question about ‘pip’ results was raised in testing

I thought I’d share what I’ve written down about my understanding of the whole python update/readiness process… but as I wasn’t able to figure out how to upload an ODT/PDF, maybe a screenshot would be appreciated.

Major Python Version Updates.odt

Note: Click the image once to open in an “image box”, then optionally click once more to maximize for optimal reading

1 Like

The next line said words to the effect of “do not do it”.

I’m not sure who “they” is, but the capture of what was written was a “before you do” consideration, not “do not do” warning…

I agree it was definitely a statement to make one stop and think… but I wouldn’t like it to a cautionary tale about dangers ahead.

Thankfully I didn’t execute that command :wink: and now that the wiki posts have been updated others shouldn’t be tempted to run it going forward either

Thats not really what that line says. Its closer to saying ‘do this but with caution’.
Thing is … there is no real case for doing that … it only produces false results and then 'sudo pip install’s everything on that list - which would be bad for any pip package on any day, but even worse now that you would be attempting to overwrite repo packages as well.
sudo pip may make sense only in the case that someone properly set up a virtual environment.
But that wasnt mentioned anywhere in those instructions so I have to conclude it was simply written out of a fundamental misunderstanding.
Ultimately those notes should not have been added in that form at all.

not sure who “they” is

The pip documentation, for one.

As for “the next line”, I should have said “the next few lines”.
Depends when/where it was copied from my original version;
I did strengthen the “don’t do it” message later on, IIRC.

I wrote it in the 2021-12-28 testing announcement, and expected problems to be picked up there.
Was working it out for myself, as I thought I might have installed some in system python by mistake.

Later in the process I found that pip freeze picks up packages that pip did not install, as you can read here

Years ago I followed the advice on pip.pypa.io and many other places, simply “pip install package”, which must be intended for Windows users, as it fails in Linux with a permission error. When it failed I simply used sudo, didn’t know it was dangerous. And last week I was forgetting that there had been malware on PyPI.