Python 3.9 to 3.10 validation and cleanup

the question about ‘pip’ results was raised in testing

I thought I’d share what I’ve written down about my understanding of the whole python update/readiness process… but as I wasn’t able to figure out how to upload an ODT/PDF, maybe a screenshot would be appreciated.

Major Python Version Updates.odt

Note: Click the image once to open in an “image box”, then optionally click once more to maximize for optimal reading

1 Like

The next line said words to the effect of “do not do it”.

I’m not sure who “they” is, but the capture of what was written was a “before you do” consideration, not “do not do” warning…

I agree it was definitely a statement to make one stop and think… but I wouldn’t like it to a cautionary tale about dangers ahead.

Thankfully I didn’t execute that command :wink: and now that the wiki posts have been updated others shouldn’t be tempted to run it going forward either

Thats not really what that line says. Its closer to saying ‘do this but with caution’.
Thing is … there is no real case for doing that … it only produces false results and then 'sudo pip install’s everything on that list - which would be bad for any pip package on any day, but even worse now that you would be attempting to overwrite repo packages as well.
sudo pip may make sense only in the case that someone properly set up a virtual environment.
But that wasnt mentioned anywhere in those instructions so I have to conclude it was simply written out of a fundamental misunderstanding.
Ultimately those notes should not have been added in that form at all.

not sure who “they” is

The pip documentation, for one.

As for “the next line”, I should have said “the next few lines”.
Depends when/where it was copied from my original version;
I did strengthen the “don’t do it” message later on, IIRC.

I wrote it in the 2021-12-28 testing announcement, and expected problems to be picked up there.
Was working it out for myself, as I thought I might have installed some in system python by mistake.

Later in the process I found that pip freeze picks up packages that pip did not install, as you can read here

Years ago I followed the advice on and many other places, simply “pip install package”, which must be intended for Windows users, as it fails in Linux with a permission error. When it failed I simply used sudo, didn’t know it was dangerous. And last week I was forgetting that there had been malware on PyPI.

And here I leave this…


I’m glad you were chiming in on the test branches @j77h. I’ve been watching all the branch discussions so I’d have a good sense of what was coming down the pike for Stable… and I started paying extra attention once the [Unstable Update] 2021-12-13 - Python 3.10 Rebuilds post was made because I had a feeling a significant change like that deserved more attention and forethought than just clicking “Apply” in the pamac GUI’s upgrade tab.

I made it clear, eventually, not to do “sudo pip install”,
and left “sudo pip freeze” there, as it only makes a list.
But for speedy people who don’t read carefully, it wasn’t clear enough I guess.

I was surprised when (apparently) Phil moved my wiki post entry to the top, and emphasized it (in testing).

Now I’m surprised that complaints didn’t happen until it got into stable – no-one noticed potential problems in testing, although I wrote posts about it…

I only noticed because of this post (and then retroactively changed the testing posts back through early december) … I dont usually pay attention to release/update announcements, especially on Stable or Testing … and on Unstable we dont have the ‘common problems/fixes’ wiki thing.

1 Like

Off topic, and only view the image if there is a good sense of humour and understanding that it’s meant to bring a smile in times of seemingly thankless efforts.

At my old job, I started getting sick of the proverbial “attaboy’s” that managers seemed to think we lived off of… so I started bringing some reality and humour to it by sharing the following image with my colleagues after we’d solve a tough problem… even in some cases where what we’d accomplished wasn’t really noticed by management.



1 Like

It really exists, and some of the actual code is revealed here.
Scroll down to “noblesse payload #2”, and see how easy it is
to steal passwords and credit card numbers saved in Chrome !

Still baffles me that people use these features … especially for sensitive things.
Its sorta funny when a friend can easily log into your email account because you left your machine on … its worse when it is easy to dupe users … its even worse when it can be accomplished by a drive-by (simply visit a page that somehow has malicious code injected) without extra protections … which are of course the same users that store things that way, dont configure security, and wont notice a problem until its too late.

But we should be fair here - when it comes to installing packages … especially 3rd party ones … the user should consider what it is they are installing…
(If you cant decipher the install/application or trust the source … the only safe thing is dont install)

1 Like

More or less what I learned with AUR packages… at least look at the PKGBUILD file to try get a sense of what it’s doing before installing. For example, if it looks like it’s going somewhere weird/unexpected to download some files, that’s a sign to abort.

Even more baffling is that “responsible” corporations offer them.

Have never allowed a browser to save anything for me, out of sheer precaution,
and didn’t know until now how easily they could have been stolen.
(I use KeePassXC with a strong password; hope it’s safe enough.)

Those exploits don’t need root access,
so even ‘pip install --user’ must be used with great care.

1 Like

Screenshot text cannot be indexed for searching. Copy and paste instead.

Note: If you find it useful to install python components via pip, it’s best to keep them in “user-space” (Home directory) by installing them via $ pip install --user <component> … this will keep the core python install happier/cleaner, and make it easier to keep track of what you’ve personally installed.

When Python gets a major update… like python 3.9 (old) => 3.10 (new)… there’s a few things to do:

  • Before Upgrading
    • pip Installed
      • run $ pip freeze --user > pip_list_user.txt to get a list of all “user” installed python components for the current (pre-upgrade) python (which should be similar to looking inside /home/<userid>/.local/lib/python3.9) (old)
      • Sometimes installing one component will install dependencies as well… to see these dependencies it can be useful to run $ pip show <component>. This could also help to shorten the “re-install” list inside pip_list_user.txt if the inter-dependencies are known.
    • Repo Installed
      • running $ pacman -Qoq /usr/lib/python3.9 (old) will list many existing “system” (and AUR) packages that pacman|pamac (the upgrade) “should” re-install for the upgraded python version (i.e. python 3.9 => 3.10)
  • After Upgrading
    • pip Installed
      • Option 1: If you want what’s in the pip_list_user.txt (old python 3.9) file to be installed under the new python 3.10, there are a few choices:
        • Reinstall them manually via $ pip install --user <component> (paying attention to dependencies to re-install with the least steps)
        • Look for an AUR/repo controlled component (removes reliance on pip)
      • Option 2: If you don’t want what’s in the pip_list_user.txt (old python 3.9) file to be installed under the new python 3.10, simply perform to the next “remove” step
      • Either path/option chosen above, you’ll also want to make sure to clear out the old 3.9user-space” via $ rm -rdv /home/<userid>/.local/lib/python3.9 (old)
    • Repo Installed
      • running $ pacman -Qoq /usr/lib/python3.9 (old) should be empty… if it’s not, you may need to run the following command to update them (i.e. rebuilding AUR packages)…$ pamac build $(pacman -Qoq /usr/lib/python3.9) --no-confirm
    • It’s a good idea to check /user/lib/python3.9 (old) to make sure it was completely removed. Had it once where one “pycache” file prevented a directory tree from being removed… solved via $ sudo rm -rdv /usr/lib/python3.9 (old)
1 Like

In response to misunderstandings of some users who know nothing about python or pip,
I have updated the wiki post in the stable update announcement, to the following.
(It’s rather long for the wiki post, maybe I’ll put it in another post and link to it…)

Original content removed because it has been greatly improved in the current version, which is here.

The way "python3.9 (old)" appears, I thought you had renamed that dir,
but seeing the backticks in the source I realize you didn’t.

It’s not necessary to remove the old ‘site-packages’ directories, they don’t do any harm.
After you’re sure you’ve got all the info you need out of them,
you can delete them, but the reason is only to save space.

Good point about seeing dependencies with ‘pip show’.

Manjaro/Arch could use a complete wiki article about this subject.
Took me a lot of hours to find out things like “‘pip freeze’ lists packages it didn’t install”,
as it’s not mentioned in the pip documentation (or it’s buried too deep).

I got familiar with Python long before I found Manjaro,
and I’m surprised at how many people know nothing about it.
(Same goes for SQL and database structure, which I knew before I learned Python.)

Don’t know how I got the original idea of using “pip freeze > file”.
The file is intended for later use with “pip install --requirement <file>

Thinking of changing it to ‘pip list’, as we now suggest only using it to look at.
The output of list is easier to read:

[jh@SSD2 220104 11:14:33 zp0]$ pip list --user
Package            Version
------------------ -------
commonmark         0.9.1
filetype           1.0.7
font-line          3.1.1
fonttools          4.11.0
gdown              3.12.2
Genshi             0.7.5
html2docx          1.3.1
mpv                0.1
pymdown-extensions 8.1.1
PySocks            1.7.1
python-docx        0.8.10
python-uinput      0.11.2
rst2html5          2.0
tqdm               4.57.0
xxhash             2.0.2
[jh@SSD2 220104 11:14:45 zp0]$ 
[jh@SSD2 220104 11:14:50 zp0]$ pip freeze --user