Python 3.9 to 3.10 validation and cleanup

not sure who “they” is

The pip documentation, for one.

As for “the next line”, I should have said “the next few lines”.
Depends when/where it was copied from my original version;
I did strengthen the “don’t do it” message later on, IIRC.

I wrote it in the 2021-12-28 testing announcement, and expected problems to be picked up there.
Was working it out for myself, as I thought I might have installed some in system python by mistake.

Later in the process I found that pip freeze picks up packages that pip did not install, as you can read here

Years ago I followed the advice on and many other places, simply “pip install package”, which must be intended for Windows users, as it fails in Linux with a permission error. When it failed I simply used sudo, didn’t know it was dangerous. And last week I was forgetting that there had been malware on PyPI.

And here I leave this…


I’m glad you were chiming in on the test branches @j77h. I’ve been watching all the branch discussions so I’d have a good sense of what was coming down the pike for Stable… and I started paying extra attention once the [Unstable Update] 2021-12-13 - Python 3.10 Rebuilds post was made because I had a feeling a significant change like that deserved more attention and forethought than just clicking “Apply” in the pamac GUI’s upgrade tab.

I made it clear, eventually, not to do “sudo pip install”,
and left “sudo pip freeze” there, as it only makes a list.
But for speedy people who don’t read carefully, it wasn’t clear enough I guess.

I was surprised when (apparently) Phil moved my wiki post entry to the top, and emphasized it (in testing).

Now I’m surprised that complaints didn’t happen until it got into stable – no-one noticed potential problems in testing, although I wrote posts about it…

I only noticed because of this post (and then retroactively changed the testing posts back through early december) … I dont usually pay attention to release/update announcements, especially on Stable or Testing … and on Unstable we dont have the ‘common problems/fixes’ wiki thing.

1 Like

Off topic, and only view the image if there is a good sense of humour and understanding that it’s meant to bring a smile in times of seemingly thankless efforts.

At my old job, I started getting sick of the proverbial “attaboy’s” that managers seemed to think we lived off of… so I started bringing some reality and humour to it by sharing the following image with my colleagues after we’d solve a tough problem… even in some cases where what we’d accomplished wasn’t really noticed by management.



1 Like

It really exists, and some of the actual code is revealed here.
Scroll down to “noblesse payload #2”, and see how easy it is
to steal passwords and credit card numbers saved in Chrome !

Still baffles me that people use these features … especially for sensitive things.
Its sorta funny when a friend can easily log into your email account because you left your machine on … its worse when it is easy to dupe users … its even worse when it can be accomplished by a drive-by (simply visit a page that somehow has malicious code injected) without extra protections … which are of course the same users that store things that way, dont configure security, and wont notice a problem until its too late.

But we should be fair here - when it comes to installing packages … especially 3rd party ones … the user should consider what it is they are installing…
(If you cant decipher the install/application or trust the source … the only safe thing is dont install)

1 Like

More or less what I learned with AUR packages… at least look at the PKGBUILD file to try get a sense of what it’s doing before installing. For example, if it looks like it’s going somewhere weird/unexpected to download some files, that’s a sign to abort.

Even more baffling is that “responsible” corporations offer them.

Have never allowed a browser to save anything for me, out of sheer precaution,
and didn’t know until now how easily they could have been stolen.
(I use KeePassXC with a strong password; hope it’s safe enough.)

Those exploits don’t need root access,
so even ‘pip install --user’ must be used with great care.

1 Like

Screenshot text cannot be indexed for searching. Copy and paste instead.

Note: If you find it useful to install python components via pip, it’s best to keep them in “user-space” (Home directory) by installing them via $ pip install --user <component> … this will keep the core python install happier/cleaner, and make it easier to keep track of what you’ve personally installed.

When Python gets a major update… like python 3.9 (old) => 3.10 (new)… there’s a few things to do:

  • Before Upgrading
    • pip Installed
      • run $ pip freeze --user > pip_list_user.txt to get a list of all “user” installed python components for the current (pre-upgrade) python (which should be similar to looking inside /home/<userid>/.local/lib/python3.9) (old)
      • Sometimes installing one component will install dependencies as well… to see these dependencies it can be useful to run $ pip show <component>. This could also help to shorten the “re-install” list inside pip_list_user.txt if the inter-dependencies are known.
    • Repo Installed
      • running $ pacman -Qoq /usr/lib/python3.9 (old) will list many existing “system” (and AUR) packages that pacman|pamac (the upgrade) “should” re-install for the upgraded python version (i.e. python 3.9 => 3.10)
  • After Upgrading
    • pip Installed
      • Option 1: If you want what’s in the pip_list_user.txt (old python 3.9) file to be installed under the new python 3.10, there are a few choices:
        • Reinstall them manually via $ pip install --user <component> (paying attention to dependencies to re-install with the least steps)
        • Look for an AUR/repo controlled component (removes reliance on pip)
      • Option 2: If you don’t want what’s in the pip_list_user.txt (old python 3.9) file to be installed under the new python 3.10, simply perform to the next “remove” step
      • Either path/option chosen above, you’ll also want to make sure to clear out the old 3.9user-space” via $ rm -rdv /home/<userid>/.local/lib/python3.9 (old)
    • Repo Installed
      • running $ pacman -Qoq /usr/lib/python3.9 (old) should be empty… if it’s not, you may need to run the following command to update them (i.e. rebuilding AUR packages)…$ pamac build $(pacman -Qoq /usr/lib/python3.9) --no-confirm
    • It’s a good idea to check /user/lib/python3.9 (old) to make sure it was completely removed. Had it once where one “pycache” file prevented a directory tree from being removed… solved via $ sudo rm -rdv /usr/lib/python3.9 (old)
1 Like

In response to misunderstandings of some users who know nothing about python or pip,
I have updated the wiki post in the stable update announcement, to the following.
(It’s rather long for the wiki post, maybe I’ll put it in another post and link to it…)

Original content removed because it has been greatly improved in the current version, which is here.

The way "python3.9 (old)" appears, I thought you had renamed that dir,
but seeing the backticks in the source I realize you didn’t.

It’s not necessary to remove the old ‘site-packages’ directories, they don’t do any harm.
After you’re sure you’ve got all the info you need out of them,
you can delete them, but the reason is only to save space.

Good point about seeing dependencies with ‘pip show’.

Manjaro/Arch could use a complete wiki article about this subject.
Took me a lot of hours to find out things like “‘pip freeze’ lists packages it didn’t install”,
as it’s not mentioned in the pip documentation (or it’s buried too deep).

I got familiar with Python long before I found Manjaro,
and I’m surprised at how many people know nothing about it.
(Same goes for SQL and database structure, which I knew before I learned Python.)

Don’t know how I got the original idea of using “pip freeze > file”.
The file is intended for later use with “pip install --requirement <file>

Thinking of changing it to ‘pip list’, as we now suggest only using it to look at.
The output of list is easier to read:

[jh@SSD2 220104 11:14:33 zp0]$ pip list --user
Package            Version
------------------ -------
commonmark         0.9.1
filetype           1.0.7
font-line          3.1.1
fonttools          4.11.0
gdown              3.12.2
Genshi             0.7.5
html2docx          1.3.1
mpv                0.1
pymdown-extensions 8.1.1
PySocks            1.7.1
python-docx        0.8.10
python-uinput      0.11.2
rst2html5          2.0
tqdm               4.57.0
xxhash             2.0.2
[jh@SSD2 220104 11:14:45 zp0]$ 
[jh@SSD2 220104 11:14:50 zp0]$ pip freeze --user

The -r file action is fine … it doesnt need to be removed - though relating it to the ‘prefer repo pkgs’ and/or mentioning ‘list’ probably makes sense.

Not sure about that.
The requirements text contains version numbers, which are probably out of date by now.
It’s intended for replicating an installation exactly, but we’re upgrading, so it doesn’t seem to fit the situation.
People who need freeze will know about it already, I think.

Do you agree?

I’m making a new post that does not mention freeze at all (only list),
with the idea of linking to it from the update’s wiki-post…

Also, it won’t mention virtual environments,
as anyone who has gone through the process of learning how to set one up
will know how to take care of it, and for the others it will be meaningless cognitive overload.

For late-comers to this thread, please see the post here.

1 Like

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.