File conflict errors related to Python 3.12

Support Software & Applications
1

In trying to apply the system update, I get a number of file conflict errors related to Python 3.12 in /usr/lib/python3.12/site-packages, the following subdirectories:

  • idna-3.7.dist-info
  • certifi-2024.7.4.dist-info
  • scripts
  • google/protobuf
  • protobuf-5.27.2.dist-info

I’m doing my best to research and sort this out. I take it the basic problem is that I already have Python 3.12.4 and that needs to be uninstalled before the system update can be applied. But I haven’t been able to figure out how to proceed in a way that removes enough to get the update applied, but doesn’t remove too much so as to cause damage.

Is this [HowTo] the one I should be following to prepare my setup for the update? I’m not confident it is, and if it is, exactly how to proceed.

I was able to recover from a previous system update debacle, with a lot of help over many days. I’m still on the fence about whether Manjaro is right for me. It’s just not going to serve if I don’t soon move through the learning curve so that updates are no longer a baffling, terrifying, error-prone mystery.

Thanks for any help or advice!

2

Either you’re in an unsupported, partial upgrade state or you have AUR packages that need rebuilds.

Please make sure your mirrors are up to date and only update repo packages. AUR packages can be dealt with afterward.

FYI, I moved your reply to a new topic for further troubleshooting.

1 Like
3

Python 3.12 is system python - that cannot be removed.

 $ python --version
Python 3.12.4

It is more likely that you have packages depending on older versions of python.

This will be true if you have content in the folder (the example is from a current Manjaro system)

 $ ls /usr/lib/python3.11/site-packages/
ls: cannot access '/usr/lib/python3.11/site-packages/': No such file or directory

And any content in that folder will also hint which packages you need to rebuild.

It is relatively easy to achieve a unterrifying and painless update.

  • list all custom package(s)
  • remove them
  • sync the system
  • rebuild custom packatge(s)

To list the custom (foreign) package(s)

pacman -Qqem

To ease the process you way want to pipe the list to a file in your home

pacman -Qqem > ~/my_custom_pkgs.txt

Then feed the list to the package manager to remove the packages

pamac remove $(cat ~/my_custom_pkgs.txt)

Sync your system

sudo pacman -Syu

After successful sync - rebuild the custom package list

pamac build $(cat ~/my_custom_pkgs.txt)

For a more insider knowledge see

3 Likes
4

Follow the advice given already.

Just to expand on it a little:

  1. It’s hard to imagine how you could have installed Python from the AUR and not be aware that you did it. Probably you didn’t.

  2. If you did install Python from the AUR:
    a. the conflict should have come up in May, when the system switched to Python 3.12
    b. this conflict could be related to having previously installed Python 3.12 from the AUR, only if this is the first time you’ve updated since before the system switched to Python 3.12

Whatever OS one chooses, one must study it thoroughly before using it for real work (that one’s business or job depends on).

Even with Windows or Mac you need to know quite a lot to use it properly,
and with other operating systems, you need to know them more intimately.

When I dropped Windows, I used Ubuntu for 6 years before switching to Manjaro.
Since then I’ve used Manjaro continuously for 7 years without re-installing.
(Have migrated it to new hardware twice during that time, without any problem.)

Even so, I find the frequent updating a little stressful. It’s bearable only because I always prepare for rollback: I make btrfs snapshots, and keep a printed summary of how to roll back in case Manjaro can’t boot after an update. (There are apps to make this easier, but they don’t suit my setup.)

5

Thanks, I really appreciate the help.

I was way off-base in my assessment of the python-related errors. It’s too embarrassing and probably not productive to explore the details of my faulty reasoning. I clearly have a long, long way to go in understanding basic topics like the relation of Manjaro to AUR and the use of system commands.

I used the method described to remove all custom packages, so that pacman -Qqem output is now null.

However, the system update using pacman -Syu still fails in exactly the same way, with a number of file conflicts in usr/lib/python3.12/site-packages. The subfolders implicated contain the following:

  • Distribution information (...dist-info) with text files like LICENSE, METADATA, RECORD, WHEEL
  • Protocol buffer data (protobuf...) with subfolders like __pycache__, testdata
  • Subfolder scripts containing a readme_gen.py script

Can these be deleted safely to clear the way for the system sync?

6

I haven’t had to work in Python for a few years. But doesn’t this happen when ever you install Python packages globally?

So you have to pip --user to install, which goes to ~/.local/usr/lib/python-*

I kind of remember packages that only work globally, I could not use the distribution’s Python. (Or else this happens.)

But surely someone that has done more with Python would know?

1 Like
7

I am not the most python competent person, but as far as i know there a two causes for such a mess: either you have AUR packages (the solution is to temporarily uninstall as shown above), or you have at some point used sudo pip install … which you should never do in manjaro, you can use venv or pipx to install locally. If that is the case, you will have to pass “overwrite” to the pacman
sudo pacman -Syu --overwrite "*"
Note that this IS dangerous since it skips all checks - inform yourself. The usual disclaimer: I am not responsible if your PC explodes :slight_smile:

1 Like
8

That’s what --overwrite does, no need to remove them first.

You can reduce the danger by doing
sudo pacman -Syu --overwrite "usr/lib/python3.12/site-packages/*"
so it will overwrite only in that dir.

(I wonder if pacman uses ** like bash does,
and whether pacman needs ** instead of * to overwrite in sub-dirs… ?)

or in any Linux distro.

Packages installed with pip have a “dist-info/INSTALLER” file,
which contains only the word “pip”.
(eg: ~/.local/lib/python3.12/site-packages/rst2html5-2.0.1.dist-info/INSTALLER)

1 Like
9

Thank you, @j77h! And thank you, @Teo, who first suggested using the --overwrite parameter. In the interest of introducing at least a modicum of caution, I used the form with the subfolder qualification. I’m now working through rebuilding packages and reinstalling applications.

I don’t understand how I ended up in this upgrade impasse. But my system hasn’t exploded yet and is now upgraded and running.

10

When I was coding deeper with Python for work and massive frameworks and packages, I could never do this. I was on year 12 of using Gentoo, or something like that, and I had to hack my way through this. But the tools around Python have grown since I was using to it to that level.

With every Python update, you will have to reinstall all the global packages again, even though they are 99% already there. And if you take on a project that requires different versioned packages, different Python versions, completely different interpreters (like pypy, cpython, etc.), then you are screwed.

But now there’s a better solution for this, it is more complicated. But once set up, you can take on many Python projects, and update like normal with much less risk.

I do not have experience with this personally, but now from what I can tell, this is the best way to do that:

Python 3.3 and above just comes with venv support for simple virtual environment support for managing libraries separately.

But this gives you way more control:
https://virtualenv.pypa.io/en/latest/

Which is also an official Manjaro package.

11

Just as a quick note: virtualenvs normally still depend on the host Python, so this would not be a solution. If you want a solution to have Python environments which are independent of the system install, I’d recommend looking at pyenv, which does just this (by downloading/compiling a separate build for you).

12

I gave the most simple way, not to mention it comes with Python. Then a much more configurable way. But you’re right, I didn’t mention pyenv. Which I would put in the middle of those. But I could of even mentioned three other ways too!

But the way you took it out of context, you make it sound like I made it the best and only solution, when I was in fact, I was discouraging it.

13

Apologies, but I don’t believe I took anything out of context. Especially when you actually said “this is the best way to do that”. Depending on the distribution provided version of Python is generally considered not best practice these days, as upgrades from the distribution can break things rather easily; just recently, I had a bunch of servers taken out because I had made virtualenvs from the system Python. And this is a particular issue on a rolling release like Manjaro.

Hence I pointed out that best practice is to not use the system provided Python if possible, and use your own install, as this avoids issues with system upgrades. However, virtualenvs are always going to be preferable to installing anything onto the system install of Python. Just don’t expect anything based on a system install to survive for more than a year, as once the new version of Python rolls around things will break.

14

I knew and said my knowledge was dated, I only use Python for quick things here and there now. I dealt with them (a decade ago) manually, which did not break system updates, and allowed for any Python interpreter and libraries. But now, you can use tools to do this for you.

I didn’t mean to say best. I even had pyenv as an option in my draft, but didn’t want to overload this person with options. Just any of these other options were better I meant.

I was actually pleading for help 3 days ago!

I was only confident in that the posted solution should not be best practice, or any practice. It backs you into a corner, and a greater potential to cause issues. We are agreed on this, correct?

15

Well, no. It seems that the problem was caused by not following best practice and installing files into the system Python. Hence, given the state that the system was in, the posted solution is indeed correct.

You provided some information on how one might avoid the problem in the first place, which I felt had some perhaps unintentional hyperbole and therefore I added some extra info on some pitfalls with what you suggested. However, as the original question was on how to recover an already broken Python install, information on avoiding triggering the problem would not help resolve it.

16

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.