Managing Python Packages on Manjaro: Error with Externally Managed Environment

This is now discouraged.

Use a virtualenv, or pipx maybe?

I had the same problem
This command fixed it by downgrading pip.

pip install pip==22.3.1 --break-system-packages

3 Likes

This is not a solution though.
So pip is deprecated now?
And should be using pipx?
What If I have +100 pip packages?, to my knowledge pipx doesn’t support -r requirement.txt.

pip is not deprecated … you just shouldnt be using it outside of a virtual environment
(or using sudo, etc)

What to do if you have many pip packages?
You can’t update them.
To my knowledge you need to uninstall all packages and then re-install it one by one.

Yes you can.

…so you must be referencing a scenario where you installed them using sudo pip.
Dont do that.

No I couldn’t since I get the Externally Managed Environment Error.
And no I didn’t use sudo pip when installing it.
I tried upgrading using pip I get the error, updating with pipupgrade is the same, and moving to pipx create conflict.
So what do you suggest?

It really depends what type of packages you plan on installing.

When installing packages using pip, it is recommended to use a virtual environment to prevent conflicts with system packages in /usr. Alternatively, pip install --user can be used to install packages into the user scheme instead of /usr. Other tools including pipx, poetry and Conda integrate environment management into their workflows.

See the Python Packaging User Guide for the official best practices for package management.

Historically, easy_install (part of python-setuptools) was used to install packages distributed as Eggs. easy_install and Eggs have been replaced with pip and Wheels. See pip vs easy_install and Wheel vs Egg for more information.

Note: There are also tools integrating pip with pacman by automatically generating PKGBUILDs for specified PyPI packages: see Creating packages#PKGBUILD generators.

https://wiki.archlinux.org/title/Python#Package_management

https://wiki.archlinux.org/title/Python/Virtual_environment

Similar to this user, I’ve just found that pip install --user doesn’t work for me anymore.

I hadn’t tried updating packages since last month’s big python 3.11 update – where I followed the provided guide as best I could – and now I get the PEP 668 message when I do pip install --user yt-dlp.

I had done pretty much all of my python package installations with the --user flag; for more specialized stuff I used miniconda3 to create environments.

[EDIT: Fixed broken link, typo]

I guess you skipped some parts then.

You can still install with ‘–user’ if you add ‘–break-system-packages’.
This simply lets ‘–user’ works as before.
The scary wording is only to warn you that version conflicts are possible.

Once more though - this is just a change making it even more apparent how they should have been handled in the first place. With virtual environments. It was slightly more permissive before (now sudo pip isnt even possible it seems) and you can regain that same --user functionality with --break-system-packages … its just more obvious now that you shouldnt and you have to add the extra flag.

That link is dead now.

The guide that I wrote is here,
and I added info about the pip changes here.

I guess I should make a shorter version of that guide,
as it’s a long read to find everything you need to know in there.

Anaconda or miniconda is a good way to do it,
as it’s both a package manager and a virtual-environment manager.
But for most people even miniconda is too big an installation
if they have only a few pip packages.

1 Like

An important thing to realize is that it cannot actually break system packages,
because the system completely ignores your user space.

What happens is that some of your own packages may fail to work
if they import modules from system space that don’t have the right version.

1 Like

I see. I did this update on 2023-06-08 and this info looks like it came out on 2023-06-24, after what I thought was the “big yearly python update”. I follow the [Stable Updates] posts via RSS, I feel like this should have been mentioned up front rather than buried in the guts somewhere.

As for venv usage: I appreciate the need to avoid touching the system python packages, but its another thing to remember on top of the long commands, and then having to source/activate the environment to use these packages.

This is not a Manjaro problem, but it would be nice if the default arrangement was that all users automatically get a default venv to which packages are installed via pip install --user (or even just pip install), with sudo pip still banned. And the user python installation automatically adds source <venv>/bin/activate to .bashrc/.zshrc, so it’s just hidden.

I’m sure there are long and heated discussions about this, and why this idea would instantly break everything on the system, which is why the gods developers in their wisdom deemed this the best of all possible worlds.

We did this some years ago and people revolted.
That doesnt really mean anything … but it happened.
(the dev at the time eventually became flustered with the attacks, reverted the changes … and it was never spoken of again except in circumstances like this)

The main problem I have is that now I can’t combine global packages with the packages in the virtual environment. There are some packages that I prefer to install globally since I use them often. But now I get the error:

Getting a Module Error When Running Pytest Even Though the Module is Installed in the Current Virtual Environment

So I have to install the same module on each virtual environment for each python project which is very bothersome.

Please read …

Quite possible … please see the bigger picture which is why you need to sandbox your environment.

You need to sandbox the environment so you do not interfere with the stability of the system.

Sandboxing effectively means you cannot combine the inside with the outside - similar to a container.

No, it’s not. This way, you guarantee that you have the exactly required version.

Additionally, you probably have a requirements.txt file to manage your project’s requirements. You have to list pytest there, otherwise, who will know that pytest is needed because you installed it globally.

Modularization is key!

I see, well I was using --user because that what I was thought I should be use since I always thought that’s the way.
This whole thing have made an issue for me because I had two versions of urllib3, one from pip and the other from the repo.
I didn’t realize it until like 3 weeks.
So, now the recommended way it to install pip packages one by one using pipx?

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.