I recently encountered an error while trying to install a Python package using pip on my Manjaro system. The error message I received was:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S python-xyz', where xyz is the package you are trying to install.
If you wish to install a non-Arch-packaged Python package, create a virtual environment using 'python -m venv path/to/venv'.
Then use path/to/venv/bin/python and path/to/venv/bin/pip.
If you wish to install a non-Arch packaged Python application, it may be easiest to use 'pipx install xyz', which will manage a virtual environment for you. Make sure you have python-pipx installed via pacman.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
I have always installed Python packages using pip install -U package without any issues. However, this error message suggests using system-wide installation or creating a virtual environment using python -m venv for non-Arch-packaged Python packages. While I understand the benefits of these approaches, I prefer the simplicity of the previous method.
Is there a way to install Python packages as before without having to write additional commands like source venv/bin/activate for a virtual environment or --break-system-packages to override the error? I often rely on packages that are not available in the official or AUR package managers, so this change adds extra steps to the installation process.
This is not a solution though.
So pip is deprecated now?
And should be using pipx?
What If I have +100 pip packages?, to my knowledge pipx doesn’t support -r requirement.txt.
No I couldn’t since I get the Externally Managed Environment Error.
And no I didn’t use sudo pip when installing it.
I tried upgrading using pip I get the error, updating with pipupgrade is the same, and moving to pipx create conflict.
So what do you suggest?
When installing packages using pip, it is recommended to use a virtual environment to prevent conflicts with system packages in /usr. Alternatively, pip install --user can be used to install packages into the user scheme instead of /usr. Other tools including pipx, poetry and Conda integrate environment management into their workflows.
Note: There are also tools integrating pip with pacman by automatically generating PKGBUILDs for specified PyPI packages: see Creating packages#PKGBUILD generators.
Similar to this user, I’ve just found that pip install --user doesn’t work for me anymore.
I hadn’t tried updating packages since last month’s big python 3.11 update – where I followed the provided guide as best I could – and now I get the PEP 668 message when I do pip install --user yt-dlp.
I had done pretty much all of my python package installations with the --user flag; for more specialized stuff I used miniconda3 to create environments.
You can still install with ‘–user’ if you add ‘–break-system-packages’.
This simply lets ‘–user’ works as before.
The scary wording is only to warn you that version conflicts are possible.
Once more though - this is just a change making it even more apparent how they should have been handled in the first place. With virtual environments. It was slightly more permissive before (now sudo pip isnt even possible it seems) and you can regain that same --user functionality with --break-system-packages … its just more obvious now that you shouldnt and you have to add the extra flag.
The guide that I wrote is here,
and I added info about the pip changes here.
I guess I should make a shorter version of that guide,
as it’s a long read to find everything you need to know in there.
Anaconda or miniconda is a good way to do it,
as it’s both a package manager and a virtual-environment manager.
But for most people even miniconda is too big an installation
if they have only a few pip packages.
I see. I did this update on 2023-06-08 and this info looks like it came out on 2023-06-24, after what I thought was the “big yearly python update”. I follow the [Stable Updates] posts via RSS, I feel like this should have been mentioned up front rather than buried in the guts somewhere.
As for venv usage: I appreciate the need to avoid touching the system python packages, but its another thing to remember on top of the long commands, and then having to source/activate the environment to use these packages.
This is not a Manjaro problem, but it would be nice if the default arrangement was that all users automatically get a default venv to which packages are installed via pip install --user (or even just pip install), with sudo pip still banned. And the user python installation automatically adds source <venv>/bin/activate to .bashrc/.zshrc, so it’s just hidden.
I’m sure there are long and heated discussions about this, and why this idea would instantly break everything on the system, which is why the gods developers in their wisdom deemed this the best of all possible worlds.
We did this some years ago and people revolted.
That doesnt really mean anything … but it happened.
(the dev at the time eventually became flustered with the attacks, reverted the changes … and it was never spoken of again except in circumstances like this)
The main problem I have is that now I can’t combine global packages with the packages in the virtual environment. There are some packages that I prefer to install globally since I use them often. But now I get the error:
Getting a Module Error When Running Pytest Even Though the Module is Installed in the Current Virtual Environment
So I have to install the same module on each virtual environment for each python project which is very bothersome.
No, it’s not. This way, you guarantee that you have the exactly required version.
Additionally, you probably have a requirements.txt file to manage your project’s requirements. You have to list pytest there, otherwise, who will know that pytest is needed because you installed it globally.
I see, well I was using --user because that what I was thought I should be use since I always thought that’s the way.
This whole thing have made an issue for me because I had two versions of urllib3, one from pip and the other from the repo.
I didn’t realize it until like 3 weeks.
So, now the recommended way it to install pip packages one by one using pipx?