PYTHON interpreter UPgrade: Poetry

I could not find shiboken 6.5.1.1. Removed the pip-pacakages:

$ pip uninstall shiboken6-generator
$ pip uninstall shiboken6
$ pip uninstall pyside6-essentials

$ pip check
No broken requirements found.

6.5.1.1 was probably the one you removed.

Iā€™d be interested to know if your system and your apps work OK now.

Btw, do you really need all of the pip packages you had installed?
poetry is an alternative to pip.
PySimpleGUI is an alternative to pyside.
If youā€™re going to build an app with PySimpleGUI or pyside
you need a much better understanding of python installationsā€¦

you did.
you need to spend more time reading your command ouput :slight_smile:
this is in one of your messages:

$ pip install --user --upgrade PySide6-Addons
Successfully uninstalled PySide6-Addons-6.5.0
Successfully installed
PySide6-Addons-6.5.1.1
PySide6-Essentials-6.5.1.1
shiboken6-6.5.1.1

be aware that pip list --user and pip install --user
do not look at system site-packages. but pip check does.
so maybe ā€˜pip checkā€™ finds conflicts that donā€™t really matter,
coz system and user can run separate versions (i think).

(itā€™s late here, i have to stop now.)

Thanks a lot for your support!

Yes, everything works fine and

$ pip check
No broken requirements found

What is the goal?

Poetry is a replacement for pip that automatically creates a virtual environment for you, it is in the repos python-poetry no need to use pip to install it.

ā€œmaybe ā€˜pip checkā€™ finds conflicts that donā€™t really matterā€.
I think this is true.

When the system starts a program, it does not see your user site-packages, so thereā€™s no conflict.

When you run a program as user, it imports from user site-packages first, and imports from system space only if what it wants is not in user space.
Conflict is possible, but you can avoid it by taking care about what you install.

I suspect that the OP had followed a guide intended for Windows or Mac users,
saying to install all those things with pip because thereā€™s no system repo or AUR.

Thank you @j77h and @codesardine

The strange thing: Poetry came obviously with the last MANJARO-update and produced the error as shown in my first post here; I have a listings of installed packages almost monthly and poetry showe up yesterday for the first time. I Never used poetry ā€¦

And: Yes, I know I may have an issue with PYTHON-packages from repository and pip. I startet with PYTHON few months ago and installed for my applications several packages like pandas, numpy, gpx-cmd-tools,, gpxpy using pip. Meanwhile I learned, that it is recommended to use a Virtual Environment. But I have no idea how to do this and how to transfer the approx. 200 PYTHON-packages, distinguishing if I really need each or not.

But a Virtual Environment might not be necessary.

Packages installed into the user-space ā€˜site-packagesā€™
(as with ā€˜pip install --user ...ā€™) cannot affect system python,
because ā€œsystem pythonā€ (i.e. a python process started by the system)
never imports packages from user-space,

The converse is not true though:
a python process started by a user
does import from system ā€˜site-packagesā€™.

BUT, it looks in user ā€˜site-packagesā€™ first
(user ā€˜site-packagesā€™ is earlier in the import path)
and then looks in system ā€˜site-packagesā€™ only if
the target package is not in user ā€˜site-packagesā€™.

So your user-space pyside/shiboken 6.5.1
was NOT in conflict with system-space pyside/shiboken 6.5.0.
(But itā€™s not needed if your user-space packages
are happy to run on system-space pyside/shiboken.)

My view is that you donā€™t need a virtual environment
as long as the packages in user ā€˜site-packagesā€™ can all
work with the same versions of common dependencies.

If you make a virtual environment with ā€˜venvā€™,
basically all it gives you is another separate user ā€˜site-packagesā€™ folder.
It still runs the system python.

A bit more about PySide6 and double installations.

Probably I was wrong to tell you to remove your PySide6 that was installed by pip.
This one:

Today I noticed that PySide6 in system site-packages weighs about ~50 Mo
but the PySide6 in user site-packages weighs about ~230 Mo.
If in future you find that some app complains about
parts of PySide6 missing, this difference would explain why.

I think it would be safe to disregard the warning from ā€˜pip checkā€™.
My own system had PySide6 installed in both places for a long time, and I never noticed.

You can re-install them with
pip install --user PySide6
This should pull in Essentials, Addons and shiboken as dependencies.
Probably it would get them out of cache and not do any download.

It might be best to do this as soon as you have time, rather than leaving it
until thereā€™s an error, and hoping that you then remember this message.

If re-installing it does cause problems, you could easily do
pip uninstall PySide6 to remove them all again.
It tells you what it would uninstall from where, and asks for yes/no;
then you should say no if it would remove it from ā€˜/usr/lib/...ā€™ ,
but say yes to remove it from ā€˜/home/your-name/.local/lib/...ā€™.

EDIT: you could have the best of both worlds
(two installs of PySide6 but no conflict) by doing
pip install --user 'PySide6==6.5.0'
(or whatever the then-current version is for system PySide6)

Thanks a lot @j77h for your support.

I lost control after all my changes. Actually I did this now as you recommended:

$ pip install --user PySide6
Requirement already satisfied: PySide6 in /usr/lib/python3.11/site-packages (6.5.0)
Requirement already satisfied: shiboken6==6.5.0 in /usr/lib/python3.11/site-packages (from PySide6) (6.5.0)
$ pip check
No broken requirements found.

Seems to be OK, what do you think? Thanks a lot!!

Ha ha! I didnā€™t think of that.
To avoid pip thinking itā€™s ā€œalready satisfiedā€, you would have to do
pip install --user 'PySide6==6.5.1.1
unless thereā€™s a better trick iā€™ve overlookedā€¦

When my system had PySide6 installed in both places,
they were not the same version, and I never saw a problem.
(i was using the one in user-space to make a little Qt-Python app.)

(itā€™s 1 a.m. here ā€“ have to shut down now.)

Seems to be a never ending story, Iā€™m sorry!

I did this (without the ā€™ in front of Pyside!):

$ pip install --user PySide6==6.5.1.1
Collecting PySide6==6.5.1.1
  Using cached PySide6-6.5.1.1-cp37-abi3-manylinux_2_28_x86_64.whl (6.7 kB)
Collecting shiboken6==6.5.1.1 (from PySide6==6.5.1.1)
  Using cached shiboken6-6.5.1.1-cp37-abi3-manylinux_2_28_x86_64.whl (173 kB)
Collecting PySide6-Essentials==6.5.1.1 (from PySide6==6.5.1.1)
  Using cached PySide6_Essentials-6.5.1.1-cp37-abi3-manylinux_2_28_x86_64.whl (80.9 MB)
Collecting PySide6-Addons==6.5.1.1 (from PySide6==6.5.1.1)
  Using cached PySide6_Addons-6.5.1.1-cp37-abi3-manylinux_2_28_x86_64.whl (126.2 MB)
Installing collected packages: shiboken6, PySide6-Essentials, PySide6-Addons, PySide6
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
shiboken6-generator 6.5.0 requires shiboken6==6.5.0, but you have shiboken6 6.5.1.1 which is incompatible.
Successfully installed PySide6-6.5.1.1 PySide6-Addons-6.5.1.1 PySide6-Essentials-6.5.1.1 shiboken6-6.5.1.1
[hessler@IBM-T14s Daten]$ pip check
shiboken6-generator 6.5.0 has requirement shiboken6==6.5.0, but you have shiboken6 6.5.1.1.

Now what? I have not any idea ā€¦

Copy-paste error. I meant to have a ā€™ at the end as well.
(Following examples in pip docs.)

But it worked OK; it brought them all back out of the cache.

ERROR: ...
shiboken6-generator 6.5.0 requires shiboken6==6.5.0, 
but you have shiboken6 6.5.1.1 which is incompatible.

Successfully installed PySide6-6.5.1.1 PySide6-Addons-6.5.1.1 
PySide6-Essentials-6.5.1.1 shiboken6-6.5.1.1

shiboken6-generator is the only one out of step, so do this:
pip install --user --upgrade 'shiboken6-generator==6.5.1.1'

Note that --user keeps this in user-space so it wonā€™t conflict with system-python.
Whether the existing ā€˜shiboken6-generator 6.5.0ā€™ is in system or user space, this is safe,
as it will either add it new to user space or upgrade it in user space,
and will not change anything in system space.

Always do pip install with --user,
NEVER with sudo, to avoid installing into system python.

As I mentioned before, some ā€˜errorsā€™ that ā€˜pip checkā€™ reports can be disregarded,
as it does not consider the separateness of system and user modules.
Unfortunately the ā€˜checkā€™ message does not say where each module is,
so you canā€™t immediately see if they are on opposite sides of the system/user divide.
So itā€™s important to mainly take notice of the error message from ā€˜pip install --userā€™.

To avoid version conflicts between user modules and system modules, remember this:

  • a python process started by the system (user is root) will never conflict with your user-space, because root never tries to import from user-space.

  • python run by you will always try to import first from user space, and will try to import from system-python only if it does not find the module in user space.

  • so if a process started by you wants to import a module from system python, and the version is wrong in system python, then simply install that module into user-space:
    pip install --user pkgname

  • the previous point may trigger more dependencies to be installed in user space (but with pyside6 I doubt this will happen); to avoid too much of this happening, sometimes you could downgrade a module in user-space, if that would be the simpler solution.
    pip install --user --upgrade 'pkgname==lower-version'

Please put this in your notes and remember to read it if such conflicts arise. :slight_smile:

(Iā€™ll try to put it in my HowTo ā€œtutorialā€, but itā€™s rather long already.)

Thank you, but ā€¦

$ pip install --user --upgrade 'shiboken6-generator==6.5.1.1'
ERROR: Could not find a version that satisfies the requirement shiboken6-generator==6.5.1.1 (from versions: none)
ERROR: No matching distribution found for shiboken6-generator==6.5.1.1

ā€¦ there is the old issue again.

Srill no problem I think.
Now itā€™s a matter of knowing your packages.

A day or two ago i was looking at the folders
where PySide6 lives in both system and user space,
(in a btrfs snapshot with python3.10)
and noticed that only one of them had ā€œshiboken6-generatorā€.
Donā€™t remember which one.

This result tells me that only the system has shiboken6-generator
(because its version matches the systemā€™s PySide6)
while PySide6_Essentials etc in user space never had (and donā€™t need) the generator.

So the previous ā€œerrorā€:

shiboken6-generator 6.5.0 has requirement shiboken6==6.5.0, but you have shiboken6 6.5.1.1.

was a false alert, because
shiboken6-generator 6.5.0 and shiboken6 6.5.0 both exist in system space,
and have no need to relate to whatā€™s in user space,
while your shiboken6 6.5.1.1 is in and only for user space.

Iā€™ll have to check details again to make sure Iā€™m right,
but for now donā€™t worry about it.

EDIT:
shiboken6-generator 6.5.0 and shiboken6_generator 6.5.0
are definitely both in system python.
These (and PySide6 6.5.0) are from Manjaro repo.

EDIT2:
The generator is bundled with shiboken6, itā€™s not a separate package.
See (Arch Linux - shiboken6 6.7.0-3 (x86_64)) in ā€˜Package Contentsā€™.
Iā€™m 90% sure that everything is OK with your setup.

EDIT3:

From (Package Details - Qt for Python)

The package shiboken6-generator is not a dependency, and itā€™s not available on PyPi.
The reason, is that it depends on libclang, which is a large library that we donā€™t package

Itā€™s for generating Python bindings in C++ libraries, which we ā€œusersā€ donā€™t normally do.

I have installed PySide6_Essentials with pip, and get the same error as you.
I have run a PySide app that I built, and it runs fine.

I have a test script that shows where it imports modules from.
This is from a normal run, importing from user space:

sys.executable     : /usr/bin/python3.11
PySide6.__version__: 6.5.1.1
PySide6.__file__   : /home/jh/.local/lib/python3.11/site-packages/PySide6/__init__.py
QtCore.__file__    : /home/jh/.local/lib/python3.11/site-packages/PySide6/QtCore.abi3.so
QtGui.__file__     : /home/jh/.local/lib/python3.11/site-packages/PySide6/QtGui.abi3.so
QtWidgets.__file__ : /home/jh/.local/lib/python3.11/site-packages/PySide6/QtWidgets.abi3.so
Qt imported OK

When I tweak the path so that it cannot import from user space,
it works fine in system space:

sys.executable     : /usr/bin/python3.11
PySide6.__version__: 6.5.0
PySide6.__file__   : /usr/lib/python3.11/site-packages/PySide6/__init__.py
QtCore.__file__    : /usr/lib/python3.11/site-packages/PySide6/QtCore.abi3.so
QtGui.__file__     : /usr/lib/python3.11/site-packages/PySide6/QtGui.abi3.so
QtWidgets.__file__ : /usr/lib/python3.11/site-packages/PySide6/QtWidgets.abi3.so
Qt imported OK

This exceeds my ablities, sorry.
I checked with pamac: pyside6 and shiboken6 are installed from repository, both version 6.5.0.2.And

$ pip list
...
PySide6              6.5.1.1
PySide6-Addons       6.5.1.1
PySide6-Essentials   6.5.1.1
...
shiboken6            6.5.1.1
shiboken6-generator  6.5.0
...

What shall I do or can I keep it as it is?

The simple thing to remember is that
pip is not aware of the separation of system and user space
so you need to be aware of it yourself when interpreting what pip tells you.

ā€˜pip listā€™ gives preference to user space: where the same package is in both spaces,
as with PySide6 and shiboken6, it shows only the one in user space.

There is not a problem with shiboken6-generator
because your user-space PySide6 does not use it.

Sorry if my previous edits overloaded you with info.
They simply prove the point that system and user can have
different versions of the PySide6 packages without any problem.

Fairly soon, the system will have pyside 6.5.1.1 (itā€™s in unstable now),
and then both spaces will have the same version.

For the future, not for any real reason, but just to avoid
being concerned about the misleading info that pip gives you,
donā€™t upgrade your PySide6 with pip, until the sytem has the latest version;
then theyā€™ll both stay on the same version.

Btw, ā€œsystem pythonā€ exists only on Linux, and I guess this is why
the pip developers havenā€™t yet fully provided for the system-space/user-space split.

Thank you @j77h for all the information ā€¦

Iā€™m starting do understand all the stuff (and will never use sudo pip install).
One question about maintenance and update for python packages (if your time allows): Actually Iā€™m running this Maintenace.sh nearly monthly:

pip list --outdated --format=columns > "$FileMaintnPath/PythonPackages ${Date}.txt"
echo "Upgrade for old  Python-Modules (pip)" > "$FileMaintnPath/PythonPackages ${Date}.txt"
pip freeze --user | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install --upgrade 

Do for have any proposal for an improvement?
Thanks again and regrads, Michael

Guido invented Python to save us all from this kind of unreadable code. :slight_smile:

The first line and last line produce completely different lists; not the same packages.
So although you may intend to use pip list --outdated, you actually donā€™t.

The second line removes the first lineā€™s list from the text file, so you never see it.

Check this again: ([HowTo] Upgrade many python packages with pip)

Seems like you need to slow down, and learn a subject thoroughly before you try to use it.

The crazy shell stuff works, but you need only awk -F "==" '{print $1}', as in
pip freeze | awk -F "==" '{print $1}' > textfile
to remove version numbers.

Thanks again,

The source of my code comes from here, may be, this makes it better understandable.