Python pip breaks and how to recover?

followait · 8 May 2022 14:40

Reproduce

$ pip install git+https://github.com/BingLingGroup/autosub.git@origin
$ pip uninstall autosub
(to remove dependecies of autosub, did `pip uninstall` as below, for some libs it needs root, and I also did it)
$ pip uninstall google-api-python-client pysrt progressbar2 six google-api-core uritemplate google-auth httplib2 google-auth-httplib2 protobuf googleapi2-common-protos rsa cachetools pyasn1-modules pyparsing python-utils pyasn1 chardet idna urllib
(now pip breaks)
$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 33, in <module>
    sys.exit(load_entry_point('pip==21.0', 'console_scripts', 'pip')())
  File "/usr/bin/pip", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 8, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/cmdoptions.py", line 20, in <module>
    from pip._vendor.packaging.utils import canonicalize_name
ModuleNotFoundError: No module named 'pip._vendor.packaging'

Tried to solve as below but failed

sudo rm -rf /usr/lib/python3.10/site-packages/pip/
sudo pacman -Syu python-pip

Reinstall python and python-pip in pamac

Question

How to recover pip?

maycne.sonahoz · 8 May 2022 16:31

Please detail how those failed exactly.

followait · 8 May 2022 16:46

All in the section Reproduce

maycne.sonahoz · 8 May 2022 16:57

So they failed to solve the issue, but didn’t fail themselves.

Should we understand you sudo pip uninstall some Python packages then?..

followait · 8 May 2022 17:11

Yes, I must have uninstalled some python packages.

maycne.sonahoz · 8 May 2022 18:11

Well that’s a problem.

It is recommended not to use pip with sudo, specifically so that it doesn’t interfere with Python packages installed by the system package manager.

Comparing the list of Python packages you forcibly removed, to the dependencies of python-pip, i spot several ones ought to be managed by the package manager:

progressbar (python-progress ?)
six (python-six)
cachetools (python-cachecontrol ?)
pyparsing (python-pyparsing)
chardet (python-chardet)
idna (python-idna)
urllib (python-urllib3 ?)

You should properly reinstall python-pip and its dependencies.

sudo pacman -Rsn python-pip
sudo pacman -S python-pip

followait · 9 May 2022 02:02

The names python-* are all right, but they have been installed while error

ModuleNotFoundError: No module named 'pip._vendor.packaging'

occurs

The commands run successfully, but still the same error when running pip

By pacman -S python-pip, all the dependencies as shown below should be installed.
Does it mean that python-pip has missed some necessary dependencies?

$ pacman -Qi python-pip
Name            : python-pip
Version         : 21.0-1
Description     : The PyPA recommended tool for installing Python packages
Architecture    : any
URL             : https://pip.pypa.io/
Licenses        : MIT
Groups          : None
Provides        : None
Depends On      : python-appdirs  python-cachecontrol  python-colorama  python-contextlib2  python-distlib
                  python-distro  python-html5lib  python-packaging  python-pep517  python-progress
                  python-requests  python-retrying  python-resolvelib  python-setuptools  python-six
                  python-toml  python-pyopenssl
...

followait · 9 May 2022 04:49

Solved by

$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python get-pip.py

I don’t know why.

New problem, now reinstall python-pip in pamac, pip beaks again

$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 33, in <module>
    sys.exit(load_entry_point('pip==21.0', 'console_scripts', 'pip')())
  File "/usr/bin/pip", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 8, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/cmdoptions.py", line 22, in <module>
    from pip._internal.cli.progress_bars import BAR_TYPES
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 14, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 25, in <module>
    from pip._vendor.retrying import retry  # type: ignore
ModuleNotFoundError: No module named 'pip._vendor.retrying'

megavolt · 9 May 2022 05:30

You must reinstall all dependencies also.

sudo pacman -S python-pip $(expac -l '\n' %E -S python-pip | sort -u)

Since it seems you removed these files also with pip

followait · 9 May 2022 05:45

The error is replaced by another error

$ sudo pacman -S python-pip $(expac -l '\n' %E -S python-pip | sort -u)  
warning: python-pip-21.0-1 is up to date -- reinstalling
warning: python-appdirs-1.4.4-6 is up to date -- reinstalling
warning: python-cachecontrol-1:0.12.6-4 is up to date -- reinstalling
warning: python-colorama-0.4.4-6 is up to date -- reinstalling
warning: python-contextlib2-0.6.0.post1-6 is up to date -- reinstalling
warning: python-distlib-0.3.4-1 is up to date -- reinstalling
warning: python-distro-1.7.0-1 is up to date -- reinstalling
warning: python-html5lib-1.1-11 is up to date -- reinstalling
warning: python-packaging-20.9-7 is up to date -- reinstalling
warning: python-pep517-0.12.0-4 is up to date -- reinstalling
warning: python-progress-1.6-5 is up to date -- reinstalling
warning: python-pyopenssl-22.0.0-1 is up to date -- reinstalling
warning: python-requests-2.27.1-1 is up to date -- reinstalling
warning: python-resolvelib-0.5.5-4 is up to date -- reinstalling
warning: python-retrying-1.3.3-13 is up to date -- reinstalling
warning: python-setuptools-1:59.4.0-1 is up to date -- reinstalling
warning: python-six-1.16.0-5 is up to date -- reinstalling
warning: python-toml-0.10.2-7 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (18) python-appdirs-1.4.4-6  python-cachecontrol-1:0.12.6-4  python-colorama-0.4.4-6
              python-contextlib2-0.6.0.post1-6  python-distlib-0.3.4-1  python-distro-1.7.0-1
              python-html5lib-1.1-11  python-packaging-20.9-7  python-pep517-0.12.0-4  python-pip-21.0-1
              python-progress-1.6-5  python-pyopenssl-22.0.0-1  python-requests-2.27.1-1
              python-resolvelib-0.5.5-4  python-retrying-1.3.3-13  python-setuptools-1:59.4.0-1
              python-six-1.16.0-5  python-toml-0.10.2-7

Total Installed Size:  8.98 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] 
(18/18) checking keys in keyring                                   [####################################] 100%
(18/18) checking package integrity                                 [####################################] 100%
(18/18) loading package files                                      [####################################] 100%
(18/18) checking for file conflicts                                [####################################] 100%
(18/18) checking available disk space                              [####################################] 100%
:: Running pre-transaction hooks...
(1/1) Creating Timeshift snapshot before upgrade...
==> skipping timeshift-autosnap due skipRsyncAutosnap in /etc/timeshift-autosnap.conf set to TRUE.
:: Processing package changes...
( 1/18) reinstalling python-appdirs                                [####################################] 100%
( 2/18) reinstalling python-requests                               [####################################] 100%
( 3/18) reinstalling python-cachecontrol                           [####################################] 100%
( 4/18) reinstalling python-colorama                               [####################################] 100%
( 5/18) reinstalling python-contextlib2                            [####################################] 100%
( 6/18) reinstalling python-distlib                                [####################################] 100%
( 7/18) reinstalling python-six                                    [####################################] 100%
( 8/18) reinstalling python-packaging                              [####################################] 100%
( 9/18) reinstalling python-setuptools                             [####################################] 100%
(10/18) reinstalling python-distro                                 [####################################] 100%
(11/18) reinstalling python-html5lib                               [####################################] 100%
(12/18) reinstalling python-pep517                                 [####################################] 100%
(13/18) reinstalling python-progress                               [####################################] 100%
(14/18) reinstalling python-retrying                               [####################################] 100%
(15/18) reinstalling python-resolvelib                             [####################################] 100%
(16/18) reinstalling python-toml                                   [####################################] 100%
(17/18) reinstalling python-pyopenssl                              [####################################] 100%
(18/18) reinstalling python-pip                                    [####################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Refreshing PackageKit...
$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 33, in <module>
    sys.exit(load_entry_point('pip==21.0', 'console_scripts', 'pip')())
  File "/usr/bin/pip", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 8, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/cmdoptions.py", line 22, in <module>
    from pip._internal.cli.progress_bars import BAR_TYPES
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 14, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 29, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/usr/lib/python3.10/site-packages/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/usr/lib/python3.10/site-packages/pip/_internal/locations/_sysconfig.py", line 8, in <module>
    from pip._internal.exceptions import InvalidSchemeCombination, UserInstallationInvalid
ImportError: cannot import name 'InvalidSchemeCombination' from 'pip._internal.exceptions' (/usr/lib/python3.10/site-packages/pip/_internal/exceptions.py)

megavolt · 9 May 2022 05:49

sudo pacman -S python-pip --overwrite "*"

followait · 9 May 2022 06:34

same error

$ sudo pacman -S python-pip --overwrite "*"
[sudo] password for abc: 
warning: python-pip-21.0-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (1) python-pip-21.0-1

Total Installed Size:  1.56 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                                     [####################################] 100%
(1/1) checking package integrity                                   [####################################] 100%
(1/1) loading package files                                        [####################################] 100%
(1/1) checking for file conflicts                                  [####################################] 100%
(1/1) checking available disk space                                [####################################] 100%
:: Running pre-transaction hooks...
(1/1) Creating Timeshift snapshot before upgrade...
==> skipping timeshift-autosnap due skipRsyncAutosnap in /etc/timeshift-autosnap.conf set to TRUE.
:: Processing package changes...
(1/1) reinstalling python-pip                                      [####################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Refreshing PackageKit...
$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 33, in <module>
    sys.exit(load_entry_point('pip==21.0', 'console_scripts', 'pip')())
  File "/usr/bin/pip", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 8, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/cmdoptions.py", line 22, in <module>
    from pip._internal.cli.progress_bars import BAR_TYPES
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 14, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 29, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/usr/lib/python3.10/site-packages/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/usr/lib/python3.10/site-packages/pip/_internal/locations/_sysconfig.py", line 8, in <module>
    from pip._internal.exceptions import InvalidSchemeCombination, UserInstallationInvalid
ImportError: cannot import name 'InvalidSchemeCombination' from 'pip._internal.exceptions' (/usr/lib/python3.10/site-packages/pip/_internal/exceptions.py)

megavolt · 9 May 2022 07:05

remove that file. It is not in the package:

pacman -Ql python-pip | grep "sysconfig.py"

followait · 9 May 2022 07:51

megavolt:

followait:

File "/usr/lib/python3.10/site-packages/pip/_internal/locations/_sysconfig.py", line 8, in <module>

remove that file. It is not in the package:
pacman -Ql python-pip | grep "sysconfig.py"

$ sudo rm /usr/lib/python3.10/site-packages/pip/_internal/locations/_sysconfig.py
$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 33, in <module>
    sys.exit(load_entry_point('pip==21.0', 'console_scripts', 'pip')())
  File "/usr/bin/pip", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 8, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/cmdoptions.py", line 22, in <module>
    from pip._internal.cli.progress_bars import BAR_TYPES
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 14, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 29, in <module>
    from pip._internal.locations import get_major_minor_version, site_packages, user_site
  File "/usr/lib/python3.10/site-packages/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
ImportError: cannot import name '_sysconfig' from partially initialized module 'pip._internal.locations' (most likely due to a circular import) (/usr/lib/python3.10/site-packages/pip/_internal/locations/__init__.py)

megavolt · 9 May 2022 07:56

ok we fixed both errors by reinstalling the dependencies. But now there are incompatible modules from v22, which does not work with v21, so it is mixed now. So you need to remove the folder again and reinstall like you have done before:

followait · 9 May 2022 08:55

pip without params works, but

$ pip install gitpython
Traceback (most recent call last):
  File "/usr/bin/pip", line 33, in <module>
    sys.exit(load_entry_point('pip==21.0', 'console_scripts', 'pip')())
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 71, in main
    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
  File "/usr/lib/python3.10/site-packages/pip/_internal/commands/__init__.py", line 96, in create_command
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 15, in <module>
    from pip._internal.cli.req_command import RequirementCommand, with_cleanup
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 16, in <module>
    from pip._internal.index.collector import LinkCollector
  File "/usr/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 16, in <module>
    from pip._vendor import html5lib, requests
ImportError: cannot import name 'requests' from 'pip._vendor' (/usr/lib/python3.10/site-packages/pip/_vendor/__init__.py)

megavolt · 9 May 2022 09:02

Again remove/reinstall python-requests

followait · 9 May 2022 09:16

python-requests is a transitive dependency of pacman

Some relationships:

python-requests required by
manjaro-application-utility required by
pacman-mirrors required by
libpamac pacman python-manjaro-sdk
…

It’s hard to remove python-requests and reinstalling it makes no difference

megavolt · 9 May 2022 09:23

Then check at least if files are changed there:

pacman -Qkk python-requests

followait · 9 May 2022 09:36

$ pacman -Qkk python-requests 
python-requests: 67 total files, 0 altered files