[24.1] self signed certificates are no longer loaded #12779

rasmus91 · 2024-06-21T09:16:09Z

Description

I am using a gitlab python repo, that proxies pypi.

After upgrading to pip 24.1 the certificate I have installed in /usr/local/share/ca-certificates is no longer loaded it seems, to validate that source.

After going back to pip 24.0 It works just fine once again.

Expected behavior

I would expect no problem installing packages through a private repo for which I have appropriate certificates installed systemwide.

pip version

24.1

Python version

3.10

OS

Ubuntu 22.04

How to Reproduce

Install certificate in /usr/local/share/ca-certificates
run sudo update-ca-certificates
setup a source repo to your private repository
install a package from it

Output

(this is output from ansible, but its still pip)

Looking in indexes: https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple
Obtaining file:///var/local/sbat-man-server-dev
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
:stderr:   error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Looking in indexes: https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      Could not fetch URL https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple/poetry-core/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='gitlab.ourinstance.com', port=443): Max retries exceeded with url: /api/v4/groups/25/-/packages/pypi/simple/poetry-core/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
      ERROR: Could not find a version that satisfies the requirement poetry-core (from versions: none)
      ERROR: No matching distribution found for poetry-core
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

Code of Conduct

I agree to follow the PSF Code of Conduct.

The text was updated successfully, but these errors were encountered:

sweetlilmre · 2024-06-21T09:52:32Z

I can confirm similar behaviour on windows.
For 24.1 pip fails with certificate errors for a self-signed cert and only works by pointing REQUESTS_CA_BUNDLE to a bundle containing the self-signed CA.
24.0 requires --use-feature truststore but works as expected. I tested all non-beta versions from 23 onwards.

ichard26 · 2024-06-21T13:18:12Z

Hey all, apologies for the disruption. Pip by default does not use the system certificate store. It is an optional feature enabled by the integration of truststore which did get bumped in 24.1. cc @sethmlarson

sethmlarson · 2024-06-21T14:05:58Z

@rasmus91 @sweetlilmre Just to confirm, you're running into this issue without using the optional "Truststore" feature or while you're using it (ie --with-feature=truststore)? Have to clarify because it's not clear from your posted reproduction steps.

rasmus91 · 2024-06-21T15:34:05Z

Not using that option, no. fre. 21. jun. 2024 16.06 skrev Seth Michael Larson ***@***.***

…

: @sweetlilmre <https://github.com/sweetlilmre> Just to confirm, you're running into this issue *without* using the optional "Truststore" feature or while you're using it (ie --with-feature=truststore)? Have to clarify because it's not clear from your posted reproduction steps. — Reply to this email directly, view it on GitHub <#12779 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARQ7C5RHVTYOACO42J24XDZIQXNXAVCNFSM6AAAAABJVQC7RSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBSHAZDGNRWHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

sweetlilmre · 2024-06-21T15:46:03Z

@sethmlarson, while using it:

versions 23.0 to 24.0 using --with-feature=truststore work (these are the versions I tested).
version 24.1 using --with-feature=truststore does not work and I need to have REQUESTS_CA_BUNDLE defined and pointing to a bundle containing the required CA.

Sorry for the ambiguity.
How it ever worked for @rasmus91 without the feature enabled surprises me.

rasmus91 · 2024-06-21T15:49:05Z

Yes, it did until 24.1. i dont understand why it wouldn't when the CA cert is installed system wide?

sethmlarson · 2024-06-21T16:51:02Z

So I'm actually suspecting that a defect in Requests is the source of the issues here, pip uses Requests 2.32.0 but there have been 3 bugfix releases since that release to Requests with some relating to certificates.

notatallshaw · 2024-06-21T22:18:32Z

I've always wanted to know how pip's vendoring process worked, so I've created two branches to test if requests is the issue:

Branch with latest version of requests: pip install git+https://github.com/notatallshaw/pip@requests-2.32.3
Branch with version of requests pip 24.0 had: pip install git+https://github.com/notatallshaw/pip@requests-2.31.0

Can someone please try their workflow that works on 24.0 but breaks on 24.1 on both of these branches and report the results for both please (may need to use --no-cache-dir and --force to make sure there's a network connetion). It would also be apprecative if you could give exact steps of your workflow.

ichard26 · 2024-06-22T01:33:22Z

Looking at the requests changelog, I strongly suspect that psf/requests#6655 is implicated. psf/requests#6716 which landed in requests v2.32.3 restores the ability to specify custom SSLContexts in sub-classes of HTTPAdapter which pip does have.

pip/src/pip/_internal/network/session.py

Line 290 in 2753c77

class HTTPAdapter(_SSLContextAdapterMixin, _BaseHTTPAdapter):

Please try @notatallshaw's branches to help us determine whether requests is the root problem and upgrading requests fixes the bug. Thanks!

sweetlilmre · 2024-06-22T06:56:05Z

Let me preface this with: I am no python / pip expert, in fact I probably don't know what I'm doing.
That said I ran the following tests:

Create 2 directories, 2.32.3 and 2.31.0
Create a venv in both
Install the appropriate branch of pip in each
Test

Results:

2.32.3:

pip --version
pip 24.0 from C:\source\piptest\2.32.3\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/pip@requests-2.32.3

pip --version
pip 24.2.dev0 from C:\source\piptest\2.32.3\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
- pip install pip-system-certs FAILS (SSL cert errors)
- pip --use-feature truststore install pip-system-certs SUCCEEDS

2.31.0

pip --version
pip 24.0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/pip@requests-2.31.0

pip --version
pip 24.2.dev0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
pip install pip-system-certs FAILS (traceback)

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\source\piptest\2.31.0\.venv\Scripts\pip.exe\__main__.py", line 7, in <module>
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\cli\main.py", line 78, in main
    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\commands\__init__.py", line 114, in create_command
    module = importlib.import_module(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\commands\install.py", line 15, in <module>
    from pip._internal.cli.req_command import (
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\cli\req_command.py", line 18, in <module>
    from pip._internal.index.collector import LinkCollector
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\index\collector.py", line 31, in <module>
    from pip._vendor import requests
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\compat.py", line 10, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'

notatallshaw · 2024-06-22T14:33:24Z

Thanks for running the tests, I'll fix the issue with the 2.31.0 branch later today and confirm.

Am I correct in understanding though that the 2.32.3 branch now succeeds where pip 24.1 fails?

sweetlilmre · 2024-06-22T15:35:37Z

As per my tests, yes.
When you have the 2.31.0 branch sorted I will test

notatallshaw · 2024-06-22T17:57:21Z

I fixed the 2.31.0 branch if you want to reinstall it and try, though it's a little less important seeing that 2.32.3 works, it at least prooves definitively it was a requests regression and unrelated to truststore.

sweetlilmre · 2024-06-24T07:54:17Z

I reran the 2.31.0 branch for completeness sake.

2.31.0

pip --version
pip 24.0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/pip@requests-2.31.0

pip --version
pip 24.2.dev0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
pip install pip-system-certs FAILS (SSL)

pip --use-feature truststore install pip-system-certs SUCCEEDS

sweetlilmre · 2024-06-24T08:05:26Z

As a final comment on this issue, --use-feature truststore as a default and not a feature would be great. Doesn't fix the wider certifi / requests certificate handling debacle, but certainly helps for bootstrapping python, especially for new users.

pfmoore · 2024-06-24T10:20:30Z

Doesn't fix the wider certifi / requests certificate handling debacle

While I'm sure you didn't mean to offend, "debacle" is a bit strong here. We published a beta with the new requests release, and didn't get any feedback on this issue. The pip maintainers themselves don't have access to environments that need this feature, and so we weren't able to trigger the issue ourselves. If you have any suggestions (or better still a PR!) for how we could test this within our test suite then that would be appreciated - although it's arguable that this is something that should be covered in the requests test suite, so maybe proposing something at their end would also be useful.

We try to avoid regressions like this, but there's a lot of pip functionality that's only used by particular parts of our user base, and often those users don't (for perfectly good reasons) get involved in testing and proving new releases for us. Negative feedback tends to reduce our motivation for taking risks in those areas, which in turn slows the pace of developments and improvements for those users. It's difficult to find a good balance.

sweetlilmre · 2024-06-24T10:48:37Z

@pfmoore first off apologies, this was in no way intended as a slight toward the pip or request maintainers.
In fact the handling of this issue has been top notch with quick responses and the option of neophytes like myself to offer some value from clear directions and things to test.

The debacle I was referring to is the general certificate handling case in the python ecosystem. truststore is a massive leap forward in terms of an improvement, similar to the intent behind pip-system-certs etc.

Unfortunately I live in a corporate environment with SSL inspection. This causes havoc all the time with new starts and even folks that have come back to python after a while, or had to reinstall for whatever reason. REQUESTS_CA_BUNDLE is always quoted as the solution but works in only some cases (seems to work for python and pip, but with certifi we inevitably end up having to patch the CA bundle, pip-system-certs used to work but has stopped recently).

truststore or something like that, baked in would be a huge win for python as a whole and debacle was the word that (perhaps a little unfortunately) I chose to express my frustration with the state of play.

Again apologies, and thank you to you, and all involved in this work.

notatallshaw · 2024-06-25T03:00:40Z

FWIW, back in my large corporation days I had to teach people to use px-proxy: https://github.com/genotrance/px?tab=readme-ov-file#installation

Which I found as the most consistent workaround to Python http ecosystem vs. enterprise security. Although I didn't get to try truststore, I'm now at a relatively small company where I don't face super esoteric security protocols

sweetlilmre · 2024-06-25T07:34:39Z

@notatallshaw thank you, I'll take a look at that.

rasmus91 added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jun 21, 2024

pradyunsg added project: vendored dependency Related to a vendored dependency and removed S: needs triage Issues/PRs that need to be triaged labels Jun 21, 2024

notatallshaw mentioned this issue Jun 21, 2024

Upgrade requests to 2.32.3 #12784

Merged

pfmoore mentioned this issue Jun 23, 2024

gh-120888: Bump bundled pip to 24.1.1 python/cpython#120889

Merged

bodgit mentioned this issue Jun 25, 2024

Ability to set desired version of pip, or to not update it at all actions/setup-python#901

Open

notatallshaw mentioned this issue Jun 25, 2024

pip 24.1 use-feature truststore needs a list of trusted-host sethmlarson/truststore#142

Closed

pradyunsg closed this as completed in #12784 Jun 25, 2024

pradyunsg changed the title ~~After latest pip update self signed certificates are no longer loaded~~ [24.1] self signed certificates are no longer loaded Jun 25, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jul 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[24.1] self signed certificates are no longer loaded #12779

[24.1] self signed certificates are no longer loaded #12779

rasmus91 commented Jun 21, 2024

sweetlilmre commented Jun 21, 2024 •

edited

Loading

ichard26 commented Jun 21, 2024 •

edited

Loading

sethmlarson commented Jun 21, 2024 •

edited

Loading

rasmus91 commented Jun 21, 2024 via email

sweetlilmre commented Jun 21, 2024

rasmus91 commented Jun 21, 2024

sethmlarson commented Jun 21, 2024 •

edited

Loading

notatallshaw commented Jun 21, 2024

ichard26 commented Jun 22, 2024

sweetlilmre commented Jun 22, 2024 •

edited

Loading

notatallshaw commented Jun 22, 2024 •

edited

Loading

sweetlilmre commented Jun 22, 2024 •

edited

Loading

notatallshaw commented Jun 22, 2024

sweetlilmre commented Jun 24, 2024

sweetlilmre commented Jun 24, 2024

pfmoore commented Jun 24, 2024

sweetlilmre commented Jun 24, 2024 •

edited

Loading

notatallshaw commented Jun 25, 2024

sweetlilmre commented Jun 25, 2024

[24.1] self signed certificates are no longer loaded #12779

[24.1] self signed certificates are no longer loaded #12779

Comments

rasmus91 commented Jun 21, 2024

Description

Expected behavior

pip version

Python version

OS

How to Reproduce

Output

Code of Conduct

sweetlilmre commented Jun 21, 2024 • edited Loading

ichard26 commented Jun 21, 2024 • edited Loading

sethmlarson commented Jun 21, 2024 • edited Loading

rasmus91 commented Jun 21, 2024 via email

sweetlilmre commented Jun 21, 2024

rasmus91 commented Jun 21, 2024

sethmlarson commented Jun 21, 2024 • edited Loading

notatallshaw commented Jun 21, 2024

ichard26 commented Jun 22, 2024

sweetlilmre commented Jun 22, 2024 • edited Loading

2.32.3:

2.31.0

notatallshaw commented Jun 22, 2024 • edited Loading

sweetlilmre commented Jun 22, 2024 • edited Loading

notatallshaw commented Jun 22, 2024

sweetlilmre commented Jun 24, 2024

2.31.0

sweetlilmre commented Jun 24, 2024

pfmoore commented Jun 24, 2024

sweetlilmre commented Jun 24, 2024 • edited Loading

notatallshaw commented Jun 25, 2024

sweetlilmre commented Jun 25, 2024

sweetlilmre commented Jun 21, 2024 •

edited

Loading

ichard26 commented Jun 21, 2024 •

edited

Loading

sethmlarson commented Jun 21, 2024 •

edited

Loading

sethmlarson commented Jun 21, 2024 •

edited

Loading

sweetlilmre commented Jun 22, 2024 •

edited

Loading

notatallshaw commented Jun 22, 2024 •

edited

Loading

sweetlilmre commented Jun 22, 2024 •

edited

Loading

sweetlilmre commented Jun 24, 2024 •

edited

Loading