Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Httpx intercept #334

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Httpx intercept #334

wants to merge 2 commits into from

Conversation

cdent
Copy link
Owner

@cdent cdent commented Mar 18, 2025

httpx provides a more modern approach to python+http with
a straightforward API that maps cleanly to gabbi/httpclient.py.

It also, in the future, will allow support for http/2.

The change switches out urllib3 from under httpclient,
replacing it with httpx.

It does not address changes required to get interception
of wsgi applications working. httpx provides its own
way of doing interception that is much different from
the old hack of wsgi-intercept. Those changes will be
next.

Replace wsgi-intercept, used in gabbi as a feature
to install the intercept and do http over a monkey-
patched socker.

The replacement is WSGITransport from httpx. This works
in a different way. Instead of monkey-patching, a
different transport is dependency-injected into the
http client. Each gabbi tests has its own client
instance.

Because of this change in concept the diff, though
not especially big, touches several parts of the code.

  1. Fixture handling removes use of the wsgi-intecept
    context manager.
  2. intercept and prefix are based to httpclient generation.
    intercept signals that the WSGITransport should be
    used.
  3. An external server process (in tests/external_server.py)
    is used by tests/test_runner.py so that gabbi-run
    can be tested effectively without wsgi-intercept.
  4. Step 3 identified some long present bugs in the
    SimpleWSGI test app to do with reading content
    from POST and PUT operations.
  5. Prefix handling and url-generation are different when
    using WSGITransport so simplewsgi has a different way
    of constructing fully qualified URLs. This uses
    removeprefix which is no supported in python 3.8, so
    support for 3.8 is dropped. 3.8 was EOL in 2024.
  6. Special tests which had all been depdending on the older
    python 3.9 now use 3.13. This is mostly because that
    made it is easier for cdent to do testing locally, but
    seems good hygiene anyway.
  7. pypy3 is upgrade to pypy3.10 as that's what's available
    these days.
  8. Docs have tried to be updated but it is likelys something
    has been missed.
  9. Some shenanigans with how hostnames are handled in SimpleWSGI
    needed to be reconciled between what a mac does and what a
    github ci node does.

This patch does not try to prepare a release. That will
be later.

@cdent
Copy link
Owner Author

cdent commented Mar 18, 2025

Current failures are due to difference with host ip resolution between my Mac (which is doing weird stuff) and the CI runners (which are behaving in what I would think is the normal way).

Will provide details and a real PR in the near future @scottwallacesh , @FND but pinging here in case you are curious about progress. Apologies for the terrible commit messages, that will be fixed once this is "real". I've been...distracted.

@scottwallacesh
Copy link
Contributor

Looks good to me. I can see it's still referencing wsgi-intercept in the requirements.txt and a few other places (host.rst, driver.py and host-headers.yaml) - not sure if they can go or not.

@cdent
Copy link
Owner Author

cdent commented Mar 19, 2025

I can see it's still referencing wsgi-intercept

Yeah, there's some cleanups to do, as well as documentation updates before we can call this something like done.

scottwallacesh and others added 2 commits March 21, 2025 18:13
httpx provides a more modern approach to python+http with
a straightforward API that maps cleanly to gabbi/httpclient.py.

It also, in the future, will allow support for http/2.

The change switches out urllib3 from under httpclient,
replacing it with httpx.

It does not address changes required to get interception
of wsgi applications working. httpx provides its own
way of doing interception that is much different from
the old hack of wsgi-intercept. Those changes will be
next.
Replace wsgi-intercept, used in gabbi as a feature
to install the intercept and do http over a monkey-
patched socker.

The replacement is WSGITransport from httpx. This works
in a different way. Instead of monkey-patching, a
different transport is dependency-injected into the
http client. Each gabbi tests has its own client
instance.

Because of this change in concept the diff, though
not especially big, touches several parts of the code.

1. Fixture handling removes use of the wsgi-intecept
   context manager.
2. intercept and prefix are based to httpclient generation.
   intercept signals that the WSGITransport should be
   used.
3. An external server process (in tests/external_server.py)
   is used by tests/test_runner.py so that gabbi-run
   can be tested effectively without wsgi-intercept.
4. Step 3 identified some long present bugs in the
   SimpleWSGI test app to do with reading content
   from POST and PUT operations.
5. Prefix handling and url-generation are different when
   using WSGITransport so simplewsgi has a different way
   of constructing fully qualified URLs. This uses
   removeprefix which is no supported in python 3.8, so
   support for 3.8 is dropped. 3.8 was EOL in 2024.
6. Special tests which had all been depdending on the older
   python 3.9 now use 3.13. This is mostly because that
   made it is easier for cdent to do testing locally, but
   seems good hygiene anyway.
7. pypy3 is upgrade to pypy3.10 as that's what's available
   these days.
8. Docs have tried to be updated but it is likelys something
   has been missed.
9. Some shenanigans with how hostnames are handled in SimpleWSGI
   needed to be reconciled between what a mac does and what a
   github ci node does.

This patch does _not_ try to prepare a release. That will
be later.
@cdent cdent force-pushed the httpx-intercept branch from c735967 to 9a07fa8 Compare March 21, 2025 18:31
@cdent cdent marked this pull request as ready for review March 21, 2025 18:34
@cdent cdent requested a review from FND March 24, 2025 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants