Use httpx.WSGITransport for wsgi app interception

cdent · cdent · commit 9a07fa8d11cc · 2025-03-21T18:17:16.000Z
Replace wsgi-intercept, used in gabbi as a feature
to install the intercept and do http over a monkey-
patched socker.

The replacement is WSGITransport from httpx. This works
in a different way. Instead of monkey-patching, a
different transport is dependency-injected into the
http client. Each gabbi tests has its own client
instance.

Because of this change in concept the diff, though
not especially big, touches several parts of the code.

1. Fixture handling removes use of the wsgi-intecept
   context manager.
2. intercept and prefix are based to httpclient generation.
   intercept signals that the WSGITransport should be
   used.
3. An external server process (in tests/external_server.py)
   is used by tests/test_runner.py so that gabbi-run
   can be tested effectively without wsgi-intercept.
4. Step 3 identified some long present bugs in the
   SimpleWSGI test app to do with reading content
   from POST and PUT operations.
5. Prefix handling and url-generation are different when
   using WSGITransport so simplewsgi has a different way
   of constructing fully qualified URLs. This uses
   removeprefix which is no supported in python 3.8, so
   support for 3.8 is dropped. 3.8 was EOL in 2024.
6. Special tests which had all been depdending on the older
   python 3.9 now use 3.13. This is mostly because that
   made it is easier for cdent to do testing locally, but
   seems good hygiene anyway.
7. pypy3 is upgrade to pypy3.10 as that's what's available
   these days.
8. Docs have tried to be updated but it is likelys something
   has been missed.
9. Some shenanigans with how hostnames are handled in SimpleWSGI
   needed to be reconciled between what a mac does and what a
   github ci node does.

This patch does _not_ try to prepare a release. That will
be later.
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
@@ -11,8 +11,6 @@ jobs:
                 include:
                 - python: 3.x
                   toxenv: pep8
-                - python: 3.8
-                  toxenv: py38
                 - python: 3.9
                   toxenv: py39
                 - python: "3.10"
@@ -23,10 +21,8 @@ jobs:
                   toxenv: py312
                 - python: "3.13"
                   toxenv: py313
-                - python: pypy-3.8
+                - python: pypy-3.10
                   toxenv: pypy3
-                - python: 3.8
-                  toxenv: py38-pytest
                 - python: 3.9
                   toxenv: py39-pytest
                 - python: "3.10"
@@ -37,12 +33,12 @@ jobs:
                   toxenv: py312-pytest
                 - python: "3.13"
                   toxenv: py313-pytest
-                - python: 3.9
-                  toxenv: py39-failskip
-                - python: 3.9
-                  toxenv: py39-limit
-                - python: 3.9
-                  toxenv: py39-prefix
+                - python: "3.13"
+                  toxenv: py313-failskip
+                - python: "3.13"
+                  toxenv: py313-limit
+                - python: "3.13"
+                  toxenv: py313-prefix
         name: ${{ matrix.toxenv }} on Python ${{ matrix.python }}
         steps:
         - uses: actions/checkout@v2
diff --git a/README.rst b/README.rst
@@ -20,7 +20,7 @@ looks like this::
 See the docs_ for more details on the many features and formats for
 setting request headers and bodies and evaluating responses.
 
-Gabbi is tested with Python 3.8, 3.9, 3.10, 3.11, 3.12, 3.13 and pypy3.
+Gabbi is tested with Python 3.9, 3.10, 3.11, 3.12, 3.13 and pypy3.10.
 
 Tests can be run using `unittest`_ style test runners, `pytest`_
 or from the command line with a `gabbi-run`_ script.
diff --git a/docs/source/host.rst b/docs/source/host.rst
@@ -6,9 +6,8 @@ Gabbi intends to preserve the flow and semantics of HTTP interactions
 as much as possible, and every HTTP request needs to be directed at a host
 of some form. Gabbi provides three ways to control this:
 
-* Using `wsgi-intercept`_ to provide a fake socket and ``WSGI``
-  environment on an arbitrary host and port attached to a ``WSGI``
-  application (see `intercept examples`_).
+* Using `WSGITransport` of httpx to provide a ``WSGI`` environment on
+  directly attached to a ``WSGI`` application (see `intercept examples`_).
 * Using fully qualified ``url`` values in the YAML defined tests (see
   `full examples`_).
 * Using a host and (optionally) port defined at test build time (see
@@ -18,15 +17,15 @@ The intercept and live methods are mutually exclusive per test builder,
 but either kind of test can freely intermix fully qualified URLs into the
 sequence of tests in a YAML file.
 
-For test driven development and local tests the intercept style of
-testing lowers test requirements (no web server required) and is fast.
-Interception is performed as part of :doc:`fixtures` processing as the most
-deeply nested fixture. This allows any configuration or database
-setup to be performed prior to the WSGI application being created.
+For Python-based test driven development and local tests the intercept
+style of testing lowers test requirements (no web server required) and
+is fast. Interception is performed as part of the per-test-case http
+client. Configuration or database setup may be performed using
+:doc:`fixtures`.
 
 For the implementation of the above see :meth:`~gabbi.driver.build_tests`.
 
-.. _wsgi-intercept: https://pypi.python.org/pypi/wsgi_intercept
+.. _WSGITransport: https://www.python-httpx.org/advanced/transports/#wsgi-transport
 .. _intercept examples: https://github.com/cdent/gabbi/blob/main/gabbi/tests/test_intercept.py
 .. _full examples: https://github.com/cdent/gabbi/blob/main/gabbi/tests/gabbits_live/google.yaml
 .. _live examples: https://github.com/cdent/gabbi/blob/main/gabbi/tests/test_live.py
diff --git a/gabbi/case.py b/gabbi/case.py
@@ -29,8 +29,6 @@
 from unittest import result as unitresult
 import urllib.parse as urlparse
 
-import wsgi_intercept
-
 from gabbi import __version__
 from gabbi import exception
 from gabbi.handlers import base
@@ -492,29 +490,19 @@ def _run_request(
         redirect=False,
         timeout=30,
     ):
-        """Run the http request and decode output.
-
-        The call to make the request will catch a WSGIAppError from
-        wsgi_intercept so that the real traceback from a catastrophic
-        error in the intercepted app can be examined.
-        """
+        """Run the http request and decode output."""
 
         if 'user-agent' not in (key.lower() for key in headers):
             headers['user-agent'] = "gabbi/%s (Python httpx)" % __version__
 
-        try:
-            response, content = self.http.request(
-                url,
-                method=method,
-                headers=headers,
-                body=body,
-                redirect=redirect,
-                timeout=timeout,
-            )
-        except wsgi_intercept.WSGIAppError as exc:
-            # Extract and re-raise the wrapped exception.
-            raise exc.exception_type(exc.exception_value).with_traceback(
-                   exc.traceback) from None
+        response, content = self.http.request(
+            url,
+            method=method,
+            headers=headers,
+            body=body,
+            redirect=redirect,
+            timeout=timeout,
+        )
 
         # Set headers and location attributes for follow on requests
         self.response = response
diff --git a/gabbi/driver.py b/gabbi/driver.py
@@ -52,7 +52,7 @@ def build_tests(path, loader, host=None, port=8001, intercept=None,
     :param loader: The TestLoader.
     :param host: The host to test against. Do not use with ``intercept``.
     :param port: The port to test against. Used with ``host``.
-    :param intercept: WSGI app factory for wsgi-intercept.
+    :param intercept: WSGI app factory for httpclient WSGITransport.
     :param test_loader_name: Base name for test classes. Use this to align the
                              naming of the tests with other tests in a system.
     :param fixture_module: Python module containing fixture classes.
@@ -158,7 +158,7 @@ def build_tests(path, loader, host=None, port=8001, intercept=None,
 
 
 def py_test_generator(test_dir, host=None, port=8001, intercept=None,
-                      prefix=None, test_loader_name=None,
+                      prefix='', test_loader_name=None,
                       fixture_module=None, response_handlers=None,
                       content_handlers=None, require_ssl=False, url=None,
                       metafunc=None, use_prior_test=True,
diff --git a/gabbi/httpclient.py b/gabbi/httpclient.py
@@ -34,7 +34,14 @@ def __init__(self, **kwargs):
         self.extensions = {}
         if 'server_hostname' in kwargs:
             self.extensions['sni_hostname'] = kwargs['server_hostname']
-        self.client = httpx.Client(verify=kwargs.get('cert_validate', True))
+        transport = kwargs.get('intercept')
+        if transport:
+            transport = httpx.WSGITransport(
+                app=transport(), script_name=kwargs.get("prefix", "")
+            )
+        self.client = httpx.Client(
+            transport=transport, verify=kwargs.get("cert_validate", True)
+        )
 
     def request(self, absolute_uri, method, body, headers, redirect, timeout):
         response = self.client.request(
@@ -191,6 +198,8 @@ def get_http(
     caption='',
     cert_validate=True,
     hostname=None,
+    intercept=None,
+    prefix='',
     timeout=30,
 ):
     """Return an ``Http`` class for making requests."""
@@ -199,6 +208,8 @@ def get_http(
             server_hostname=hostname,
             timeout=timeout,
             cert_validate=cert_validate,
+            intercept=intercept,
+            prefix=prefix,
         )
 
     headers = verbose != 'body'
@@ -213,4 +224,6 @@ def get_http(
         server_hostname=hostname,
         timeout=timeout,
         cert_validate=cert_validate,
+        intercept=intercept,
+        prefix=prefix,
     )
diff --git a/gabbi/suite.py b/gabbi/suite.py
@@ -19,8 +19,6 @@
 import sys
 import unittest
 
-from wsgi_intercept import interceptor
-
 from gabbi import fixture
 
 
@@ -46,16 +44,11 @@ def run(self, result, debug=False):
         are no fixtures.
         """
 
-        fixtures, intercept, host, port, prefix = self._get_intercept()
+        fixtures, host, port = self._get_fixtures()
 
         try:
             with fixture.nest([fix() for fix in fixtures]):
-                if intercept:
-                    with interceptor.Urllib3Interceptor(
-                            intercept, host, port, prefix):
-                        result = super(GabbiSuite, self).run(result, debug)
-                else:
-                    result = super(GabbiSuite, self).run(result, debug)
+                result = super(GabbiSuite, self).run(result, debug)
         except unittest.SkipTest as exc:
             for test in self._tests:
                 result.addSkip(test, str(exc))
@@ -85,7 +78,7 @@ def run(self, result, debug=False):
     def start(self, result, tests=None):
         """Start fixtures when using pytest."""
         tests = tests or []
-        fixtures, intercept, host, port, prefix = self._get_intercept()
+        fixtures, host, port = self._get_fixtures()
 
         self.used_fixtures = []
         try:
@@ -100,38 +93,25 @@ def start(self, result, tests=None):
                 test.run = noop
                 test.add_marker('skip')
             result.addSkip(self, str(exc))
-        if intercept:
-            intercept_fixture = interceptor.Urllib3Interceptor(
-                intercept, host, port, prefix)
-            intercept_fixture.__enter__()
-            self.used_fixtures.append(intercept_fixture)
 
     def stop(self):
         """Stop fixtures when using pytest."""
         for fix in reversed(self.used_fixtures):
             fix.__exit__(None, None, None)
 
-    def _get_intercept(self):
+    def _get_fixtures(self):
         fixtures = [fixture.GabbiFixture]
-        intercept = host = port = prefix = None
+        host = port = None
         try:
             first_test = self._find_first_full_test()
             fixtures = first_test.fixtures
             host = first_test.host
             port = first_test.port
-            prefix = first_test.prefix
-            intercept = first_test.intercept
-
-            # Unbind a passed in WSGI application. During the
-            # metaclass building process intercept becomes bound.
-            try:
-                intercept = intercept.__func__
-            except AttributeError:
-                pass
+
         except AttributeError:
             pass
 
-        return fixtures, intercept, host, port, prefix
+        return fixtures, host, port
 
     def _find_first_full_test(self):
         """Traverse a sparse test suite to find the first HTTPTestCase.
diff --git a/gabbi/suitemaker.py b/gabbi/suitemaker.py
@@ -90,6 +90,8 @@ def make_one_test(self, test_dict, prior_test):
                                          caption=test['name'],
                                          cert_validate=test['cert_validate'],
                                          hostname=hostname,
+                                         intercept=self.intercept,
+                                         prefix=self.prefix,
                                          timeout=int(test["timeout"]))
         if prior_test:
             history = prior_test.history
diff --git a/gabbi/tests/external_server.py b/gabbi/tests/external_server.py
@@ -0,0 +1,32 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+import sys
+from wsgiref import simple_server
+
+from gabbi.tests import simple_wsgi
+
+
+def run(host, port):
+    server = simple_server.make_server(
+        host,
+        int(port),
+        simple_wsgi.SimpleWsgi(),
+    )
+    server.serve_forever()
+
+
+if __name__ == "__main__":
+    host = sys.argv[1]
+    port = sys.argv[2]
+    run(host, port)
diff --git a/gabbi/tests/gabbits_intercept/host-header.yaml b/gabbi/tests/gabbits_intercept/host-header.yaml
@@ -1,4 +1,4 @@
-# Test, against wsgi-intercept, that SNI and host header handling behaves.
+# Test, against intercepted WSGI app, that SNI and host header handling behaves.
 
 tests:
 
diff --git a/gabbi/tests/simple_wsgi.py b/gabbi/tests/simple_wsgi.py
@@ -30,17 +30,22 @@ def __call__(self, environ, start_response):
         global METHODS
         global CURRENT_POLL
 
+        script_name = environ.get('SCRIPT_NAME', '')
+        path_info = environ.get('PATH_INFO', '').removeprefix(script_name)
         request_method = environ['REQUEST_METHOD'].upper()
-        query_data = urlparse.parse_qs(environ.get('QUERY_STRING', ''))
-        request_url = environ.get('REQUEST_URI',
-                                  environ.get('RAW_URI', 'unknown'))
-        path_info = environ.get('PATH_INFO', '')
+        query_string = environ.get('QUERY_STRING', '')
+        query_data = urlparse.parse_qs(query_string)
+        request_url = script_name + path_info
         accept_header = environ.get('HTTP_ACCEPT')
         content_type_header = environ.get('CONTENT_TYPE', '')
 
-        full_request_url = self._fully_qualify(environ, request_url)
+        full_request_url = self._fully_qualify(
+            environ,
+            request_url,
+            query_string,
+        )
 
-        if accept_header:
+        if accept_header and accept_header != '*/*':
             response_content_type = accept_header
         else:
             # JSON doesn't need a charset but we throw one in here
@@ -64,7 +69,8 @@ def __call__(self, environ, start_response):
             return []
 
         if request_method.startswith('P'):
-            body = environ['wsgi.input'].read()
+            length = int(environ.get('CONTENT_LENGTH', '0'))
+            body = environ['wsgi.input'].read(length)
             if body:
                 if not content_type_header:
                     start_response('400 Bad request', headers)
@@ -147,7 +153,7 @@ def __call__(self, environ, start_response):
         return [query_output.encode('utf-8')]
 
     @staticmethod
-    def _fully_qualify(environ, url):
+    def _fully_qualify(environ, url, query_data):
         """Turn a URL path into a fully qualified URL."""
         split_url = urlparse.urlsplit(url)
         server_name = environ.get('SERVER_NAME')
@@ -159,4 +165,4 @@ def _fully_qualify(environ, url):
             netloc = server_name
 
         return urlparse.urlunsplit((server_scheme, netloc, split_url.path,
-                                    split_url.query, split_url.fragment))
+                                    query_data, split_url.fragment))
diff --git a/gabbi/tests/test_runner.py b/gabbi/tests/test_runner.py
diff --git a/requirements.txt b/requirements.txt
diff --git a/setup.cfg b/setup.cfg
diff --git a/tox.ini b/tox.ini

Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,4 @@`
`1`		`-# Test, against wsgi-intercept, that SNI and host header handling behaves.`
	`1`	`+# Test, against intercepted WSGI app, that SNI and host header handling behaves.`
`2`	`2`
`3`	`3`	`tests:`
`4`	`4`