Skip to content

Commit 13e281d

Browse files
tomchristietomchristie
authored
Add proxy configuration to ConnectionPool. (#974)
* Add proxy configuration to ConnectionPool * Update tests for new proxy API, and nocover old classes. * Update CHANGELOG * Iterate refactor * Revert "Iterate refactor" This reverts commit ee9cfe1. --------- Co-authored-by: Tom Christie <tomchristie@users.noreply.github.com>
1 parent 0bfcee4 commit 13e281d

16 files changed

+231
-116
lines changed

‎CHANGELOG.md

+4
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file.
44

55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
66

7+
## [Unreleased]
8+
9+
- Support `proxy=…` configuration on `ConnectionPool()`.
10+
711
## Version 1.0.6 (October 1st, 2024)
812

913
- Relax `trio` dependency pinning. (#956)

‎docs/async.md

-15
Original file line numberDiff line numberDiff line change
@@ -34,14 +34,6 @@ async with httpcore.AsyncConnectionPool() as http:
3434
...
3535
```
3636

37-
Or if connecting via a proxy:
38-
39-
```python
40-
# The async variation of `httpcore.HTTPProxy`
41-
async with httpcore.AsyncHTTPProxy() as proxy:
42-
...
43-
```
44-
4537
### Sending requests
4638

4739
Sending requests with the async version of `httpcore` requires the `await` keyword:
@@ -221,10 +213,3 @@ anyio.run(main)
221213
handler: python
222214
rendering:
223215
show_source: False
224-
225-
## `httpcore.AsyncHTTPProxy`
226-
227-
::: httpcore.AsyncHTTPProxy
228-
handler: python
229-
rendering:
230-
show_source: False

‎docs/proxies.md

+27-20
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ Sending requests via a proxy is very similar to sending requests using a standar
77
```python
88
import httpcore
99

10-
proxy = httpcore.HTTPProxy(proxy_url="http://127.0.0.1:8080/")
10+
proxy = httpcore.Proxy("http://127.0.0.1:8080/")
11+
pool = httpcore.ConnectionPool(proxy=proxy)
1112
r = proxy.request("GET", "https://www.example.com/")
1213

1314
print(r)
@@ -31,10 +32,11 @@ Proxy authentication can be included in the initial configuration:
3132
import httpcore
3233

3334
# A `Proxy-Authorization` header will be included on the initial proxy connection.
34-
proxy = httpcore.HTTPProxy(
35-
proxy_url="http://127.0.0.1:8080/",
36-
proxy_auth=("<username>", "<password>")
35+
proxy = httpcore.Proxy(
36+
url="http://127.0.0.1:8080/",
37+
auth=("<username>", "<password>")
3738
)
39+
pool = httpcore.ConnectionPool(proxy=proxy)
3840
```
3941

4042
Custom headers can also be included:
@@ -45,10 +47,11 @@ import base64
4547

4648
# Construct and include a `Proxy-Authorization` header.
4749
auth = base64.b64encode(b"<username>:<password>")
48-
proxy = httpcore.HTTPProxy(
49-
proxy_url="http://127.0.0.1:8080/",
50-
proxy_headers={"Proxy-Authorization": b"Basic " + auth}
50+
proxy = httpcore.Proxy(
51+
url="http://127.0.0.1:8080/",
52+
headers={"Proxy-Authorization": b"Basic " + auth}
5153
)
54+
pool = httpcore.ConnectionPool(proxy=proxy)
5255
```
5356

5457
## Proxy SSL
@@ -58,10 +61,10 @@ The `httpcore` package also supports HTTPS proxies for http and https destinatio
5861
HTTPS proxies can be used in the same way that HTTP proxies are.
5962

6063
```python
61-
proxy = httpcore.HTTPProxy(proxy_url="https://127.0.0.1:8080/")
64+
proxy = httpcore.Proxy(url="https://127.0.0.1:8080/")
6265
```
6366

64-
Also, when using HTTPS proxies, you may need to configure the SSL context, which you can do with the `proxy_ssl_context` argument.
67+
Also, when using HTTPS proxies, you may need to configure the SSL context, which you can do with the `ssl_context` argument.
6568

6669
```python
6770
import ssl
@@ -70,11 +73,13 @@ import httpcore
7073
proxy_ssl_context = ssl.create_default_context()
7174
proxy_ssl_context.check_hostname = False
7275

73-
proxy = httpcore.HTTPProxy('https://127.0.0.1:8080/', proxy_ssl_context=proxy_ssl_context)
76+
proxy = httpcore.Proxy(
77+
url='https://127.0.0.1:8080/',
78+
ssl_context=proxy_ssl_context
79+
)
80+
pool = httpcore.ConnectionPool(proxy=proxy)
7481
```
7582

76-
It is important to note that the `ssl_context` argument is always used for the remote connection, and the `proxy_ssl_context` argument is always used for the proxy connection.
77-
7883
## HTTP Versions
7984

8085
If you use proxies, keep in mind that the `httpcore` package only supports proxies to HTTP/1.1 servers.
@@ -91,29 +96,31 @@ The `SOCKSProxy` class should be using instead of a standard connection pool:
9196
import httpcore
9297

9398
# Note that the SOCKS port is 1080.
94-
proxy = httpcore.SOCKSProxy(proxy_url="socks5://127.0.0.1:1080/")
95-
r = proxy.request("GET", "https://www.example.com/")
99+
proxy = httpcore.Proxy(url="socks5://127.0.0.1:1080/")
100+
pool = httpcore.ConnectionPool(proxy=proxy)
101+
r = pool.request("GET", "https://www.example.com/")
96102
```
97103

98104
Authentication via SOCKS is also supported:
99105

100106
```python
101107
import httpcore
102108

103-
proxy = httpcore.SOCKSProxy(
104-
proxy_url="socks5://127.0.0.1:8080/",
105-
proxy_auth=("<username>", "<password>")
109+
proxy = httpcore.Proxy(
110+
url="socks5://127.0.0.1:1080/",
111+
auth=("<username>", "<password>"),
106112
)
107-
r = proxy.request("GET", "https://www.example.com/")
113+
pool = httpcore.ConnectionPool(proxy=proxy)
114+
r = pool.request("GET", "https://www.example.com/")
108115
```
109116

110117
---
111118

112119
# Reference
113120

114-
## `httpcore.HTTPProxy`
121+
## `httpcore.Proxy`
115122

116-
::: httpcore.HTTPProxy
123+
::: httpcore.Proxy
117124
handler: python
118125
rendering:
119126
show_source: False

‎docs/table-of-contents.md

+1-2
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,13 @@
1010
* Connection Pools
1111
* `httpcore.ConnectionPool`
1212
* Proxies
13-
* `httpcore.HTTPProxy`
13+
* `httpcore.Proxy`
1414
* Connections
1515
* `httpcore.HTTPConnection`
1616
* `httpcore.HTTP11Connection`
1717
* `httpcore.HTTP2Connection`
1818
* Async Support
1919
* `httpcore.AsyncConnectionPool`
20-
* `httpcore.AsyncHTTPProxy`
2120
* `httpcore.AsyncHTTPConnection`
2221
* `httpcore.AsyncHTTP11Connection`
2322
* `httpcore.AsyncHTTP2Connection`

‎httpcore/__init__.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@
3434
WriteError,
3535
WriteTimeout,
3636
)
37-
from ._models import URL, Origin, Request, Response
37+
from ._models import URL, Origin, Proxy, Request, Response
3838
from ._ssl import default_ssl_context
3939
from ._sync import (
4040
ConnectionInterface,
@@ -79,6 +79,7 @@ def __init__(self, *args, **kwargs): # type: ignore
7979
"URL",
8080
"Request",
8181
"Response",
82+
"Proxy",
8283
# async
8384
"AsyncHTTPConnection",
8485
"AsyncConnectionPool",

‎httpcore/_async/connection_pool.py

+42-2
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
from .._backends.auto import AutoBackend
99
from .._backends.base import SOCKET_OPTION, AsyncNetworkBackend
1010
from .._exceptions import ConnectionNotAvailable, UnsupportedProtocol
11-
from .._models import Origin, Request, Response
11+
from .._models import Origin, Proxy, Request, Response
1212
from .._synchronization import AsyncEvent, AsyncShieldCancellation, AsyncThreadLock
1313
from .connection import AsyncHTTPConnection
1414
from .interfaces import AsyncConnectionInterface, AsyncRequestInterface
@@ -48,6 +48,7 @@ class AsyncConnectionPool(AsyncRequestInterface):
4848
def __init__(
4949
self,
5050
ssl_context: ssl.SSLContext | None = None,
51+
proxy: Proxy | None = None,
5152
max_connections: int | None = 10,
5253
max_keepalive_connections: int | None = None,
5354
keepalive_expiry: float | None = None,
@@ -89,7 +90,7 @@ def __init__(
8990
in the TCP socket when the connection was established.
9091
"""
9192
self._ssl_context = ssl_context
92-
93+
self._proxy = proxy
9394
self._max_connections = (
9495
sys.maxsize if max_connections is None else max_connections
9596
)
@@ -125,6 +126,45 @@ def __init__(
125126
self._optional_thread_lock = AsyncThreadLock()
126127

127128
def create_connection(self, origin: Origin) -> AsyncConnectionInterface:
129+
if self._proxy is not None:
130+
if self._proxy.url.scheme in (b"socks5", b"socks5h"):
131+
from .socks_proxy import AsyncSocks5Connection
132+
133+
return AsyncSocks5Connection(
134+
proxy_origin=self._proxy.url.origin,
135+
proxy_auth=self._proxy.auth,
136+
remote_origin=origin,
137+
ssl_context=self._ssl_context,
138+
keepalive_expiry=self._keepalive_expiry,
139+
http1=self._http1,
140+
http2=self._http2,
141+
network_backend=self._network_backend,
142+
)
143+
elif origin.scheme == b"http":
144+
from .http_proxy import AsyncForwardHTTPConnection
145+
146+
return AsyncForwardHTTPConnection(
147+
proxy_origin=self._proxy.url.origin,
148+
proxy_headers=self._proxy.headers,
149+
proxy_ssl_context=self._proxy.ssl_context,
150+
remote_origin=origin,
151+
keepalive_expiry=self._keepalive_expiry,
152+
network_backend=self._network_backend,
153+
)
154+
from .http_proxy import AsyncTunnelHTTPConnection
155+
156+
return AsyncTunnelHTTPConnection(
157+
proxy_origin=self._proxy.url.origin,
158+
proxy_headers=self._proxy.headers,
159+
proxy_ssl_context=self._proxy.ssl_context,
160+
remote_origin=origin,
161+
ssl_context=self._ssl_context,
162+
keepalive_expiry=self._keepalive_expiry,
163+
http1=self._http1,
164+
http2=self._http2,
165+
network_backend=self._network_backend,
166+
)
167+
128168
return AsyncHTTPConnection(
129169
origin=origin,
130170
ssl_context=self._ssl_context,

‎httpcore/_async/http_proxy.py

+3-7
Original file line numberDiff line numberDiff line change
@@ -51,12 +51,7 @@ def merge_headers(
5151
return default_headers + override_headers
5252

5353

54-
def build_auth_header(username: bytes, password: bytes) -> bytes:
55-
userpass = username + b":" + password
56-
return b"Basic " + base64.b64encode(userpass)
57-
58-
59-
class AsyncHTTPProxy(AsyncConnectionPool):
54+
class AsyncHTTPProxy(AsyncConnectionPool): # pragma: nocover
6055
"""
6156
A connection pool that sends requests via an HTTP proxy.
6257
"""
@@ -142,7 +137,8 @@ def __init__(
142137
if proxy_auth is not None:
143138
username = enforce_bytes(proxy_auth[0], name="proxy_auth")
144139
password = enforce_bytes(proxy_auth[1], name="proxy_auth")
145-
authorization = build_auth_header(username, password)
140+
userpass = username + b":" + password
141+
authorization = b"Basic " + base64.b64encode(userpass)
146142
self._proxy_headers = [
147143
(b"Proxy-Authorization", authorization)
148144
] + self._proxy_headers

‎httpcore/_async/socks_proxy.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ async def _init_socks5_connection(
102102
raise ProxyError(f"Proxy Server could not connect: {reply_code}.")
103103

104104

105-
class AsyncSOCKSProxy(AsyncConnectionPool):
105+
class AsyncSOCKSProxy(AsyncConnectionPool): # pragma: nocover
106106
"""
107107
A connection pool that sends requests via an HTTP proxy.
108108
"""

‎httpcore/_models.py

+25
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
from __future__ import annotations
22

3+
import base64
4+
import ssl
35
import typing
46
import urllib.parse
57

@@ -489,3 +491,26 @@ async def aclose(self) -> None:
489491
)
490492
if hasattr(self.stream, "aclose"):
491493
await self.stream.aclose()
494+
495+
496+
class Proxy:
497+
def __init__(
498+
self,
499+
url: URL | bytes | str,
500+
auth: tuple[bytes | str, bytes | str] | None = None,
501+
headers: HeadersAsMapping | HeadersAsSequence | None = None,
502+
ssl_context: ssl.SSLContext | None = None,
503+
):
504+
self.url = enforce_url(url, name="url")
505+
self.headers = enforce_headers(headers, name="headers")
506+
self.ssl_context = ssl_context
507+
508+
if auth is not None:
509+
username = enforce_bytes(auth[0], name="auth")
510+
password = enforce_bytes(auth[1], name="auth")
511+
userpass = username + b":" + password
512+
authorization = b"Basic " + base64.b64encode(userpass)
513+
self.auth: tuple[bytes, bytes] | None = (username, password)
514+
self.headers = [(b"Proxy-Authorization", authorization)] + self.headers
515+
else:
516+
self.auth = None

‎httpcore/_sync/connection_pool.py

+42-2
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
from .._backends.sync import SyncBackend
99
from .._backends.base import SOCKET_OPTION, NetworkBackend
1010
from .._exceptions import ConnectionNotAvailable, UnsupportedProtocol
11-
from .._models import Origin, Request, Response
11+
from .._models import Origin, Proxy, Request, Response
1212
from .._synchronization import Event, ShieldCancellation, ThreadLock
1313
from .connection import HTTPConnection
1414
from .interfaces import ConnectionInterface, RequestInterface
@@ -48,6 +48,7 @@ class ConnectionPool(RequestInterface):
4848
def __init__(
4949
self,
5050
ssl_context: ssl.SSLContext | None = None,
51+
proxy: Proxy | None = None,
5152
max_connections: int | None = 10,
5253
max_keepalive_connections: int | None = None,
5354
keepalive_expiry: float | None = None,
@@ -89,7 +90,7 @@ def __init__(
8990
in the TCP socket when the connection was established.
9091
"""
9192
self._ssl_context = ssl_context
92-
93+
self._proxy = proxy
9394
self._max_connections = (
9495
sys.maxsize if max_connections is None else max_connections
9596
)
@@ -125,6 +126,45 @@ def __init__(
125126
self._optional_thread_lock = ThreadLock()
126127

127128
def create_connection(self, origin: Origin) -> ConnectionInterface:
129+
if self._proxy is not None:
130+
if self._proxy.url.scheme in (b"socks5", b"socks5h"):
131+
from .socks_proxy import Socks5Connection
132+
133+
return Socks5Connection(
134+
proxy_origin=self._proxy.url.origin,
135+
proxy_auth=self._proxy.auth,
136+
remote_origin=origin,
137+
ssl_context=self._ssl_context,
138+
keepalive_expiry=self._keepalive_expiry,
139+
http1=self._http1,
140+
http2=self._http2,
141+
network_backend=self._network_backend,
142+
)
143+
elif origin.scheme == b"http":
144+
from .http_proxy import ForwardHTTPConnection
145+
146+
return ForwardHTTPConnection(
147+
proxy_origin=self._proxy.url.origin,
148+
proxy_headers=self._proxy.headers,
149+
proxy_ssl_context=self._proxy.ssl_context,
150+
remote_origin=origin,
151+
keepalive_expiry=self._keepalive_expiry,
152+
network_backend=self._network_backend,
153+
)
154+
from .http_proxy import TunnelHTTPConnection
155+
156+
return TunnelHTTPConnection(
157+
proxy_origin=self._proxy.url.origin,
158+
proxy_headers=self._proxy.headers,
159+
proxy_ssl_context=self._proxy.ssl_context,
160+
remote_origin=origin,
161+
ssl_context=self._ssl_context,
162+
keepalive_expiry=self._keepalive_expiry,
163+
http1=self._http1,
164+
http2=self._http2,
165+
network_backend=self._network_backend,
166+
)
167+
128168
return HTTPConnection(
129169
origin=origin,
130170
ssl_context=self._ssl_context,

0 commit comments

Comments
 (0)