gabbi, http/2, httpx?

[cdent]

#331 raises the question of whether it might be useful to upgrade gabbi to use something other than urllib3 for its http client and somehow add explicit support for http/2.

It appears that httpx is the way to go on this.

One advantage of it is that it would allow us to do wsgi-intercept style behavior without wsgi-intercept (which is badly showing its age).

Thoughts? (ping @scottwallacesh and @FND since they were in previous discussions).

[scottwallacesh]

I'm actually in the process of replacing urllib3 with httpx.

Right now it's passing 171 tests but failing 139.

[cdent]

Oh, awesome. I guess you had the aforementioned nap ;)

Do you understand the issues with replacing wsgi-intercept? Perhaps that is something we should accomplish separately.

[scottwallacesh]

Do you understand the issues with replacing wsgi-intercept?

Not fully, no. At the moment, as it's been nicely abstracted, I'm just working on gabbi/httpclient.py and hoping for a seamless transition.

[cdent]

Excellent, we'll put off worrying about intercept and http/2 until after replicating existing http/1 behavior.

[scottwallacesh]

Well, for better or worse, I've raised a draft PR (#333) with the initial swap of httpx for urllib3.

[FND]

I understand the impetus here and agree that it would sometimes be useful for Gabbi to support evil binary protocols.

However, I would caution that this is probably a niche concern, so perhaps we shouldn't burden users with additional dependencies when they happily get by with glorious text-based protocols.

A superficial analysis suggests urllib3 is self-contained while httpx comes with another six transitive dependencies. This might not be meaningful, but it's worth considering at least (explicit > implicit etc.).

I'm guessing that replacing urllib3 wholesale is gonna be a big undertaking (kudos to Scott for starting on that journey). I'm further guessing that an incremental transition, with httpx only kicking in on demand for H2, is not a realistic option? Otherwise it might be worth considering options for it to be an optional dependency.

Apologies for being a bit of a killjoy. My concerns regarding dependencies might be overly informed by other ecosystems.

[scottwallacesh]

I fully appreciate the concerns. At the moment it's just "exploratory" work to see if httpx can be a drop-in replacement for urllib3. Actually adding HTTP 2 (or greater) support is a whole other ball game.

[cdent]

The extra dependencies is a fair concern, and you're right we shouldn't just do it because it is cool, but:

• dependency chaos is not as big of deal in python as it is in say node and its behave-alikes (especially with this group[1])
• urllib3 has moved forward to a version 2 but gabbi pins to <2 because 2 breaks wsgi-intercept in interesting ways, httpx has it's own solution for interception
• I recall there being other funky things about urllib3 v2 when I looked at it
• httpx is more modern-python aware so is probably a move in the right direction irrespective of HTTP 2
• I haven't seen what @scottwallacesh is doing, but at some point in the distant past we replaced httplib2 with urllib3 in gabbi and it was a) pretty easy, b) done with an eye to making future changes similarly easy by encapsulating the http client so I'm hoping/guessing that a transition now would not be super painful

Given all that, I'd say it is certainly worthy of exploration and if it isn't too dramatic, use it for http/1 and then see what /2 looks like in the future.

[1] I think we saw a presentation from these people at the pycon we were at in Coventry.

[FND]

Thanks for spelling that out: Makes perfect sense and seems like a valid rationale. (All I remember about Coventry is being yelled at by some disembodied voice.)

[cdent]

Switching to httpx will probably warrant a major version update (to 4). Although the external interface doesn't change the way intercepting works is sufficiently different that it is probably useful to signal it in the most visible way possible.

Agree? Disagree?