Spring Cloud Gateway MVC is using a single http2 connection to downstream

[jensmatw]

Describe the bug
If the downstream server accepts http2 and for example offers 100 concurrent streams (tomcat default) the http client in gateway mvc is using up to those 100 streams in one connection, then on the 101st parallel stream, it throws an exception (too many concurrent streams) even when the tomcat would allow a lot more connections (default is 200 tomcat threads). I'm not sure if this is a problem in either JDK, springs RestTemplate or SCG mvc. Somewhere should be a connection pool which uses multiple real connections instead of trying to put everything in a single one.

Sample
Use a route to a default config tomcat (spring boot) with a long enough response time and add up to 101 requests in parallel (or reduce tomcat concurrent streams to a smaller number for easier testing).

[spencergibb]

By default, we use the jdk http client

This seems relevant https://www.baeldung.com/java-httpclient-connection-management#enhanced-httpclient

jdk.httpclient.maxstreams – set this property to control the maximum number of H2 streams that are permitted per HTTP connection (defaults to 100).

[jensmatw]

The http client is working more or less correctly: it uses the max value that the server is telling him (if it is a tomcat, it is 100 by default). The problem is, when 100 is reached, the http client just throws an exception, it has no own logic to create more connections. So currently, the gateway-mvc can't handle more than 100 (or whatever server is telling) parallel streams which is very limiting. The reactive gateway on the other hand uses webclient which brings it's own connection pool managed in a reactive thread pool or something. So it just creates more connections when streams max is reached.

I'm not sure if it is on springs resttemplate code or if there should be a "connection pool manager" in the gateway. But i think it is very bad that the 101st connection is just rejected by an exception because the gateway will not scale.

[spencergibb]

The reactive gateway uses netty HttpClient, not WebClient. Did you try setting the property I mentioned above?

[jensmatw]

The setting you mention would increase the amount of streams from client side. If the server (eg. the microservice downstream) is only accepting 100 streams (like a default tomcat) per connection, it will not work.

There was a JDK bug where the client did not respect the servers settings that are transmitted in the handshakelike part. But now it does.

As long as you are under control of the servers in your route configuration you could increase the max streams on all those webservers. But it would perform poorly if you put too many streams in a single connection.

If you have a route configured in the gateway to a tomcat. As soon as there are 101 clients sending requests to the gateway for that particular route via individual connections, the gateway dispatches all of them through a single connection to a that tomcat and the exception ("too many concurrent streams") is thrown.

[spencergibb]

I only control the client side in mvc gateway. It could be tomcat downstream or python or .net.

I don't understand what you think I could do with the jdk http client under my control.

[jensmatw]

This could only be implemented client side but I know that we can't modify the jdk http client.

I think you are right, it is nothing for the gateway to implement connection management to add more connections when max streams is reached.

But I still think that no one expects that the gateway-mvc is only capable of serving 100 simultaneous requests per route. The gateway-webflux scales because netty seems to manage the connection pool for itself.

Maybe at least the documentaiton could be updated, telling that the gateway-mvc in default config is only for "testing" while the user has to configure another http client for "production" use.

Or the default client could be changed to jetty (they handle that in the client: https://stackoverflow.com/a/55987161) but I think this would require a lot of testing (I had a bad experience when quickly testing other clients).

For an implementaiton, maybe it would be better suited in Spring somewhere around JdkClientHttpRequestFactory.

[rstoyanchev]

@jensmatw did you test with HttpClient from Reactor Netty? It is used on the reactive side, and we have control over it when changes need to be made.

[jensmatw]

@rstoyanchev thanks for your hint. I already tried some clients, not sure if netty was among them. I had a hard time, because every client had another issue (apache, jetty, okhttp).

I will try netty and jetty again, they sound promising.

When netty is working flawless, maybe it is just enough to give users a hint in the documentation.

[spencergibb]

Maybe at least the documentaiton could be updated, telling that the gateway-mvc in default config is only for "testing" while the user has to configure another http client for "production" use.

I wouldn't say the jdk client isn't useful for production. Maybe a WARNING about this particular issue. We should also document better using other http clients. Curently, boot supports apache, jetty, and okhttp by simply adding those dependencies and setting spring.cloud.gateway.mvc.http-client.type=autodetect. To use another client, such as reactor netty you need to create a bean of type ClientHttpRequestFactory.

[violetagg]

Curently, boot supports apache, jetty, and okhttp by simply adding those dependencies

@spencergibb Is there any reason for not adding Reactor Netty? Some issues?

[spencergibb]

So spring-cloud-gateway-server-mvc only depends on spring-boot-starter-web. Users are free to add any third party http client.

[violetagg]

So spring-cloud-gateway-server-mvc only depends on spring-boot-starter-web. Users are free to add any third party http client.

I cannot understand RestClient supports Reactor Netty out of the box

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[violetagg]

This one spring-projects/spring-framework#33635 adds auto-detection in Spring Framework for Reactor Netty HttpClient

[rstoyanchev]

I also created spring-projects/spring-boot#42587 for Boot support.