crypto/tls: set timeout so that Close does not block indefinitely

[NWilson]

What did you do?

Reading the documentation for tls.Conn (https://golang.org/pkg/crypto/tls/) the deadline behaviour is not clearly explained.

• It's not obvious that Read calls may block forever unless the write deadline is set to a non-zero time. (This is because handshaking is done internally, which writes.) Since this is different to net.TLSConn, it should be called out.
• It's reasonable to understand that Handshake calls need to have both read and write deadline set... but could be called out explicitly.
• But the killer: calling Close on a tls.Conn risks blocking forever, unless the write deadline has been set! (It writes out a close alert.)

This last one is the worst. It would be very easy to have a defer'ed call to Close, and assume that it will always return promptly, when in fact it may be possible for a malicious client to fill up the socket buffers with handshaking data, so that writing the close alert blocks (on an OS with tiny socket buffers?).

I have (or had) some code that does the following:

for {
  conn, _ := listener.Accept() // error handling omitted for clarity
  if tooManyConns() {
    conn.Close() // oops, may block!
    continue
  }
  go handleConn(conn)
}

I have now carefully gone and wrapped all calls to Close with a safeClose wrapper method that calls SetWriteDeadline first!

Perhaps Close should actually call SetWriteDeadline on the underlying connection with a sane value (eg time.Now().Add(100*time.Millisecond)) to ensure that no-one else is caught out.

Thankfully, http.conn.serve does call SetWriteDeadline as soon as possible for TLS connections, and never sets a zero write deadline, so that code at least is safe, and can't call Close with a zero deadline.

What did you expect to see?

More mention of what deadlines need to set, to prevent each method blocking.

[andybons]

@FiloSottile

[networkimprov]

Assuming the above points are all correct, a lot of folks use the tls API incorrectly. I have a TLS+TCP server that calls SetReadDeadline() and Close(). I don't recall mention of SetWriteDeadline() in any example code I read.

Maybe this calls for additional APIs, e.g. SetTLSReadDeadline() & CloseTLS()? Or a go vet item?

[networkimprov]

Ping @FiloSottile

[networkimprov]

@andybons, could this be milestoned for 1.13 and made release-blocker?

Correct use of TLS is a security concern, and one of the suggested fixes is to change .Close() behavior.

[FiloSottile]

I agree this needs addressing, but it does not warrant changing subtle behavior while in deep feature freeze. The only fix for Go 1.13 will be probably in documentation, and we can discuss in 1.14 with more flexibility.

[networkimprov]

Thanks, happy to give feedback on the CL when ready...

[networkimprov]

@FiloSottile, you tagged this for 1.13. It's now been re-tagged for 1.14; was that a mistake?

[networkimprov]

@FiloSottile we might want to tackle this one early in 1.14 cycle...

@gopherbot add release-blocker

[FiloSottile]

I'm thinking of introducing a default deadline on the close notify send, and document in Read and Write, where they say they call Handshake, that that means they need both read and write deadlines.

[networkimprov]

Re setting a default deadline, is there any conflict with http.Server/Client? See #21133 (comment)

[FiloSottile]

Reusing a net.Conn after a tls.Close is weird but I suppose not impossible? Sure, we can just reset it before leaving Close.

[networkimprov]

Just wondering whether the implicit deadlines that http uses might conflict with an implicit deadline in tls.Close?

[FiloSottile]

Ah, well by the time tls.Close is called it doesn't really matter, does it? We are tearing down, so we can just override them.

[networkimprov]

@bradfitz any thoughts?

[bradfitz]

I haven't looked into this issue, but from a quick skim of this thread, it does indeed seem like things are too complicated/surprising for users.

[networkimprov]

@bradfitz any comment re a new implicit timer in tls.Close? Could that conflict with implicit timers in http?

[bradfitz]

A new timer in tls.Close sounds good. If crypto/tls doesn't and net/http doesn't already(?), then it seems that net/http should work around any such blocking and close the underlying net.Conn directly. But it might be already. I'd have to go refresh my memory of the code.

In any case, crypto/tls doing it (regardless of whether a higher layer is also doing it) sounds good.

[ianlancetaylor]

Looks like no change was made for 1.14, so retargeting for 1.15. Also looks like the current thinking is to change the code, so dropping the documentation label.

[networkimprov]

Could we target this for 1.16?

cc @odeke-em in case it's of interest...

[FiloSottile]

The changes here are

• set a Deadline in Close of 5s before sending and trying to read the closeNotify
• call attention to the fact that Read?Write call Handshake so need both deadlines in their docs

[gopherbot]

Change https://golang.org/cl/266037 mentions this issue: crypto/tls: set Deadline before sending close notify alert

[networkimprov]

Could the docs also give guidance on the deadline durations for Read & Write?

[katiehockman]

I think that can be in a separate issue/CL. If we were to provide guidance about an appropriate deadline, then that would be more appropriate in the godoc for SetDeadline, SetReadDeadline, and/or SetWriteDeadline, not in Read or Write.
We would also need to determine what a sensible deadline would be, or how to provide guidance for that.

[networkimprov]

Since you have to set a write deadline on Read, and a read deadline on Write, it would be helpful to know if there's a minimum value necessary in TLS.

[katiehockman]

Sure. Feel free to file another issue. That is out of scope for this one.

[kozlovic]

@katiehockman @FiloSottile This is a bit unfortunate. I knew it could block, so I was setting my own deadline of 100ms (did not really care if the alert is sent or not, but I did not want it to block). With this change, my code now will block up to 5 seconds. Anything I can do to close the low level connection without blocking (up to 5sec) here?

[networkimprov]

Can I ask what services or clients are stalling on close often enough to cause problems?

[kozlovic]

@networkimprov This is in context of NATS Server. When a client is closed, I try to flush and close the connection. Because of TLS, I was setting a low deadline. I have a test that produced the condition that may cause the TLS close to stall and they started to fail, and I realize that this is because of Go overriding my own deadline: https://github.com/nats-io/nats-server/blob/f82ba22be252755aefbf33af8d66db8cf7421b53/server/client.go#L4352
Here is one of the test that checks for that: https://github.com/nats-io/nats-server/blob/f82ba22be252755aefbf33af8d66db8cf7421b53/server/routes_test.go#L1185

[networkimprov]

Could you open a new issue to suggest that TLS close not override an existing deadline, or that a tls.Config field be added to let you specify the close deadline? And then mention that issue here. Thanks!

[networkimprov]

Ping @kozlovic re new issue...

[kozlovic]

@networkimprov Sorry for delay, took some time off. I created the issue #45162.