WebSocket custom SSL validation only works when using client certificates

[esardaya]

Describe the bug
It's possible to disable (or provide custom) SSL validation by setting SslCertificateAuthentication like in this example:

serviceCertificate.SslCertificateAuthentication = new X509ServiceCertificateAuthentication()
{
    CertificateValidationMode = X509CertificateValidationMode.None,
    RevocationMode = X509RevocationMode.NoCheck,
};

However, this only works when using client certificate authentication.
Looking at:

wcf/src/System.ServiceModel.Http/src/System/ServiceModel/Channels/ClientWebSocketTransportDuplexSessionChannel.cs

Line 155 in f789e63

if (_channelFactory is HttpsChannelFactory<IDuplexSessionChannel> httpsChannelFactory && httpsChannelFactory.RequireClientCertificate)

It seems like the WebSocket RemoteCertificateValidationCallback is only ever set if RequireClientCertificate is true, which will only be the case when using client certificates.

It should be possible to disable or customize server SSL certificate validation regardless of the auth type being used.

To Reproduce

1. Create a binding where WebSockets are always enabled, without setting any client certificate auth.

var binding = new NetHttpBinding(BasicHttpSecurityMode.Transport)
{
    MaxReceivedMessageSize = int.MaxValue,  
    WebSocketSettings = { TransportUsage = WebSocketTransportUsage.Always },
};

2. Disable SSL validation with:

serviceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.Custom;
serviceCertificate.Authentication.CustomCertificateValidator = new DisableCertificateValidation();
serviceCertificate.SslCertificateAuthentication = new X509ServiceCertificateAuthentication()
{
    CertificateValidationMode = X509CertificateValidationMode.None,
    RevocationMode = X509RevocationMode.NoCheck
};

3. Try to call a service that uses a self-signed SSL certificate, or one that doesn't match the domain.

Expected behavior
The call should succeed.

Actual behavior
Call fails with an error similar to this one:

Exception info dump:
System.ServiceModel.CommunicationException: Unable to connect to the remote server
 ---> System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[mconnew]

@imcarolwang, in .NET Framework, we use HttpWebRequest to handle the initial connection including the WebSockets handshake, and then wrap the connection stream in a WebSocket to send/receive websocket messages. This enabled us to do everything with the initial connection that we can do with Http requests (certificates, headers etc). This wasn't available in earlier .NET Core days so we had to use the WebSocket class to make the connection and were limited to what settings can get set on it.
Today things are very different in .NET. Internally they do something similar to what WCF did in NetFx, but using SocketsHttpHandler to make the initial connection. We now need to change our implementation to do the same thing so that everything can be done that we used to do.

Here's the entry point for the implementation on .NET Framework:
https://referencesource.microsoft.com/#System.ServiceModel/System/ServiceModel/Channels/ClientWebSocketTransportDuplexSessionChannel.cs,122
Some of the implementation called is in this method. This is where the HttpWebRequest is retrieved, which includes setting the credentials, certificate etc:
https://referencesource.microsoft.com/#System.ServiceModel/System/ServiceModel/Channels/ClientWebSocketTransportDuplexSessionChannel.cs,122
This is where the resulting response is used to get the stream and wrap it in a websocket:
https://referencesource.microsoft.com/#System.ServiceModel/System/ServiceModel/Channels/ClientWebSocketTransportDuplexSessionChannel.cs,357
One thing you'll see is there was an api added to provide an external implementation of the websockets framing logic. This was because websockets weren't supported for Windows 7, and this was a way to use them. You don't need to worry about anything to do with the connectionFactory (with a type ClientWebSocketFactory).

An overview of the process on .NET is this:

1. Create a SocketsHttpHandler, set certificate, certificate validation callback, and credentials. Wrap in an HttpMessageInvoker.
2. Create an HttpRequestMessage. Set the required WebSocket headers, including any sub protocols, set the service url etc.
3. Send the request using the invoker
4. Validate the response headers, including upgrade status code, websocket specific headers, sub protocol.
5. Fetch the content stream from the response content
6. Wrap the content stream in a websocket.

The .NET code which does something similar starts here:
https://source.dot.net/#System.Net.WebSockets.Client/System/Net/WebSockets/WebSocketHandle.Managed.cs,71
Don't worry about all the logic handling Http 2, just explicitly specify Http 1.1. Adding the headers required for WebSockets can be found here:
https://source.dot.net/#System.Net.WebSockets.Client/System/Net/WebSockets/WebSocketHandle.Managed.cs,380
That code seems more complicated than what WCF is using. Just stick to the simpler model that WCF is using as we're more opinionated. They have to handle a wider range of options that a client can sepcify than WCF needs to.
This code is used to create the "invoker", basically a SocketsHttpHandler wrapped in an HttpMessageInvoker. This is where you configure credentials, proxies, certificates, certificate validation callbacks etc:
https://source.dot.net/#System.Net.WebSockets.Client/System/Net/WebSockets/WebSocketHandle.Managed.cs,280
Response validation is here:
https://source.dot.net/#System.Net.WebSockets.Client/System/Net/WebSockets/WebSocketHandle.Managed.cs,447
The code to wrap the stream in a websocket can be seen here:
https://source.dot.net/#System.Net.WebSockets.Client/System/Net/WebSockets/WebSocketHandle.Managed.cs,192

Let me know if you need help understanding any of it.

[imcarolwang]

@mconnew, this is a very detailed write-up, and I believe I understand the main idea. However, after further checking the implementation by referring to the .NET source, I got stuck trying to understand where the options parameter for setting up the HttpMessageInvoker comes from if it's to be applied in the WCF implementation. The options is of type ClientWebSocketOptions, which seems to contain many configurable options that I believe are crucial for handling HTTP requests.

Relevant .NET source: https://source.dot.net/#System.Net.WebSockets.Client/System/Net/WebSockets/WebSocketHandle.Managed.cs,248

I’ll read more carefully to better understand the process, but if you could shed some light on this, that would be great!

[mconnew]

ClientWebSocketOptions is what's passed to the WebSocket api to create a WebSocket when you aren't using the method of manually handling the initial handshake using SocketsHttpHandler. The various things passed in ClientWebSocketOptions, WCF gets from other places such as ClientCredentials (for client certificate for example), or HttpTransportBindingElement.WebSocketSettings (for sub protocol or keep alive interval for example).