Troubleshoot SSL/TLS mutual authentication with openssl s_server
Each secure communication between a browser and a HTTP server begins with a SSL/TLS handshake.
What is a SSL/TLS handshake? Its the process that kicks off a communication session that uses SSL or TLS as encryption and authentication protocol. The handshake is for instance responsible for negociating the algorithms to use for the encryption between the client and server sides.
During a SSL/TLS handshake between a browser and a HTTP server, a lot of steps can occur. For instance in TLS v1.2:
- Client Hello
- Server Hello
- Server Certificate
- Server Key Exchange
- Server Hello Done
- Client Key Exchange
- Change Cipher Spec
- …
If an error occurs during this complex process, messages displayed by the browser might be far from straightforward (here with my french Firefox and a smartcard):
In the present case, the browser couldn’t correctly complete a TLS handshake using the smartcard certificates (incompatible algorithms required from the server). The “PKCS11” part of the error indicates smartcard access problem.
But what part of the negociation causes this message ? How can we find information about the algorithms selected for the connection ? What tools could we use to investigate ?
OpenSSL: THE SSL library
The vast majority of SSL/TLS tools is based on the openssl open source library. Especially,the ngx_http_ssl_module module for Nginx and apache mod_ssl module for httpd are based on this library. The following examples are based on Apache httpd.
More than just a library, openssl comes with many tools for SSL tinkering. For instance, to troubleshoot a server connection, the openssl s_client command is a must. Indeed, s_client acts as a command-line HTTPS client, allowing analysis of SSL server responses. There is also openssl s_server command for creating an SSL server. We’ll explore this command in the next section.
openssl s_server: how to see the result of the SSL handshake
The s_server option of openssl allows to quickly start a small SSL server with which the browser will be able to communicate.
Here is a sample command that launches an openssl server on port 443 that logs error information:
openssl s_server -key private/localhost.key -cert localhost.crt -accept 443 -state \
-www -CAfile ca/ca.pem -verify 2 -debug -security_debug_verbose -tls1_2 -msg
The table below describes succinctly the options that were passed to the above command:
| Option | Description |
|---|---|
| key | the private key to use |
| cert | the certificate to use |
| accept | the server listening port |
| CAfile | the certification authority used to validate client certificates |
| state | displays SSL session states in the console during the server runtime |
| www | returns connection informations to the client through an HTML page |
| verify | asks the client for a certificate (mutual authentication) and sets the verification depth of the chain |
| debug | displays in the console debug infos and hexdumps |
| security_debug_verbose | displays debug infos of the TLS/SSL framework |
| tls1_2 | uses only TLS 1.2 for the transaction |
| msg | dumps all protocol messages in the console with hexdumps |
You will find more details on the manual pages for openssl, here for the 1.1.1 version
The CAfile references the certification authority for your PKI, if you need more details on how to create a PKI (Private Key Infrastructure) with openssl, good articles on the subject are available.
The main benefit of testing with openssl s_server is the www option which enables the browser to display both handshake and transaction details.
Opening https://localhost:443 on a browser shows the SSL transaction details in the browser window. The following screenshot show an example of such transaction.
The details given in the console and web page can help to troubleshoot the problems you are facing. The next section show a concrete example of error diagnosis.
Example: select the signature algorithms to activate for a handshake
Getting back to the beginning of this post, the PKC11 error shown in the first screenshot comes from a real case. Here are the steps of the error analysis:
- The error was displayed while using a smartcard as certificate provider in firefox
- It was only present while connecting to a server with TLS v1.2 using the httpd mod_ssl module compiled against openssl v.1.1.1, there was no error with openssl v.1.1.0
Spawning a server with the following command for both versions of openssl allowed to display and compare the algorithms negotiated with Firefox in TLS v.1.2 (the arguments -tls1_2, -tls1 and -tls1_1 are also supported):
openssl s_server -key private/localhost.key -cert localhost.crt -accept 443 -state \
-www -CAfile ca/ca.pem -verify 2 -debug -security_debug_verbose -tls1_2 -msg
Finally, we were able to get rid of the error by specifying the list of authorized algorithms for openssl. In our case the RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512 signature algorithms were causing trouble with our configuration, hence we removed them from the list:
openssl s_server -key private/localhost.key -cert localhost.crt -accept 443 -state \
-www -CAfile ca/ca.pem -verify 2 -debug -security_debug_verbose -tls1_2 -msg \
-sigalgs ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA1:RSA+SHA1
Conclusion
In this post, we have used an openssl tool called openssl s_server with the www option for analyzing SSL/TLS handshake errors. It allows browsers that connect to this server to show debug information about SSL/TLS exchanges. This post also showed a concrete example about how to diagnose and fix SSL/TLS handshake errors.
