Troubleshoot SSL/TLS mutual authentication with openssl s_server

4 minute(s) read

Each secure communication between a browser and a HTTP server begins with a SSL/TLS handshake.

What is a SSL/TLS handshake? Its the process that kicks off a communication session that uses SSL or TLS as encryption and authentication protocol. The handshake is for instance responsible for negociating the algorithms to use for the encryption between the client and server sides.

During a SSL/TLS handshake between a browser and a HTTP server, a lot of steps can occur. For instance in TLS v1.2:

  • Client Hello
  • Server Hello
  • Server Certificate
  • Server Key Exchange
  • Server Hello Done
  • Client Key Exchange
  • Change Cipher Spec

If an error occurs during this complex process, messages displayed by the browser might be far from straightforward (here with my french Firefox and a smartcard):


In the present case, the browser couldn’t correctly complete a TLS handshake using the smartcard certificates (incompatible algorithms required from the server). The “PKCS11” part of the error indicates smartcard access problem.

But what part of the negociation causes this message ? How can we find information about the algorithms selected for the connection ? What tools could we use to investigate ?

OpenSSL: THE SSL library

The vast majority of SSL/TLS tools is based on the openssl open source library. Especially,the ngx_http_ssl_module module for Nginx and apache mod_ssl module for httpd are based on this library. The following examples are based on Apache httpd.

More than just a library, openssl comes with many tools for SSL tinkering. For instance, to troubleshoot a server connection, the openssl s_client command is a must. Indeed, s_client acts as a command-line HTTPS client, allowing analysis of SSL server responses. There is also openssl s_server command for creating an SSL server. We’ll explore this command in the next section.

openssl s_server: how to see the result of the SSL handshake

The s_server option of openssl allows to quickly start a small SSL server with which the browser will be able to communicate.

Here is a sample command that launches an openssl server on port 443 that logs error information:

openssl s_server -key private/localhost.key -cert localhost.crt -accept 443 -state \
        -www -CAfile ca/ca.pem -verify 2 -debug -security_debug_verbose -tls1_2 -msg

The table below describes succinctly the options that were passed to the above command:

keythe private key to use
certthe certificate to use
acceptthe server listening port
CAfilethe certification authority used to validate client certificates
statedisplays SSL session states in the console during the server runtime
wwwreturns connection informations to the client through an HTML page
verifyasks the client for a certificate (mutual authentication) and sets the verification depth of the chain
debugdisplays in the console debug infos and hexdumps
security_debug_verbosedisplays debug infos of the TLS/SSL framework
tls1_2uses only TLS 1.2 for the transaction
msgdumps all protocol messages in the console with hexdumps

You will find more details on the manual pages for openssl, here for the 1.1.1 version

The CAfile references the certification authority for your PKI, if you need more details on how to create a PKI (Private Key Infrastructure) with openssl, good articles on the subject are available.

The main benefit of testing with openssl s_server is the www option which enables the browser to display both handshake and transaction details.

Opening https://localhost:443 on a browser shows the SSL transaction details in the browser window. The following screenshot show an example of such transaction.

SSL transaction details

The details given in the console and web page can help to troubleshoot the problems you are facing. The next section show a concrete example of error diagnosis.

Example: select the signature algorithms to activate for a handshake

Getting back to the beginning of this post, the PKC11 error shown in the first screenshot comes from a real case. Here are the steps of the error analysis:

  • The error was displayed while using a smartcard as certificate provider in firefox
  • It was only present while connecting to a server with TLS v1.2 using the httpd mod_ssl module compiled against openssl v.1.1.1, there was no error with openssl v.1.1.0

Spawning a server with the following command for both versions of openssl allowed to display and compare the algorithms negotiated with Firefox in TLS v.1.2 (the arguments -tls1_2, -tls1 and -tls1_1 are also supported):

openssl s_server -key private/localhost.key -cert localhost.crt -accept 443 -state  \
         -www -CAfile ca/ca.pem  -verify 2 -debug -security_debug_verbose -tls1_2 -msg

Finally, we were able to get rid of the error by specifying the list of authorized algorithms for openssl. In our case the RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512 signature algorithms were causing trouble with our configuration, hence we removed them from the list:

openssl s_server -key private/localhost.key -cert localhost.crt -accept 443 -state \
        -www -CAfile ca/ca.pem  -verify 2 -debug -security_debug_verbose -tls1_2 -msg \ 


In this post, we have used an openssl tool called openssl s_server with the www option for analyzing SSL/TLS handshake errors. It allows browsers that connect to this server to show debug information about SSL/TLS exchanges. This post also showed a concrete example about how to diagnose and fix SSL/TLS handshake errors.

Written by

Gerben Castel

Father of 3, Software Architect, forever developer and tech enthusiast