Troubleshooting ssh connection problems

1 – server identification keys changed

If the server to which we need to connect has been reinstalled, it is normal to receive a warning when we reconnect to it. The program used for the connection (client) warns us that the identity of the server has changed and does not correspond to the one previously stored through the cryptographic identification keys.

If we know that there has actually been a change on the server, we can allow the client to accept the new identification keys.

Depending on the program used for the connection, the warning message and the ways of accepting the new keys are different. Let’s look at the different cases.

Command line session via ssh

Usually, when the server’s identification keys differ from those stored by the client, ssh refuses to connect and displays a message like this:

[alice@minipc ~]$ ssh alice@superpc.pd.infn.it
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:PiDF5OCsdmvJDRe0brmQXArcOl1wmtitS6pXg34hsYI.
Please contact your system administrator.
Add correct host key in /home/alice/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/alice/.ssh/known_hosts:3
RSA host key for superpc has changed and you have requested strict checking.
Host key verification failed.

In this case, to accept the new identification key, you must manually delete the old one from the known_hosts file. The message indicates the path of the file (/home/alice/.ssh/known_hosts in the example) and the line to delete (line 3 in the example).

To delete the line, we can manually edit the file or, more simply, use the command

ssh-keygen -R <server name>

In our example the command will be

ssh-keygen -R superpc

At this point, when we try to connect again, the program will warn us that the server’s identity is unknown, but will allow us to continue the connection and accept the new key.

Connection with Bitvise SSH client or with MobaXterm

Bitvise and MobaXterm automatically handle the change of the server identification key: if the key is modified, a window will appear with a warning and the option to accept or reject the connection (see images below). By accepting the connection, the new key will be stored in place of the old one

2 – Incompatibility of cryptographic algorithms between client and server

Over the years, some of the encryption algorithms used by ssh have changed: older ones have been found to be vulnerable and have been replaced by newer, more robust algorithms. As a result it can happen that the client and server cannot negotiate a common encryption algorithm and therefore the connection cannot be established. If the client used for the connection is newer than the server, then in many cases it is possible to force it to use one of the algorithms available on the server. Conversely, when the server is much newer than the client, there is no other solution than to upgrade the client or use an intermediate server that accepts connections from the old client and is able to connect to the desired server.

Let’s see some examples.

Recent client, old server

[alice@newpc ~]$ ssh oldpc.pd.infn.it
Unable to negotiate with 192.168.0.100 port 22: no matching host key type found.
Their offer: ssh-rsa,ssh-dss

Here you can see that the old server only supports ssh-rsa and ssh-dss algorithms for the host key. You can force the client to use them via the HostKeyAlgorithms option:

[alice@newpc ~]$ ssh -o HostKeyAlgorithms=+ssh-rsa,ssh-dss oldpc.pd.infn.it
alice@oldpc.pd.infn.it's password:

In other cases, the client and server fail to negotiate a method to exchange keys used to encrypt the connection:

[alice@newpc ~]$ ssh veryoldpc
Unable to negotiate with 192.168.0.77 port 22: no matching key exchange method found.
Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

This time, using the KexAlgorithms option we can force the client to use one of the obsolete algorithms supported by the server:

[alice@newpc ~]$ ssh -o KexAlgorithms=+diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 veryoldpc
Unable to negotiate with 192.168.0.77 port 22: no matching host key type found.
Their offer: ssh-rsa,ssh-dss

Here we see that we have solved the cryptographic key exchange problem, but now we have the host key type incompatibility problem, already seen in the previous case, which we know how to solve with the HostKeyAlgorithms option. So using both options we can connect:

[alice@newpc ~]$ ssh -o KexAlgorithms=+diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 -o HostKeyAlgorithms=+ssh-rsa,ssh-dss veryoldpc
alice@veryoldpc.pd.infn.it's password:

Another typical error that can be encountered when using a recent client to connect to an old server is

Bad server host key: Invalid key length

The reason is that the server uses an RSA-type identification key that is too small in length (usually 1024 bit instead of 2048 or more).

In such cases, we can use the -o PubkeyAcceptedAlgorithms=ssh-dss option to force the client to use DSS keys instead of RSA (provided the server supports them). On RedHat and derivative systems, the ssh client also supports the -o RSAMinSize=1024 option to lower the minimum allowed length of the ssh RSA key.

Once we have found the options needed to connect to a certain host, we can insert them into the ~/.ssh/config configuration file so that we do not have to write them every time. In the file we can indicate the options needed for each host like this:

Host oldpc.pd.infn.it
    HostKeyAlgorithms +ssh-rsa,ssh-dss

Host veryoldpc.pd.infn.it veryoldpc otherpc.mydomain.org KexAlgorithms +diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 HostKeyAlgorithms +ssh-rsa,ssh-dss

After the Host directive, you can enter multiple host names separated by spaces and even the same host with and without domain, so that the directive is used regardless of how we write the host name on the command line.

More information about the options to provide to the ssh command or to insert into the configuration file can be found in the ssh_config(5) man page:

man ssh_config

Old client, new server

The opposite situation can also occur if we use an old client to connect to a server updated with the latest versions of ssh:

[alice@oldpc ~]$ ssh newpc.pd.infn.it
no kex alg

In this case we don’t have many options: the server refuses the connection because our client uses obsolete algorithms. If we don’t have a way to reconfigure the server to allow access with old and less secure algorithms (which is not advisable anyway), the only way to solve the problem is to update the client; or, if possible, use an intermediate server that accepts the connection from our old client and in turn has a client recent enough to be accepted by the server we want to connect to.