Friday, February 11, 2011

Why SSH Connection is too slow on Linux (sometimes)?

Source: http://www.walkernews.net/2009/04/06/how-to-fix-scp-and-ssh-login-prompt-is-very-slow-in-linux/

Apparently, today is not a good day for me. But it's not too bad too as I still have enough fingers to count all these bad incidents :-)
OK, back to topic. How to find out that what cause the SSH or SCP login prompt to slowdown? How could you fix this so-called slow or delayed SSH and SCP login prompt?

This is one of the "bad incident" happened on me today – as the boss was looking at me to scp program patches to a remote Linux-based application server, the SSH login prompt took more than 1 minute to appear on screen.

To be precise, the stopwatch showed that it took exactly 1 minute and 25 seconds to display SSH login prompt!

Luckily, I do not have to Google for more than a minute to find the cause and solution :-)

What causes SCP and SSH login prompt to slowdown?

For my case, the GSSAPI authentication feature was causing the delayed SSH login prompt!

You can confirm the causes of your case by using the -v option switch. For example, the following is the verbose response of SSH login process started with -v option:
dev01 [/home/devstl]$ ssh -v appssupp@10.50.100.111 ...... ...... ...... debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-with-mic,password debug1: Next authentication method: gssapi-with-mic debug1: An invalid name was supplied Cannot determine realm for numeric host address  debug1: An invalid name was supplied Cannot determine realm for numeric host address  debug1: An invalid name was supplied Cannot determine realm for numeric host address  debug1: Next authentication method: publickey debug1: Trying private key: /home/devstl/.ssh/identity debug1: Trying private key: /home/devstl/.ssh/id_rsa debug1: Trying private key: /home/devstl/.ssh/id_dsa debug1: Next authentication method: password appssupp@10.50.100.111's password: 

How to fix SCP and SSH delayed login prompt?

The answer for my case is apparently by disabling GSSAPI authentication, which can be done in one of these three ways:

The "fix" is tested with SSH clients installed by openssh-clients-3.9p1-8.RHEL4.15 RPM file.

1) Specify the option to disable GSSAPI authentication when using SSH or SCP command, e.g.:
ssh -o GSSAPIAuthentication=no appssupp@10.50.100.111 

2) Explicitly disable GSSAPI authentication in SSH client program configuration file, i.e. edit the /etc/ssh/ssh_config and add in this configuration (if it's not already in the config file):
GSSAPIAuthentication no 

3) Create a file called config in .ssh directory of respective user home directory (or whichever user home directory that need to get rid of this show login prompt). For example, edit /home/devstl/.ssh/config (create the config file if it's not currently exist) and add in the GSSAPIAuthentication no option.

1) /etc/ssh/ssh_config is a global SSH client configuration file that affects all system users who are using SSH client programs.

2) /home/devstl/.ssh/config is local SSH client configuration file that only affects the user account called devstl. Whatever SSH client options specified in this local file overwrite the options stated in global SSH client configuration file.

After disabling GSSAPI authentication, SSH login prompt is back to "normal" now:
dev01 [/home/devstl]$ ssh -v appssupp@10.50.100.111 ...... ...... ...... debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-with-mic,password debug1: Next authentication method: publickey debug1: Trying private key: /home/devstl/.ssh/identity debug1: Trying private key: /home/devstl/.ssh/id_rsa debug1: Trying private key: /home/devstl/.ssh/id_dsa debug1: Next authentication method: password appssupp@10.50.100.111's password: 

As you can see, the SSH login is not currently authenticated via public key cryptography method, which I've to fix it later :-(

OR
If your login times are really high, it may be that reverse DNS is not working correctly. We have an ISP whose DNS servers sometimes don't respond to reverse DNS queries. It was a bit of a puzzle because it has worked for a long time. Our hunch is that the recent DOS attacks have made name resolution a little fragile lately. The symptom shows up in the logs:

tail /var/log/secure 

We have keys set up, but notice that there is a fifteen second delay from accepting the key to opening a session:

Feb  3 09:48:45 main sshd[9692]: Accepted publickey for root from 1.6.4.2 port 57559 ssh2 Feb  3 09:49:00 main sshd[9692]: pam_unix(sshd:session): session opened for user u1 by (uid=0) 

The fix is to either add the IP address to /etc/hosts, or modify your sshd_config file (for us the path is /etc/ssh/sshd_config) and set UseDNS to no:

#ShowPatchLevel no UseDNS no #PidFile /var/run/sshd.pid #MaxStartups 10 

Restart sshd for the change to take effect:

# /etc/init.d/sshd restart Stopping sshd:                                             [  OK  ] Starting sshd:                                             [  OK  ] # 

Now we get a quick login:

Feb  3 10:06:49 main sshd[12160]: Accepted publickey for root from 1.6.4.2 port 57528 ssh2 Feb  3 10:06:49 main sshd[12160]: pam_unix(sshd:session): session opened for user u1 by (uid=0) 


0 Comments: