The FreeBSD Diary

The FreeBSD Diary (TM)

Providing practical examples since 1998

If you buy from Amazon USA, please support us by using this link.
[ HOME | TOPICS | INDEX | WEB RESOURCES | BOOKS | CONTRIBUTE | SEARCH | FEEDBACK | FAQ | FORUMS ]
Incoming ssh is being refused 20 January 2000
Need more help on this topic? Click here
This article has 1 comment
Show me similar articles
This article details an ssh problem I had after I upgraded to OpenSSH 1.2.1 and how it was eventually solved.
The symptoms
I was trying to connect to my box from a remote host.  I was getting this:
$ ssh mydomain.org
Connection closed by foreign host.

And in the logs on my box I was finding this:

Jan 17 18:12:29 ducky sshd[29693]: 
                       refused connect from metis.otherdomain.net
Jan 17 18:12:58 ducky sshd[29696]: 
                       refused connect from metis.otherdomain.net
Jan 17 18:13:03 ducky sshd[29698]: 
                       refused connect from metis.otherdomain.net
Jan 17 18:15:54 ducky sshd[29752]: 
                       refused connect from metis.otherdomain.net

My initial place to check was /etc/hosts.allow to ensure that sshd was permitted.

sshd : ALL : allow

Yes, OK.  That's allowed.  Hmm.  What's going on?

Next I tried this:

# ssh 127.0.0.1
ssh_exchange_identification: read: No such file or directory

That's very strange.

Attempts to connect from remote hosts gave this message:

# ssh otherbox
ssh_exchange_identification: read: Undefined error: 0

I checked the OpenSSH source code and found this:

/*                 
 * Waits for the server identification string, and sends our own
 * identification string.
 */
void                    
ssh_exchange_identification()
{                                        
  char buf[256], remote_version[256];     /* must be same size! */
  int remote_major, remote_minor, i;
  int connection_in = packet_get_connection_in();
  int connection_out = packet_get_connection_out();

  /* Read other side\'s version identification. */ 
  for (i = 0; i < sizeof(buf) - 1; i++) {
     if (read(connection_in, &buf[i], 1) != 1)
        fatal("ssh_exchange_identification: read: %.100s", 
                                                 strerror(errno));

Well, that didn't mean anything to me.

Other things I tried
I compared /usr/local/etc/sshd_config on both boxes and they were identical. 

I deinstalled and reinstalled OpenSSH a few times.

I tried some debugging examples:

# /usr/local/sbin/sshd -p 1234 -d
debug: sshd version OpenSSH-1.2.1
debug: Bind to port 1234 on 0.0.0.0.
Server listening on 0.0.0.0 port 1234.
Generating 768 bit RSA key.
RSA key generation complete.

$ ssh -v localhost
SSH Version OpenSSH-1.2.1, protocol version 1.5.
Compiled with SSL.
debug: Reading configuration data /usr/local/etc/ssh_config
debug: ssh_connect: getuid 0 geteuid 0 anon 0
debug: Connecting to localhost [127.0.0.1] port 22.
debug: Allocated local port 1014.
debug: Connection established.
ssh_exchange_identification: read: No such file or directory
debug: Calling cleanup 0x805359c(0x0)

I posted messages to both OpenBSD and FreeBSD lists (thanks to those who helped).

I asked on my two favourite IRC channels (they know who they are).

The first glimmer of hope
Odinn and I were discussing this problem.  He suggested I start comparing the two boxes for differences.  We started looking at various things, configuration files, etc.  I noticed sshd was different in size.
good box bad box
-r-xr-xr-x 1 root wheel 96424 Jan 19 11:19 /usr/local/sbin/sshd -rwxr-xr-x 1 root wheel 125553 Jan 18 13:52 sshd
On Odinn's suggestion, I copied the working sshd to the failing box (after killing sshd) and tried that.

It worked! WOO HOO!  But why did it work?

I started talking to hearnia and he started me down the right track.  He suggested I compare the ldd output of each binary.

good box bad box
# ldd sshd
sshd:
libcrypto.so.1 => /usr/local/lib/libcrypto.so.1 (0x28078000)
libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x28108000)
libutil.so.2 => /usr/lib/libutil.so.2 (0x2811d000)
libz.so.2 => /usr/lib/libz.so.2 (0x28126000)
libwrap.so.2 => /usr/lib/libwrap.so.2 (0x28133000)
libc.so.3 => /usr/lib/libc.so.3 (0x2813a000)
ldd sshd
sshd:
libcrypto.so.1 => /usr/local/lib/libcrypto.so.1 (0x2807b000)
libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x2810b000)
libutil.so.2 => /usr/lib/libutil.so.2 (0x2810e000)
libz.so.2 => /usr/lib/libz.so.2 (0x28117000)
libwrap.so.7 => /usr/local/lib/libwrap.so.7 (0x28124000)
libc.so.3 => /usr/lib/libc.so.3 (0x2812b000)

The main issue here is the libwrap line.  The good box was using the tcp wrappers library which was supplied as part of the FreeBSD base system.  The bad box was using the library supplied with the tcp_wrappers port.  That's the problem!

You can also see the libraries listed here:

# ldconfig -r | grep libwrap
47:-lwrap.2 => /usr/lib/libwrap.so.2
51:-lwrap.7 => /usr/local/lib/libwrap.so.7
Description of the problem
The problem was caused by the existance of two tcp_wrapper libraries on the same box.  One library (libwrap.so.2) was supplied as part of the base FreeBSD system.  The other (libwrap.so.7) was installed as part of the tcp wrapper port.

When I built OpenSSH 1.2.1, the sshd binary was linked against libwrap.so.7.   The tcp_wrapper check would have tried to read /usr/local/etc/hosts.allow, which did not exist on my box, and would have failed.  The file descriptor was closed because of this error.   A read was later attempted with that closed file descriptor.  This read failed, which in turn, generated the error (ssh_exchange_identification: read: No such file or directory).

So here is a brief outline of what was happening during my ssh connection attempt:

  • sshd asks tcp_wrappers: "how about this connection?"
  • tcp_wrappers looks at /usr/local/etc/hosts.allow
  • the file doesn't exist
  • tcp_wrappers finds no rules allowing this connection
  • tcp_wrappers replies to sshd "no way!  this connection can't happen!"
  • sshd says, "sorry, you can't connect".
Fixing it
The first part of fixing this was to remove the tcp_wrappers port.  I first found out which version of tcp_wrappers I had installed, then removed it:
# ls /var/db/pkg | grep wrap
tcp_wrappers-7.6
# pkg_delete tcp_wrappers-7.6

The next step was to ensure hosts.allow was where it should be.  I found hosts.allow in /etc/ on my box and not in /etc/user/local/etc/.   So I didn't have to change anything there.  I had previously ensured that my /etc/inetd.conf was up to date with the latest tcp_wrapper requirements (see man inetd.conf) and my article on tcp_wrappers may also be useful.

Note that the default /etc/hosts.allow file allows *everything*.  There are many interesting examples in there and you should customize it according to your needs.

I reinstalled OpenSSH with the following steps:

  • killall -HUP sshd  (stop the existing daemon)
  • pkg_delete OpenSSH-1.2.1 (remove the existing package)
  • cd /usr/ports/security/openssh
  • make clean
  • make
  • make install

See also my article on installing OpenSSH.


Need more help on this topic? Click here
This article has 1 comment
Show me similar articles