The FreeBSD Diary

NOTE: I wrote this article on 23 November, as things were happening, and as I worked on the problem.

This morning was not a good start. First, all my IRC connections had died. Checking with Netsaint, all my boxes were offline. Great! My gateway is having problems. I ssh'd to my gateway. ifconfig showed it had a public IP address. I could ping that IP address. netstat -nr showed I had a gateway. I could ping that. I could also ping google.ca. OK, the gateway works. Next, I checked my browser, yes, it could get to Google. OK, so what's wrong?

I could not ssh to any of my boxes in New Zealand, Vancouver, Ottawa, or San Jose. They were all affected...

I got onto IRC and started chatting about this. Suggestions where made that I got owned. I doubted it. Netsaint showed all the problems started at the same time (give or take a few minutes). I tried ssh'ing to another NZ-based box, and I got in. OK, so all is not bad. That was one of the boxes listed as a problem by Netsaint.

OK, what's up?

Actually, don't blame Canada. Blame DNS. All of these problems are explained by a DNS change. The public IP address on my gateway changed. Lovely!

This explains everything. All the problems noted by Netsaint were related to Netsaint client calls... Ping and http were fine. I missed that part. The firewall rules on the boxes restrict incoming connections by IP address. That explains why Netsaint was complaining. Here is what it said for the NYI box:

Notice that ping and http are fine. These services are checked directly by Netsaint. The other services are checked by connecting to a port on the remote box and talking to netsaint_statd. However, I have configured netsaint_statd to speak only to a specific IP address; one that I no longer have access to.

It is because I have a dynamic IP address that I have been using the dns/noip port since I started with Rogers. Why did I go with Rogers? I called them on June 30. They were here and had installed my connection within 18 hours. In fact, they showed up on a statutory holiday, and took less then 15 minutes to get me hooked up. I loved that. I tried to get DSL, but they said it would take more than a week. And I'd have to use PPPoE. With cable, I have direct ethernet

Despite having a dynamic IP address, this is the first time it has changed. I was never sure if noip would work or not. Well, it worked. Invisibly. Witness:

$ host bast.unixathome.org
bast.unixathome.org is an alias for dlangille.no-ip.org.
dlangille.no-ip.org has address 74.104.199.163
bast.unixathome.org is an alias for dlangille.no-ip.org.
bast.unixathome.org is an alias for dlangille.no-ip.org.

Yep, that's right. All my hostnames that point to my servers at home use that hostname:

$ host beta.freshports.org
beta.freshports.org is a nickname for dlangille.no-ip.org
dlangille.no-ip.org has address 74.104.199.163

That's good. I don't have to run around and update all my DNS. That investment has paid off.

You see that FreshPorts BETA has the right IP address. That's good. But nobody could get to the websites. Why? My firewall rules. They have the external IP address hard-coded. That would be OK if that was static, but it's not. Here's what I had:

external_addr="70.26.229.230"

In the ruleset, you'll see this:

rdr on $ext_if proto tcp from any to $external_addr port http -> $webserver

That's a problem.

How do you avoid hardcoding an IP address that changes? pf can handle that. mux, brix, and dnns all pointed out a couple of solutions. Namely:

• self - From http://www.openbsd.org/faq/pf/tables.html:

  Specifying Addresses In addition to being specified by IP address, hosts may also be specified by their hostname. When the hostname is resolved to an IP address, all resulting IPv4 and IPv6 addresses are placed into the table. IP addresses can also be entered into a table by specifying a valid interface name or the self keyword. The table will then contain all IP addresses assigned to that interface or to the machine (including loopback addresses), respectively.

  OK, that's useful. Using a hostname is good. But when the IP address changes, the ruleset is not updated, so it is not quite what I need.

• ($ext_if) - From http://www.openbsd.org/faq/pf/filter.html:

  The name of a network interface or group in parentheses ( ). This tells PF to update the rule if the IP address(es) on the named interface change. This is useful on an interface that gets its IP address via DHCP or dial-up as the ruleset doesn't have to be reloaded each time the address changes.

  Yes, that is just what I need for my home gateway!

Changing the rule set was easy. I removed the external_addr="70.26.229.230" entry. I then replaced all instances of $external_addr with ($ext_if). To test those rule changes:

# pfctl -n -f /etc/pf.rules
#

And then to invoke those new rules:

# pfctl -f /etc/pf.rules
#

With those changes, http://beta.freshports.org/ was back online. And so were all the other websites I have at home. Similarly, my mail started flowing in as well.... A good start. Now, let's go look at those servers.

Now that I have the beta websites working and mail flowing, now I need to gain access to my servers. All the servers restrict incoming ssh to specific IP addresses. One of them has a serial console connection. I've been able to ssh to each box and update the firewall rules and reconfigure netsaint_statd.

The changes to the firewall rules consisted of updating one line in the PF configuration file. I had used a macro to define the IP address of my home box, so updating the rules was quite painless, once I could get to that box.

I had to change this line in netsaint_statd:

my @allowed_hosts = ("74.104.199.163");

I had to kill and restart that netsaint_statd. It is not set up to take a HUP and /usr/local/etc/rc.d/netsaint_statd.sh does not take a stop or restart argument.

I also updated /etc/hosts and /etc/hosts.allow

When I visited a few other hosts, I found a problem. Can you see it?

my @allowed_hosts; # = ("10.55.0.18");

For starters, that's the wrong IP address. More importantly, there is a comment in there.. OUCH. I corrected that too, on a number of hosts.

Just after I got my mail flowing, and websites back online, I started checking my email. In my logs, I found this:

Nov 23 22:31:59 bast dhclient: New IP Address (fxp0): 74.104.199.163
Nov 23 22:31:59 bast dhclient: New Subnet Mask (fxp0): 255.255.255.128
Nov 23 22:31:59 bast dhclient: New Broadcast Address (fxp0): 255.255.255.255
Nov 23 22:31:59 bast dhclient: New Routers (fxp0): 74.104.199.129
Nov 23 22:32:20 bast noip2[76688]: execl /usr/home/dan/bin/ip-updated.sh failed (Permission denied)

Well! Not that it would have helped me, because I didn't check my email first thing, but there's my first, yet unfound until now, clue about the problem.

I fixed the problem listed above:

# ls -l /usr/home/dan/bin/ip-updated.sh
-rw-r--r--  1 dan  dan  76 Jul  1  2005 /usr/home/dan/bin/ip-updated.sh
# chmod +x /usr/home/dan/bin/ip-updated.sh
# cat /usr/home/dan/bin/ip-updated.sh
!#/bin/sh
echo 'IP address has been updated' | mail dan@xeon.example.org
#

NOTE: 29 November 2006: As I type this, I have no outside connectivity. I suspect a routing problem outside my gateway. I checked the logs, and found what I thought was another IP address change. I was wrong. I was looking at the incident from 23 Nov. However, all is not lost. I did notice that the above permission problem was caused by a typo, not a permission issue. !#/bin/sh should be #!/bin/sh. I just called my ISP. There is a problem and they're working on it.

Where is that script called from? That took me a few minutes to track down. I looked at /usr/local/share/doc/noip/README.FIRST:

# /usr/local/bin/noip2 -S
1 noip2 process active.

Process 1030, started as /noip2
Using configuration from /usr/local/etc/no-ip2.conf
Last IP Address set 74.104.199.163
Account someone@example.com
configured for:
        host  dlangille.no-ip.org
Executing /usr/home/dan/bin/ip-updated.sh upon successful update.
Address check every 1 minute, directly connected via /dev/fxp0.

NOTE: email addresses in this section have been obscured.

For what it is worth, my friend Eric Rosenquist also had an IP address change last night. At 20:33:11. He is also with Rogers.

What did I learn?

1. I improved my firewall rules so they did not refer to IP addresses that can change.
2. I corrected problems with my netsaint_statd configuration on a few clients.
3. I fixed my no-ip.org configuration so that email notifications work.

I restrict ssh logins to specify IP addresses. That is why I could not ssh in from home. I also allow ssh from a few servers that do not restrict incoming ssh. Coincidentally, one of those servers was offline at the time.

In short, it wasn't a disaster. Nothing was owned. If IP address changes happened more regularly, then I think I would have known sooner what the problem was. Let's see what happens next time. I'm better prepared now.

One last note: As I went to publish this article, I realized another configuration file that needed to be updated: stunnel.conf. I use stunnel to create a secure tunnel between my cvs server and my webserver. This enables cvsup to work in private and without exposing the service to snooping eyes. For what it's worth, the configuration file now uses a hostname, not an IP address. That means I will just need to restart the service in future.