The FreeBSD Diary

A new version of FreshPorts came out on Tuesday. I would be surprised if most of you reading this didn't notice. There are a few reasons why it came in very quietly.

1. No real visible changes - you wouldn't notice any changes if all you looked at was the front page. Well, you might notice the page loads a bit faster...
2. I didn't send out an announcement as I normally do. That's because of a blacklist problem, which was not of my own doing... I'll tell you more about that later and how I've gotten around this [hopefully] temporary problem.

This new release was a long time coming. It's been just over a year since the last release. It was a long staggered beta process. I need to shorten that duration. I think volunteers would work better on this type of project if there is a shorter testing period. I kept coming up with new features to add and that would affect several other items. That's called scope creep.

Part of the problem was I kept putting off the implementation date. That's a luxury not found in commercial projects. Time lines do not slip so easily if money is involved. Nor is scope creep tolerated like it was on this release.

Luckily, I got good help from volunteers in testing and coding, but I'd like more. If you are interested in helping to develop FreshPorts, you know where to find me. Many people have come up with great ideas, but as is the case with many projects, there's nobody around to do the work.

I don't think I'll be accused of self-promotion if I say the upgrade went rather well. I started about 6:34 am EST and the web site was back up for normal operations at 10:34 am EST. I didn't plan for a four hour window. My test upgrades normally took about 15 minutes. But those tests consisted only of

1. pg_dump freshports
2. copy tarball to home where I'm running 7.3.2
3. import the dumped schema/data
4. run the PostgreSQL contrib/adddepend script which tidies up 7.2 schemas with respect to serial, foreign keys, primary keys etc.
5. make change to the database for the new features (easily done...
6. Then run a to refresh the statistics for the index, query optimizer, etc.

These test runs did not include upgrading PostgreSQL to 7.3.2. That's pretty straight forward and can be done in parallel. However, when upgrading a server, it's better to do things serially. There is much less room for error when you are concentrating on just one task at a time.

I also had to reconfigure the email processing daemon to reflect a much improved strategy for handling multiple versions of FreshPorts on a single machine, each running under the same user. That took me 20 minutes or so... setting up the new directories, ensuring procmail was set up in the new method.

The truth be told, I had finished everything by 8:11 am. But I found I'd forgotten to populate a new table, and that was fixed by about 8:39 am. But by 8:51, I'd noticed a problem with virtual categories which results in duplicate entries in various situations. That took until 10:05^* to fix. This was solved by selecting virtual categories and physical categories using different techniques and then using a UNION to combine the results. The categories page uses this technique.

^*Those wondering if I keep detailed notes, with times, duration, etc, of each step, no, I don't do that. I'm getting these times from /var/log/httpd-error.log.

It appears my IP address has been lumped in with some nasty company. There's spammers around me. To be fair to my provider, who has always given me good service and does not tolerate spam, this new IP block was inherited from others. That wouldn't be so bad, if one or two addresses were blacklisted. I could still continue to function. But it seems that some blacklist policies result in the entire IP block being listed instead of the just the ones used by the spammers. Not just a /24. This is a whole /20 which has been blacklisted.

Unfortunately, that means that the sins of others are being paid for by the innocent. Namely, the subscribers to the FreshPorts, FreshSource, and FreeBSD Diary mailing lists. Not to mention my other domains.... Luckily, I do not use that server for my outgoing mail, but it is listed as a secondary mail exchange server. I'm sorry that many of you have not been getting your FreshPorts notifications. I'm getting between 50 and 60 bounces a day as various mail servers around the world are rejected the FreshPorts messages. And I'm sure that still more of you would have the notification tagged as spam by your mail filtering. Such is the power of the blacklist.

I started off by forwarding the bounces by hand, but when it got to be 50 or more bounces a night, I gave up. Now I'm just deleting the bounces and I'm pursuing other options. I can't ask all of you to add me to your whitelist, but thank you to those that have. My hosting provider is in the process of getting a new IP block and hopefully things will change very soon. But in the meantime, the situation has gone beyond a novelty.

For those interested have a look at the Spamhaus listing. I'd like to get off the listing, but my attempts have failed. There's also a /20 listing at rfc ignorant.

I am fortunate enough to have more than one server out there. I'm using another mail server to send out the FreshPorts notifications. This idea came to me late Tuesday night and by Wednesday afternoon, I'd set up a stunnel, reconfigured my mailing software, and notifications are ready to go out tonight. Here's hoping...