How To Make Unplanned Service Improvements

Posted September 21 2001

Printer-friendly version

The last few days have been very entertaining here at the magnificent ClarionMag corporate headquarters. On Tuesday, a day I took off to do some needed work on the house, the nimda worm hit the 'net, and caused some service interruptions to Clarion Magazine. That in itself wasn't a big problem, just another in a series of network-clogging Windows server viri that I couldn't personally do anything about (since I don't run any Windows web servers).

On Thursday morning, however, I lost the ability to query my ISP's name servers. Actually that's not completely true. I could query the name servers, but I never got any replies. I could however ping those same machines. DNS queries went out, DNS responses got dropped on the way back. At least that's how it seemed. The end result was that I couldn't surf anywhere, or get mail (without resorting to IP numbers), or do a host of other everyday tasks. Some ClarionMag users were also affected, although others could still get to the web site.

The nimda worm complicated things, although its role isn't completely clear to me yet. But the main problem was the dropped packets, and that apparently had to do with my running two subnets on the same wire. About a year ago my ISP was purchased by Group Telecom; one of the happy consequences of that acquisition was GT's plan to upgrade the local infrastructure for business Internet customers. I was to be moved to a new network layer, which meant a new block of IP addresses. I was particularly concerned that this not result in any interruption of service to ClarionMag customers, so my local GT tech suggested we temporarily run both the new and old subnets at the same time, allowing me to migrate my servers across without downtime. For full control over the process, I elected to set up my own primary DNS.

That migration would probably have taken place by now, but a while ago I ran into some unusual problems with zone transfers, and, oddly enough, DNS replies getting lost in transmission.

Whatever indication that was of the problem to come, Thursday morning everything fell apart. I spent a lot of time on the phone with my ISP, but at best I was able to get replies from their secondary DNS, but not the primary. Both were functioning normally from other vantage points, just not from my subnets. The name server I couldn't get replies from was on the same Class B network as my new subnet, while the name server I could get replies from was on the same Class A network as my old subnet.

This seemed like a clear indication of a routing problem. I elected to change over completely to the new subnet, and have my ISP propagate the DNS changes overnight. The move to the new subnet cleared up the dropped DNS packets, and I was able to get full Internet access again.

The new IP block in itself doesn't represent a service improvement, but is the necessary first part of moving to the new network. The actual cutover to the new layer is now just a matter of switching some wires around, and I'm assured should result in a service interruption of only 15 minutes or so. Barring any unforeseen developments, I will be able to give Clarion Magazine subscribers some advance notice of that event.

Thank you again for your patience.

Dave Harms

Publisher