The Internet is a modern wonder. However if you have to run servers on it, often the biggest wonder is that it works at all. There are so many things that can go wrong with the transport of data from one place to another.
Yesterday, several of the NZ networks were having problems communicating with our server. There was a routing problem in San Diego where packets coming back from the server were routed into oblivion. So the site appeared to be offline. This happened if you were (like I was) on the Orcon, IHug, and probably other networks. Other local networks were feeding in and getting responses from different routes.
I detected it in the morning and informed the hosting company and my ISP by voice message and e-mail. Unfortunately I’ve just started a new job where I’m doing a lot of design work with other people that occupied my full day. So I wasn’t able to have a good look at it until late in the afternoon.
I found my e-mails to the hosting company still sitting in the outbox because the mail server wasn’t able to talk to the hosting company. By resending those e-mails using a g-mail account I was able to contact hosting support who hadn’t been able to detect the problem (my rushed voice message in the morning was a bit short of required information). They started to look for the problem router where the routing table was sending returning packets to oblivion. They also started to work with to give the site an IP that would bypass the affected route.
Meanwhile I’d managed to talk to the server using putty from my iPhone. By doing a traceroute out from the server, and a traceroute from my home system going to the server we managed to pinpoint the router with a problem. The hosting support then organised a reroute around that problem system.
It got implemented last night. So everything should be back online this morning, and I’m able to write you this post. Looking at the traffic from yesterday, it clipped about 30-40% off the usual load of page views. As well as the people who weren’t able to read or comment, some of our authors were unable to put up their posts.
A quiet day, that is unlikely to be repeated today. It looks like the government is providing lots of material for us and you to write about.
Powered by WPtouch Mobile Suite for WordPress
