A quiet day

Written By: - Date published: 8:34 am, June 4th, 2010 - 6 comments
Categories: admin - Tags:

Sysop

The Internet is a modern wonder. However if you have to run servers on it, often the biggest wonder is that it works at all. There are so many things that can go wrong with the transport of data from one place to another.

Yesterday, several of the NZ networks were having problems communicating with our server. There was a routing problem in San Diego where packets coming back from the server were routed into oblivion. So the site appeared to be offline. This happened if you were (like I was) on the Orcon, IHug, and probably other networks. Other local networks were feeding in and getting responses from different routes.

I detected it in the morning and informed the hosting company and my ISP by voice message and e-mail. Unfortunately I’ve just started a new job where I’m doing a lot of design work with other people that occupied my full day. So I wasn’t able to have a good look at it until late in the afternoon.

I found my e-mails to the hosting company still sitting in the outbox because the mail server wasn’t able to talk to the hosting company. By resending those e-mails using a g-mail account I was able to contact hosting support who hadn’t been able to detect the problem (my rushed voice message in the morning was a bit short of required information). They started to look for the problem router where the routing table was sending returning packets to oblivion. They also started to work with to give the site an IP that would bypass the affected route.

Meanwhile I’d managed to talk to the server using putty from my iPhone. By doing a traceroute out from the server, and a traceroute from my home system going to the server we managed to pinpoint the router with a problem. The hosting support then organised a reroute around that problem system.

It got implemented last night. So everything should be back online this morning, and I’m able to write you this post. Looking at the traffic from yesterday, it clipped about 30-40% off the usual load of page views. As well as the people who weren’t able to read or comment, some of our authors were unable to put up their posts.

A quiet day, that is unlikely to be repeated today. It looks like the government is providing lots of material for us and you to write about.

6 comments on “A quiet day ”

  • Comments are now closed

  1. Name 1
    4 June 2010 at 9:08 am

    I’m with Slingshot and had no problem connecting to your servers at any time yesterday.

    A traceroute to you this morning records 22 external hops:

    Tracing route to 69-55-230-31.in-addr.arpa.johncompanies.com [69.55.230.31]
    over a maximum of 30 hops:

    1 1 ms <1 ms <1 ms 192.168.88.1
    2 713 ms 23 ms 17 ms 172.16.4.1
    3 18 ms 88 ms 56 ms 172.16.2.1
    4 316 ms 95 ms 92 ms p32.argent.cust.commverge.net.nz [202.180.81.32]
    5 30 ms 51 ms 81 ms vlan93-cpcak3-e1.tranzpeer.net [202.180.76.68]
    6 60 ms 137 ms 890 ms vlan7-cpcak3-s1.tranzpeer.net [202.180.81.49]
    7 110 ms 95 ms 53 ms gi10-0-0-405.gw1.akl1.asianetcom.net [203.192.167.29]
    8 29 ms 84 ms 52 ms ge-0-0-0-0.gw1.akl2.asianetcom.net [203.192.185.45]
    9 171 ms 209 ms 174 ms po1-1-0.gw3.lax1.asianetcom.net [203.192.185.62]
    10 249 ms 306 ms 188 ms te7-6.mpd01.lax05.atlas.cogentco.com [154.54.11.9]
    11 196 ms 189 ms 430 ms te2-1.ccr02.lax05.atlas.cogentco.com [154.54.28.66]
    12 239 ms 211 ms 198 ms te4-2.ccr01.lax01.atlas.cogentco.com [154.54.6.229]
    13 251 ms 173 ms 210 ms te4-2.ccr01.lax04.atlas.cogentco.com [154.54.24.70]
    14 214 ms 195 ms 178 ms audio-visual-services-group-inc.demarc.cogentco.com [38.104.82.234]
    15 267 ms 223 ms 195 ms border11.po2-20g-bbnet1.lax.pnap.net [216.52.255.104]
    16 173 ms 218 ms 189 ms core2.ge2-3.ocy002.pnap.net [64.94.109.6]
    17 220 ms 201 ms 207 ms border1.po1-2g-bbnet1.ocy002.pnap.net [216.52.96.34]
    18 190 ms 192 ms 232 ms core2-g2-1.ocy002.pnap.net [70.42.209.10]
    19 240 ms 172 ms 234 ms border3.ge2-0-bbnet1.sdg.pnap.net [63.251.127.7]
    20 461 ms 211 ms 187 ms castleaccess-2.border3.sdg.pnap.net [63.251.125.10]
    21 240 ms 313 ms 202 ms ge1-1-6509-a.castleaccess.com [69.43.169.80]
    22 196 ms 229 ms 211 ms 69.43.129.84
    23 177 ms 172 ms 195 ms 69-55-233-164.in-addr.arpa.johncompanies.com [69.55.233.164]
    24 175 ms 185 ms 241 ms 69-55-233-153.in-addr.arpa.johncompanies.com [69.55.233.153]
    25 206 ms 227 ms 312 ms 69-55-230-31.in-addr.arpa.johncompanies.com [69.55.230.31]

    Trace complete.

    For an .org.nz URL this seems a bit excessive.

    • lprent 1.1
      4 June 2010 at 10:36 am

      The reasons for this have little to do with tech and a lot to do with getting the sites server out of the way of nuisance complaints. This was extensively described in the comments on this post.

      It used to be that we sited offshore partially because of cost and partially because of legal issues. The cost issue has gone away – you can get just as cheap servers here.

      The legal one hasn’t. We don’t trust this government or its wingnut minions not to attempt to shut the dite down with nuisance complaints at a time that is convenient for them. It’d be one of those things that a devious control freak (say McCully) would probably find useful…

      Unfortunately hosting services here will limit their liability by forcing us to remove material or shut the site on receipt of a letter from a lawyer alleging a complaint. The complainant doesn’t have to prove anything – just make a complaint. And who hasn’t got a mate who is a lawyer? So nuisance complaints are cheap and easy to make.

      By siting in the US, we significantly raise the cost of nuisance complaints. You now have to find a lawyer to frame such complaints in the legal framework of the state the server or hosting company is in. So people can’t take us out cheaply.

      Of course if anyone does feel that they have been unfairly criticized by this site, then they are welcome to complain to us. We’re usually pretty good for handling the effects of our inadvertent factual errors on people affected. If they receive no satisfaction then they can take a case to our local courts. But they’d have to prove their complaint and damage to the satisfaction of a court.

      If a local hosting company is willing to give us a contract without one of those drop-out clauses in it, at least requiring some proof of the validity of a claim, we’d be happy to resite back into NZ.

  2. prism 2
    4 June 2010 at 9:15 am

    Thanks for that interesting glimpse of the perils and the active problem solving needed in maintaining the continuance of the blog.

  3. jcuknz 3
    4 June 2010 at 9:29 am

    I don’t understand what you are doing but I value it … thanks for your efforts.

  4. mummybot 4
    4 June 2010 at 7:53 pm

    Good job Mr L!

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.