Routing issues

9 comments on “Routing issues ”

• Comments are now closed

1. Rich 1

   1 April 2011 at 11:17 am

   Access seems fine right now.

   Also, I use Dreamhosts. Cheap, reliable, good customer service, and a reasonable AUP. I’m not sure how robust they are to takdown complaints, but their polices seem quite firm. (http://abuse.dreamhost.com/libel/ for instance).

   Also, if you need a backup site, I may be able to provide something for free. Email me?
   • lprent 1.1

     1 April 2011 at 11:43 am

     Will do.

     These days the primary server is running on a reasonably recent dual core dedicated Xeon that started at running well under 20% normal’ peak and 40% spiking CPU during the day last March, and now runs 40% normal peak and 80% spiking. The spiking typically happens when posts are being made with the SEO notifications kicking off demand, whilst at the same time background processes like the search indexing run, and we have multiple comments happening at the same time. Unfortunately I have to do capacity planning based on the spikes rather than the normal and it is difficult to prevent spiking in software (ie it is an OR queuing problem). So I need more cores and more CPU cycles.

     The idea is to move to a much higher spec system in NZ and leave our existing US system running in the warm backup role. The latter should have enough grunt in an emergency to carry us through the election year growth (probably about another 50% page views) if I turn off some of the background processes.

     If we can’t find something suitable in NZ, then it is pretty easy to get offshore servers with more cores and more CPU cycles and where we don’t have the potential malicious legal issues we have in NZ.
2. lprent 2

   1 April 2011 at 12:46 pm

   FYI from the hosting company

   We wanted to provide you with an update about the network connectivity
   issues we were seeing today. As of approximately 3pm PT the network has
   returned to normal and we are receiving assurances from our network
   provider that the situation should remain stable. Latency will also
   improve as additional carriers are brought online.

   As to what happened, we are still not being given a clear picture. The
   engineers working on the issue will provide a detailed explanation as soon
   as the situation has been fully resolved and understood. We are more than
   happy to pass that engineering report along to you (please just let us
   know you’d like to receive it.) What we’ve been told is there was a
   routing issue with one of our carriers (XO) which propagated to our other
   carriers- we have connections from multiple backbone providers to prevent
   problems or outages with one carrier from causing a complete outage. We
   realize that has absolutely not been the case today, and we will be
   receiving an explanation and plan for how this can be avoided in the
   future.

   This type of disruption is somewhat sporadic in that traffic from some
   ISPs and networks was affected, while traffic from others was unaffected.
   Also, as the routes were restored via backup carriers, you may have
   noticed that the connection would work briefly then go down again. There
   my be some more of these hiccups as this is brought to resolution in the
   next 12-24 hours, however we expect those to me momentary (on the order of
   a few seconds.)

   Please know we will continue to monitor this situation very closely and we
   will be back in touch if further issues arise.

   Not quite as bad as the hosting companies I have had for a different operations.

   One was in Florida the day that they had a really good hurricane come through. They had multiple failures in their connection providers resulting in only a single connection staying active. The volume of traffic was such that it kept overloading. After that we always kept the servers separated in different sites and the sites were selected for their lack of expected natural disasters

   But of course we had a different hoster that had a contractor for a company down the street from them ram a metal girder straight through their high voltage underground power line. The power surge was sufficient to fry the switches that were meant to divert them to the battery backup system (and the generators that were meant to feed that). Consequently our servers in that site were out for a day while they rewired everything.

   Disaster planning is a bit of a pain when traffic keeps rising. You usually find out it doesn’t work when the systems fail. Of course you could do what I did last month. I forced a test failure and found out that the warm backup couldn’t even get close to handling normal loads.

   Anyway, I think I’d better order that server today. They may be able to provision it tomorrow. Otherwise it will be a week before I have time to do the setup.
3. chris 3

   1 April 2011 at 1:38 pm

   Use linode. Their VPS service is amazing and cheap! Steer clear of dreamhost, as a programmer I would expect nothing less than self-configured servers 😛
   • lprent 3.1

     1 April 2011 at 2:56 pm

     We can’t survive on VPS’es. If you look at the ‘Online’ tab on the upper right you will see why. That is a picture of the number of readers (including the spiders) within a couple of minutes on the site. During our day that usually sits between 40-60. But it goes up as high as 150 when comment spikes. Overnight it drops down to 10-15 base load because that is when many of the spiders pick up their data steadily reading their way through the 7000 odd posts and 250,000 odd comments that are online.

     Remember that these are unique IP’s. There are many many connections within that count for all of those little graphics, CSS and JS.

     We have been booted off three VPS’es for having too much traffic so far. The last one was running purely as a warm backup storing comment and post updates. Right now we’re on the verge of having to start spreading across several servers to handle traffic, which is what I intend to set up when the new server comes online. NZ readers will read the NZ server, and overseas users (mostly spiders) will read the US server.
     • chris 3.1.1

       2 April 2011 at 11:08 am

       Ahh, yeah that makes sense, EC2?
4. Lanthanide 4

   1 April 2011 at 4:02 pm

   “Overnight it drops down to 10-15 base load because that is when many of the spiders pick up their data steadily reading their way through the 7000 odd posts and 250,000 odd comments that are online.”

   Isn’t there some way to force the spiders not to re-cache this stuff all the time? Most of those pages and comments won’t be changing very frequently (if ever), so they should be able to index it once and then not need to come back for a couple of months, and only then just to check that the pages are still valid.
5. BLiP 5

   1 April 2011 at 5:01 pm

   What’s that cdn.topsy thing that can take f o r e v e r to load?
   • lprent 5.1

     1 April 2011 at 5:38 pm

     Another thing to do this weekend, was meant to do it last week but I was helping to do the list.

• Comments are now closed

• Log in
• Donate
• Contribute post

• Articles RSS