In theory it should be seamless…

Written By: - Date published: 8:15 pm, April 25th, 2011 - 17 comments
Categories: admin, The Standard - Tags:

Shifting the site to a GeoDNS with fallback capabilities this evening*. This involves changing the domain name server addresses again. Now like last time this is meant to be a seamless shift. However that may or may not be the case.

Last time there were some quite ISP’s with interesting ways of interpreting what a time to live meant (10 minutes means ten minutes damnit – not 10 hours). The replicated slave server turned out to have a few interesting issues and essentially failed. So there were some poor souls who were deprived of the site for quite a few hours.

This time, the replication has been running for the last week. I fixed the last few bugs in it last night. The DNS is starting to dynamically flip people to different servers as I write. In theory it should be seamless… If it isn’t then I’ll apologize in advance.

Now is a good time to cook dinner with the iPad in hand to monitor the change.

* In essence this means that when you request thestandard.org.nz, the domain name server will detect where you are asking from, and divert you to the nearest operational server.

For anyone in New Zealand, Australia, and Oceania this means your page comes from Auckland. Everyone else (including most of the spiders and spambots) will come from San Diego and not through the crowded Southern Cross cables. The Auckland server will lose about half of its weekday traffic.

It also means that we can have NZ servers without worrying about nuisance issues. There is now a hot server running offshore running a same operating site with a second or so delay. Either server can run the full site albeit slowly. It is also possible to easily add other lower cost servers hooked into the same system.

17 comments on “In theory it should be seamless… ”

  • Comments are now closed

  1. Rex Widerstrom 1
    25 April 2011 at 8:24 pm

    Still pinging at 103ms from here, @ approx 2021NZT.

    Seriously, the spambot kings aren’t even trying any more. “Useful post, thanks, I will bookmark this site” every time?

    I caught one suggesting the RSS was awry… that at least made me stop and read it for a few seconds. If they’re going to plague admins they could at least put the effort in to making it entertaining. Or they could just catch leprosy. Either is good.

    • lprent 1.1
      25 April 2011 at 8:31 pm

      😈 They do seem rather unimaginative…

      Yeah the current TTL was set at 5 hours so it will take a bit of time to shift over. But googlebot, slurp, baidu and a number of the overseas ISP’s appear to have done so now.

      Yay! Steak preparing time

  2. lprent 2
    25 April 2011 at 9:09 pm

    Looks good so far. The nice thing is that this should be the last major change to the site before the election.

  3. infused 3
    25 April 2011 at 10:10 pm

    Round Robbin and TTL of 1 hour does the trick. You set the TTL to 1hr a week before though. DNS servers will sometimes ignore this and refresh when they want to as DNS can create a lot of traffic.

    • lprent 3.1
      25 April 2011 at 11:22 pm

      I’d noticed that some DNS’es were a bit good at ignoring domain changes regardless what the TTL was.

      The TTL on the new configuration is currently set to 15 minutes. We really don’t need instantaneous fallbacks (yet). I’ll have a look at the DNS queries at the server side after I have a few weekdays to look at. I really don’t want to have millions of DNS lookups due to too short a TTL. Also don’t want to have massively long delays when the system needs to flip.

      But this new configuration appears to be flipping over pretty well with few problems as far as I can see. There is an issue with the timing of changes to robots.txt between the two systems that I will have to sort out. And I still haven’t started the sphinx search indexer for a similar reason. I’ll probably have to write some plugin excluders based on the system along with some .htaccess. But that can wait for the system to settle so I can see the traffic issues.

      In the meantime it is back to some nice clean c++ coding tomorrow. Doing system work is somewhat boring….

  4. BLiP 4
    26 April 2011 at 3:27 pm

    As an end-user can I just say: never mind the obfuscating tech-whizz kid gobbledeegook, where’s the search function??? ; )

    • lprent 4.1
      26 April 2011 at 4:16 pm

      I will attempt to turn it back on tonight. It is slightly complicated when the search is on two machines

      • BLiP 4.1.1
        27 April 2011 at 12:19 am

        No rush, mate. I did get used to it when it was working but using Google is okay as well Take your time and get it right at your end, please. Thanks for all you do.

  5. rosy 5
    26 April 2011 at 6:27 pm

    Hi Lprent
    I’m  not sure if you know of any issues on the overseas server so thought I should mention….
    I’m having trouble posting, with the same issue Carol mentioned of the anti-spam not being accepted, but also:
    – Some pics won’t display – like today’s caption contest I can’t see at all
    – Sometimes the formatting functions won’t work (intermittent and possibly related to:
    – loading and editing is really slow – up to 5 minutes and then I’ve deleted the comment (by using delete, refresh and exiting the site because just deleting takes minutes as well).

    • rosy 5.1
      26 April 2011 at 6:29 pm

      hah – anti-spam worked with that comment, but comment editor still broken – quicker to write a new comment than to edit the one above.

      • lprent 5.1.1
        26 April 2011 at 9:39 pm

        Interesting. Looking at your location it appears that it is 325ms ping from the NZ server to there, and only 161ms from the US server to there (and 169ms from my home system in NZ to the west coast US server). You shouldn’t be getting much difference in terms of latency. The US server is probably superior in terms of shifting data.

        But the logs do show a lot more heavy CPU spikes on the US server compared to the NZ. It is those bloody bots.

        The anti-spam issue should be fixed now – dropped the javascript editor for the moment which was where the problem was coming from.. Try that.

        If you’re confident about your terminal / console /editor skills on the ? Mac ? then add the following line to the /etc/hosts file (in windows it is c:\windows\system32\drivers\etc\hosts for a 32bit system) (in mac/linux you need to sudo the editor to be able to save the file)

        120.138.23.92 thestandard.org.nz

        That will flip you to using the NZ server regardless of what the DNS says. Remove it to revert. But I suspect you’ll find that it is worse not better. It looks to me like a large chunk is just bad links between you and the server today.

        • rosy 5.1.1.1
          26 April 2011 at 10:21 pm

          Yay! I have pictures. I’ll leave that IP address line in and change it back if anything goes wrong. I’ll let you know if that’s the case. Thanks heaps 🙂

    • lprent 5.2
      26 April 2011 at 6:39 pm

      Ok. I will have to test it out. There are delays whilst the images get copied from one server to another, but they’re not meant to be more than a few minutes.

      The anti-spam is generated on that server. So that sounds like a loading problem. 

      I did do all of the testing for those things and it was a bit sluggish when posting comments – mostly because it was updating a remote database. But It looks like I’ll need to test when the system is under load.

      • Carol 5.2.1
        26 April 2011 at 6:50 pm

        Thanks for all the work you’re putting into this, as I realise changing servers can be a major headache.

  6. lprent 6
    27 April 2011 at 12:04 am

    Dual servers is affecting

    1. The re-edit functionality. This looks like it is working correctly for people in NZ / aussie and failing for everyone elsewhere. I just ordered a extra license which will hopefully fix it it.

    2. The ipad/iphone functionality is probably OK from my testing. I just ordered a extra licenses.

    3. The comment save speed is pretty poor on the US server. Working on that.

    4. The images from new posts don’t come through fast enough. Authors scheduling those at least 15 minutes in advance would be a simple fix. But a more paranoid push strategy would help as well. Need to save in n places or on a separate web server (which is what wordpress.com appear to do).

  7. MrSmith 7
    27 April 2011 at 9:41 am

    Will we get the options to put in links etc again ? these options are gone now.

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.