Outage

Written By: - Date published: 9:56 am, February 1st, 2012 - 33 comments
Categories: uncategorized - Tags:

There was some maintenance done on the main NZ server last night.

All looked ok when I went to sleep at about 6am, but appears to have started having problems from about 8:30 onwards with intermittently waiting on some network resource.

Problem looks external to my server and the server’s support staff are chasing it at present. It is intermittent so hopefully this post will display.

If it persists for too long, then I’ll switch to a backup server. But I suspect they will find it shortly. In the meantime I’d better head off to work.

 

33 comments on “Outage ”

  • Comments are now closed

  1. Lanthanide 1
    1 February 2012 at 10:02 am

    Reloading pages generally seems to work after a couple of tries.

    • lprent 1.1
      1 February 2012 at 10:10 am

      Yeah. top shows the system going into wait (over 90%) and just sitting there for up to a minute, and the suddenly releasing.

      But the network system just stopped completely just then and looked like it restarted.

      Ummm started ok, but looks like we are back to waits

      • lprent 1.1.1
        1 February 2012 at 10:15 am

        Looks like the techs have found the system bit that has a problem though…

        • lprent 1.1.1.1
          1 February 2012 at 10:19 am

          Mind you, anything has to be better than the day in 2002 (?) that a contractor in the US manager to hammer a metal pole through the main underground power line to the hosting centre that our main server was housed at. Turned out the backup power system fried itself while trying to turn on.

          Or the earlier incident when a hurricane went through Florida and took out power to a hosting centre there.

  2. felix 2
    1 February 2012 at 11:01 am

    Gosman crashed the server again?

    • fender 2.1
      1 February 2012 at 3:44 pm

      lol…way to go gosman
      bombard the site with repative emptyheaded crap like comparing farmland to trademe and trying to impress with his superior knowledge of kindergarden economics, playdough stuff.

    • lprent 2.2
      1 February 2012 at 4:32 pm

      I don’t think that he has ever managed to crash the server. But computers aren’t too fussed about quality, it is the quantity that they tend to notice…

      All-time comments.
      1: Colonial Viper (13,071)
      2: Draco T Bastard (9,451)
      3: Felix (9,326)
      4: lprent (8,759)
      5. Pascal’s bookie (6,399)

      Out of 371,487 approved comments, of which you contributed 2.51% and CV has did 3.52%

      31: Gosman (2,220)

      Now at this point I’d like to emphasize for any young kiwi’s that it is NOT a contest. If I see any wannabe record breakers they will wind up like

      164: kiwiteen123 (438)

      😈

      • McFlock 2.2.1
        1 February 2012 at 4:43 pm

        League tables!
        That will help the quality of comments appearing on The Standard! Maybe the administrators could also enter into a charter partnership with a business to improve its performance?

        🙂

        • lprent 2.2.1.1
          1 February 2012 at 5:18 pm

          No way. I pull them out occasionally to stir people with. However I learnt my lesson from kiwiteen who treated them as an exercise in the unimportance of quantity.

          BTW:
          22. mcflock (2,818)

        • Colonial Viper 2.2.1.2
          1 February 2012 at 6:02 pm

          I’d back league tables. Currently I’m well ahead of the league – by quantity anyways 😈

          • fender 2.2.1.2.1
            1 February 2012 at 6:39 pm

            IMO your comments are of a consistently high quality CV
            When are you running for parliament?

  3. lprent 3
    1 February 2012 at 11:03 am

    Ok fixed..

    Looks like there was some kind of a configuration problem in apache2.2-common. Pulled the whole of it out and reinstalled. Then fixed the enabled modules. Ummm still getting highish waits

  4. lprent 4
    1 February 2012 at 11:37 am

    Ok. Dropped the fastcgi which is where the problem seemed to be happening.

    Site will be somewhat slower until I have a look at it tonight.

    • lprent 4.1
      1 February 2012 at 12:34 pm

      At present the best guess is that sometime in the past there was a bad upgrade of apache2 which was waiting for restart that I never gave it. There was some moaning in the apache groups back before the election, and I haven’t taken the server down since at least October.

      The maintenance reboot triggered the problem on fastcgi which showed up when the load started winding up for the morning.

      I’ll probably take the whole site down on the weekend and completely reinstall apache2. But I’ll run a clean test on a different machine to see if I can isolate the problem in fastCGI. There is no way that it should start getting 90+% wait time.

      For the moment, I will take it off default apache process mode this evening and move to one of the threaded systems.

      But now I’d better get back to these interprocess named semaphores. Next release is due shortly, and this is the stuff I get paid for 🙂

  5. lprent 5
    1 February 2012 at 11:43 am

    The only thing I can see that is a problem is the very very large comment areas. The one that is at almost 600 comments looks like it is giving up before it all sent.

    I’ll (finally) put my fix in later.

  6. Jackal 6
    1 February 2012 at 11:43 am

    I don’t seem to be able to log in. All the links including the homepage have been downloading a small document saying:

  7. Jackal 7
    1 February 2012 at 11:50 am

    <?php
    /**
    * Front to the WordPress application. This file doesn't do anything, but loads
    * wp-blog-header.php which does and tells WordPress to load the theme.
    *
    * @package WordPress
    */

    /**
    * Tells WordPress to load the WordPress theme and output it.
    *
    * @var bool
    */
    define('WP_USE_THEMES', true);

    /** Loads the WordPress Environment and Template */
    require('./wp-blog-header.php');
    ?

    • lprent 7.1
      1 February 2012 at 12:27 pm

      Just logged in on my three test identities.

      Try flushing your browser cache. It probably caught a php fragment in it while I was purging and reinstalling apache2-mod-php5.

  8. lprent 8
    1 February 2012 at 1:00 pm

    Testing old version of ajax edit

  9. The Voice of Reason 9
    1 February 2012 at 2:03 pm

    Cheers for all the hard work, LP. I’ve got no idea what it is you do to make this site function, might as well be voodoo for all I know, but the results are terrific!
     
    If you compare The Standard with KB, the differences are obvious. KB still looks like it was designed as a school project some time in the early nineties and the comments’ layout is just a context free spew down the page. Actually, so are most of the comments, so perhaps that bit is deliberate!
     
    Anyhoo, thanks again.

    • lprent 9.1
      1 February 2012 at 2:43 pm

      I get rather slack about it in summer, mostly because my office at home is a bit cramped and way way to hot with a few computers running (and I haven’t been able to squeeze in the air-cond unit (yet)).. And last year I was concerned with other matters during winter – mostly the balance between cigarettes and weight.

      But the layout is mostly from mummybot. I just tweak it for functionality. But I do have to do the tweaks now. If only the weather would turn to a nice wholesome programmer friendly rain and sleet…..

  10. ropata 10
    1 February 2012 at 2:59 pm

    I thought this news may have had something to do with it … now is a good time to patch the system

    http://www.theinquirer.net/inquirer/news/2142701/hundreds-wordpress-web-sites-hacked

    • lprent 10.1
      1 February 2012 at 4:40 pm

      Nope. But it is the reason that I tend to upgrade wordpress, plugins, and security updates pretty fast usually within a few days of release after I try them out on the test system.

  11. taxicab 11
    1 February 2012 at 4:45 pm

    Have you tried to access either of the following today “norightturn , tumeke , maui street ” all blocked by norton quite a coincidence !!

    • lprent 11.1
      1 February 2012 at 5:23 pm

      That is in that curious operating system called windoze isn’t it?

      However all three of those sites have the words .blogspot.com after it. I suspect that therein lies the coincidence….

  12. taxicab 12
    1 February 2012 at 5:43 pm

    sites back up now

  13. Drongo 13
    2 February 2012 at 2:31 am

    Odgers paid some nerds in Hong Kong to hack in to grab commenters’ IP addresses. She has them now and is emailing them to Slater at the minute. Things should settle down soon.

    • Colonial Viper 13.1
      4 February 2012 at 1:00 am

      Nice thing is we can pay those same nerds to counterhack her systems. Should be hilarious.

  14. felix 14
    3 February 2012 at 10:15 pm

    Hey Lynn, I’ve been having a weird issue the last day or so. If I click on a link in the sidebar comments box, it loads the post instead of the specific comment. However the comment URL does appear in the address bar of the browser and on reload it brings up the comment.

    Possibly related to the other issues? Using XP & Firefox.

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.