Outage last night

Written By: - Date published: 8:19 am, November 8th, 2014 - 17 comments
Categories: admin, The Standard - Tags:

Dolphin Over World 2010The site had a strange problem with a database solid state drive last night while restarting after doing operating system upgrades. This morning it was completely offline and causing problems with the other drives in the array.

I took the entire database raid array offline this morning because I didn’t trust its integrity, shifted to a backup drive, and restored to the last good backup from 2300 last night.

There will have been some comments lost between 2300 and 0044 when the outage was reported.

This is the first time I have had a SSD fail on me, and it looks like it was just a connector problem. The behaviour of the raid was more of an issue, and one that I will be looking at tomorrow.

There was a planned hardware upgrade with a upgrade of my home server this weekend to a 8 core water cooled CPU and motherboard, but I will leave that for another evening. It is mainly there to allow me to get my workstation back for some game playing.

17 comments on “Outage last night ”

  1. Skinny 1

    There I was thinking I had copped a ban for drunken waffling lol. Sounded like a bit of a headache, probably just as bad as mine this morning.

  2. Manuka AOR 2

    Really good it is back. I was starting to get anxiety/ withdrawal symptoms 😀

  3. Chooky 3

    us Chooks in the hen house are very concerned ….we think John Key Five Eyes and the ferrets are on to us….we will be lurking and moving incognito today

  4. lprent 4

    It is more of a flex issue. The fail over system isn’t working. But the system I need to test it from is the same one that TS is running on.

    Get a second system to the same standard, workstation gets freed up, I can do the testing without taking TS out.

    Just waiting for a night when I can send a few hours after midnight.

  5. GregJ 5

    Whew – glad its working again. I live on the other side of the World – when The Standard ain’t up I begin to fear Armageddon is coming!! 😯

  6. greywarshark 6

    lprent
    I was trying to send this to you last night thinking you might have some time to kill.
    Wrong. Thanks for getting all going again.

    All the best with what sounds an interesting project Lyn. I had a look at the donation site for Lyn and saw that she received more than her hoped for amount so that is good.

    This is a link to a trip over India on a C90 motorcycle Lynn. You could get into the ambience of the country while Lyn is there though you are still at home. It is funny and fascinating. But you may have seen it.
    http://www.c90adventures.co.uk/news/76-india

  7. One Anonymous Bloke 7

    I have to ask: what games are you playing?

    • lprent 7.1

      Starting diablo 3 running under wine. Have been using some large new maps on homeworld2 in wine. 0ad occasionally.

      I don’t have time to play many games these days. Those that I do play have to be shortish and reasonably strategic or at least tactical. Then I can play them while I mull on something else and not get bored.

      • Draco T Bastard 7.1.1

        You may like Ancient Space then. Similar game play to Homeworld but not with the sense of vastness that Homeworld has as the graphics are used to enclose space (They really are fantastic graphics though). I’m not overly fond of the resource system as it comes across as artificial but do enjoy the game.

        Don’t know if it would work under WINE.

  8. infused 8

    let me guess. you have some shitty on board raid card? that’s your issue.

    • lprent 8.1

      Why would I use one of those. They keep dying.

      Nope this was mdadm. The problem was with the shitty data connector on the SSD.

  9. Kiwiri - Raided of the Last Shark 9

    As far as I am concerned, The Standard is also, for me when I come online to look for material to read, The Sunshine.

  10. Instauration 10

    Lynn
    SSD promise much – but unless you pay mega for SAS, non-nearline you essentially have a device that is optimised for read, and DB writes will quickly erode the mtbf.
    Avoid RAID5 – it rarely provides an optimal r/w charachteristic for any DB.
    RAID10 – SAS – an easy life 🙂

    • lprent 10.1

      It depends what you are putting on it.

      Think about what it means on a blog database in reality rather than in mere theory. The database write usage is pretty minimal compared to the read simply because the numbers of writes in day are in the thousands, and the reads are massive. It is essentially a write once database because few records are deleted.

      A blog database needs to be optimized for read.

      My back of the envelope calcs indicate that I will have a write lifetime of >15 years. The SSD that failed had a connector problem. Probably because it had a crappy clipin. I tested it with 5 SATA cables and none of them locked in properly. Yet all of the other SSDs were fine clipping in.

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.