Outage and an irritating lack of support

Written By: - Date published: 9:07 am, April 9th, 2013 - 15 comments
Categories: admin, The Standard - Tags:

As I’m sure that many of you are aware, we had an outage commencing at about 2205 NZST last night and finally getting fixed at about 0825 this morning.

I got notified about 10 minutes into the outage by my various ping systems that the system was down. The main server with the database was just completely off the net, including from web consoles. So I checked everything that I could, including requesting that the system reboot, to no avail. Meanwhile the suppliers own system had noticed that the server was inaccessible and informed me. So I sent off an email and received an automated acknowledgement of receipt.

I was  questing to find out how far the problem extended. On the network everything was there  up to the last step to the server.  It was quite clear that at least a bank of other servers at the same node had also gone dark which pointed to a structural problem that should have been making a big noise of their boards .

I figured that it was nearly midnight, most of the readers would have gone offline,  and they would know of the problem from my email at least. I’d check later in the morning to find out when it went back online. So I suppressed the server alarms from my cellphone and went to sleep.

Arggh, woke up and there was no response and no server. So I rang their polite phone support monkey service  at 0630 who said that they’d make sure someone knew about it. I finally got a response from support that arrived at 0707 about 8 hours from my first support email and about half an hour after I phoned.

Apologies for not getting back to you earlier. The underlying hosting
environment had to be reinitialized.

Why do I get the distinct impression that was the first time they’d realised that they had a problem. Don’t they read their frigging emails?  If they had then they could have rebooted eight hours earlier.

Sure, I’m a penny pinching sysop. But this server still generates several thousand dollars a year in revenue for the supplier and I only expect them to run the infrastructure. I don’t expect gold-plated support, but I do expect people to at least read the frigging emails I send them and to respond to them within hours.

Anyway, by that point I was  engaged in warming a backup system and refreshing it with the latest available data from 1700 the previous day. I’m going to have to increase the frequency of database backup deltas. I may even reimplement database replication now we’re out of the southern cross cables horrendously expensive data.

But I’m starting to look for a more support responsive primary server provider.

 

15 comments on “Outage and an irritating lack of support”

  1. karol 1

    I understand your frustration, Lynn.

    Thanks for getting the site back up again so soon. I only noticed it was offline this morning, when I was planning to do my Thatcher post. I actually wrote up the post on a word document while the site was offline, and it came back up when I was ready to post it.

    However, such inaction from the servers’ people gives you unnecessary stress and work, and is a disruption for people wanting to comment on and/or read stuff on TS. And as you say, it’s not good business on behalf of the people who get money from this site.

    Thanks Lynn for your dedication to keeping this blog online.

  2. freedom 2

    my guess is many of us here simply suspected Big Brother took it down because of all the seedlings of change that have sprouted thanks to the tireless efforts of The Standard’s (and others’) green thumb of reality 🙂

  3. infused 3

    Should maybe not host overseas then? You kept going on how cheap it is – well now you know why.

    I’ll host it for cheap 😉

    And I was trying to get to it. Kept getting cloudflare, which I think is useless by the way. It’s generally far out of date. But I guess it’s better than nothing.

    • lprent 3.1

      The servers locally are just as cheap and have service that isn’t that much different (I had an overnight outage last year that lasted about 12 hours because of hardware failure and 8 hours before getting a support response). But only provided you do not have a lot of overseas traffic.

      I wasn’t fleeing the NZ hosts. After all our audience is 95% from NZ (which is why I moved it back here in the first place). I was fleeing the cost of having google, bing, facbook and other bots reading the site and unpredictably ratcheting up our overseas data charges at $3-1/Gb (price keeps coming down but the bots keep getting more hungry) by hundreds of dollars per month.

      If I could get a flat rate for a 600+Gb of traffic per month in NZ with someone I trust then I’d happily use it.

      Cloudflare is mostly there to serve the images, css and js as a CDN and to thereby take part of the server load.

      • infused 3.1.1

        Easy. I run local hosting here. Hardware failure shouldn’t be an issue these days, especially not a 12 hour outage.

        Not that you’d probably want to host it with me anyway, but I’d need to know exactly what resources you were paying for, and how much inbound international traffic you have.

        • lprent 3.1.1.1

          Hardware failure shouldn’t be an issue these days…

          … and it wasn’t. However simply getting the attention of the support people accounted for 9 of those twelve hours. Then there was the usual remote testing, getting someone into the rack to look at it, followed by testing there, and then replacement took less than half an hour.

          The problem is seldom getting the problem fixed. Most of the time it is getting someone to look at the frigging thing.

          Not that you’d probably want to host it with me anyway,

          Probably not.

          …but I’d need to know exactly what resources you were paying for, and how much inbound international traffic you have.

          And that is rather the point (which if you’re actually read my previous comment would have been obvious).

          I can tell you that the total international traffic will be between ~15GB/mo and ~200GB/mo because that is the range this site has had since mid 2011. The amount of inbound traffic is probably smaller than that – but probably pretty significiant. And what frigging difference does inbound make anyway? The charges are always for international traffic totals as far as I’m aware.

          The interest from overseas isn’t as steady as the local traffic which is pretty damn steady at between 50GB/mo and 100GB/mo.

          To handle the traffic spikes I need between 2 (local) and 4 (local+international bots) CPU core’s and 1-2GB of RAM (the latter is mostly to hold PHP files and most recent database queries in APC). Never bothered to look at the diskspace since I’ve never found a resource limit.

          Quite simply the amount of overseas traffic depends on how the webbots are screwing up in a particular month. There was one month when facebook had a problem with their caching of likes and every person seeing a like on every page read reloaded the whole post.

          It’d be easy enough to have a local server and a overseas server. But why bother. Easier to just move out of the range of the dumbarse monopoly pricing from Southern Cross Cable. It does mean that we push the cost indirectly onto the local readers but it does mean that we get foreseeable bills.

          • infused 3.1.1.1.1

            My point is, you shouldn’t need to get the attention of support people. That’s shit service. That’s what System Center and Ops manager are for. Obviously they are not using these tools. Having to reboot their VM Infrastructure? Sounds like they are running it on Xen or something equally as crap.

            We don’t charge for outbound international. Hence my question.

            • lprent 3.1.1.1.1.1

              I agree about the service. Wasn’t that happy about the local services I have used as well. However I’m also extremely interested in keeping costs down. I do have a couple of places that I have used before who are responsive. I suspect I will be heading back to one of them.

              Strategically the idea is that if we have to dump advertising and run completely off donations then the costs need to be *low*. This allows us to not be beholden to anyone, which allows us to write whatever we want to and put in authors who are going to piss people off. So that gives an upper limit on what level of service I’m prepared to pay for.

              We don’t charge for outbound international.

              Interesting. I searched for something like that as that would largely get rid of my issues with overseas costs traffic. Didn’t find it. All of the sites I looked at showed “international traffic” regardless of direction.

    • felix 3.2

      Cloudflare is a pain in the arse.

  4. prism 4

    Now Trade me is out – all I get is a revolting circle but finally, the site! Dah dah. Boomkeesh. No, their logo on the search line but the page says The service is unavailable. This on top of continuing problems since the last ownership change. When the rich kids were passing the Trme parcel at the party some of the goodies must have dropped out.
    Now trade me is up – I don’t know whether I’m Arthur or Martha.

  5. Mark 5

    Could be a few reasons, IMHO..

    Someone is on strike..
    Someone would prefer to be with their family than worry about your little issues..
    The irony of your not using local services became too much and it crashed due to this.
    You haven’t paid your bill
    The volume of bullshit on here overwhelmed everything.
    Dotcon’s theft of IP to sell for untaxed profit has resulted in DOS to his supporters.
    Vast RWNJ conspiracy to shut down free speech.
    You are not the world class SYSOP you think you are.
    🙂

    [lprent: Yeah right.

    And I know that you really are just myth making moronic dickhead. It takes a special kind of stupidity to attempt to put boasts in someone else’s mouth, but evidently you are the kind of pathetic little arsehole who does.

    Programming code in C++ is what I’m good at. Running this site is merely something I do in whatever spare time I have that isn’t taken up with more important tasks. ]

    • ghostrider888 5.1

      to be frank, and vernacular, ‘Mark’, why don’t you Fuck, yes, Fuck, Off back to your porn sites!

  6. xtasy 6

    I noticed there were server issues, but as it was late, yes, due to that, I decided to call it a night in front of the computer and prepare for bed.

    Such things happen, and I was relieved this morning, to see The Standard is again (and still) up and running.

    Thanks lprent

  7. lprent 7

    Damn – another 15-20 minutes of outage. Looks like the same reason.

    Update: Looking at the lack of comments it looks like the site has been inaccessible since ~1115 (pingdom says since 1119). I guess I know what I will be happening this weekend. Moving the server to somewhere a bit more stable and responsive.

Recent Comments

Recent Posts

  • Better protections for students in halls of residence
    The Government is moving swiftly to change the law to improve the welfare and pastoral care of students living in university halls of residence and other tertiary hostels. Cabinet has agreed to several changes, including creating a new mandatory Code of Practice that sets out the duty of pastoral care ...
    BeehiveBy beehive.govt.nz
    4 hours ago
  • New trapping guide for community and expert trappers alike
    The Minister for Conservation Eugenie Sage has launched a new comprehensive trapping guide for community trappers to help them protect our native birds, plants and other wildlife, at Zealandia in Wellington today. ‘A practical guide to trapping’, has been developed by the Department of Conservation (DOC), and was launched during ...
    BeehiveBy beehive.govt.nz
    10 hours ago
  • Widening Access to Contraceptives Welcomed
    Associate Health Minister Julie Anne Genter welcomes PHARMAC’s move to improve access to long-acting reversible contraception (LARCs). PHARMAC has today announced it will fund the full cost of Mirena and Jaydess for anyone seeking long term contraception, lifting previous restrictions on access to Mirena. “I welcome women having greater choices ...
    BeehiveBy beehive.govt.nz
    12 hours ago
  • Major upgrade for Taranaki Base Hospital
    The Government has approved the next stage of a major redevelopment of Taranaki Base Hospital, which will deliver new and improved facilities for patients. Health Minister Dr David Clark has announced details of a $300 million dollar project to build a new East Wing at the New Plymouth hospital. It ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Extra support for rural families
    Extra funding will allow Rural Support Trusts to help farming families, says Minister for Rural Communities and Agriculture Damien O’Connor. “I know that rural families are worried about some of the challenges facing them, including the ongoing uncertainty created by the Mycoplasma bovis outbreak. “Those concerns sit alongside ongoing worries ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Howard Leaque Beekeeper programme graduation
    Thank you for the opportunity to be here to present certificates to the 16 graduates who have completed a beekeeping course delivered by the Howard League.  Let us start by acknowledging Auckland Prison’s Deputy Prison Director Tom Sherlock, and Acting Assistant Regional Commissioner of Corrections Northern Region Scott Walker - ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Finance Minister to attend APEC meetings
    Finance Minister Grant Robertson leaves this weekend to attend the APEC Finance Ministers meeting in Santiago, Chile. Discussions between APEC Finance Ministers at the meeting will include the effects of the current global economic uncertainty, risks for APEC economies and sustainable development of the region. While at APEC Grant Robertson ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Pacific languages are a source of strength, they ground us and build confidence
    The Minister for Pacific Peoples Aupito William Sio says for Pacific people, language can be a source of strength. It can help ground us and give us confidence. When we speak them, our languages provide us with an immediate and intimate access to our identity and our story - and ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Major boost to support disabled people in sport and recreation
    The Coalition Government has announced an action plan to improve the wellbeing of disabled New Zealanders by addressing inequalities in play, active recreation and sport. The initiative includes training to develop a workforce that understands the needs of children and young people with a range of impairments, advocacy for fit ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • More prefab homes to be built as red tape cut
    The construction sector is being freed up to allow more homes to be built more quickly as the Government cuts through some of the red tape of the Building Act.  “Every New Zealander deserves a warm, dry, safe home and old inefficiencies in the Building Act make building slow and ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Further details of Prince of Wales and Duchess of Cornwall visit to New Zealand
    Prime Minister Jacinda Ardern has welcomed further details on the Prince of Wales and Duchess of Cornwall’s visit to New Zealand next month. Their Royal Highnesses will visit New Zealand from 17-23 November – their third joint visit to New Zealand and first in four years. They arrive in Auckland ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • O’Connor in Thailand to push for RCEP deal
    Minister of State for Trade and Export Growth and Minister of Agriculture, Damien O’Connor, heads to Thailand today to attend the final Regional Comprehensive Economic Partnership (RCEP) Ministerial meeting, as negotiations enter their final stages. “The RCEP Agreement would anchor New Zealand in a regional agreement that covers 16 countries, ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Young Pacific people can access earning and learning opportunities in Hawke’s Bay, Otago and South...
    Pacific young people living in the Hawke’s Bay, Southland and Otago regions will have access to support services that have proved successful in helping young people find new earning and learning opportunities. “Tupu Aotearoa is about changing Pacific young peoples’ lives. Our young people are talented, they are smart, they ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Protecting wellbeing – ACC HQSC Trauma Forum
    Introduction As the Minister for ACC I thank you all for the work that you do supporting New Zealanders in their literally most vulnerable moments. From those who hold people’s lives in their hands, to the people who research technique, technology and trends, your work is highly valued. A special ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • NZ economy in good shape – notes prepared for speeches in Christchurch
    Notes prepared for speeches in Christchurch – Wednesday 9 October 2019 Today’s topic, “trends and opportunities for the New Zealand economy,” is certainly one getting a great deal of commentary at the moment. Looking across the media landscape lately you’ll notice we aren’t the only ones having this discussion. There ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • World Mental Health Day a reminder of the importance of mental health work
    Minister of Health Dr David Clark and Associate Minister of Health Peeni Henare say this year’s World Mental Health Day theme is a reminder of why the Government’s work on mental health is so important. “This year the World Federation for Mental Health has made suicide prevention the main theme ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Cultural Ministers Meeting
    Associate Arts, Culture and Heritage Minister Carmel Sepuloni will represent the government at Australia’s Meeting of Cultural Ministers in Adelaide this week. “This year’s meeting is special because New Zealand is expected to become an International Member of the Meeting of Cultural Ministers at this Australian forum,” Carmel Sepuloni said. “The meeting is an opportunity to ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • 608 claims resolved by GCCRS in first year
    The Greater Christchurch Claims Resolution Service has resolved 608 insurance and EQC claims in its first year in operation, Minister Megan Woods has announced. The government service, which celebrates its first birthday today, provides a one stop shop to help Cantabrians still battling to get their homes repaired or rebuilt ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • NZ economy in good shape
    Today’s topic, “trends and opportunities for the New Zealand economy,” is certainly one getting a great deal of commentary at the moment. Looking across the media landscape lately you’ll notice we aren’t the only ones having this discussion. There has been an increasing amount of attention paid to the outlook ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • NZTA to refocus on safety following review
    The Government is acting swiftly to strengthen NZTA’s regulatory role following a review into the Transport Agency, and Ministry of Transport’s performance as its monitor, Transport Minister Phil Twyford said today. An independent review by Martin Jenkins has found NZTA failed to properly regulate the transport sector under the previous ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Joint Cooperation Statement on Climate Change between the Netherlands and New Zealand
    The Netherlands and New Zealand have a long-standing and close relationship based on many shared interests and values. We value the rule of law, our democracies, and multilateralism.  And we value our environment – at home and globally. Right now there are major global challenges in all of these areas – ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Government putting right Holidays Act underpayment in Health
    The Government is putting right a decade’s worth of underpayment to nurses, doctors and other health workers, says Health Minister Dr David Clark.  Initial sampling of District Health Boards payroll records has found that around $550-$650 million is owed to DHB staff to comply with the Holidays Act. It’s expected ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Government accounts show strong economy
    A strong surplus and low debt show the economy is performing well, and means the Government is in a good position to meet the challenges of global economic uncertainty. “The surplus and low levels of debt show the economy is in good shape. This allows the Government to spend more ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Ministers approve application to expand Waihi mine
    New applications from mining company OceanaGold to purchase land in Waihi for new tailings ponds associated with its gold mines have been approved. Minister of Finance Grant Robertson and Associate Minister of Finance David Parker considered the applications under the Overseas Investment Act. Earlier this year, applications from OceanaGold to ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Tuia 250 Voyage flotilla launches with tribute to tangata whenua
    New Zealanders in Tūranganui-a-Kiwa / Poverty Bay will witness Māori, Pākehā and Pacific voyaging traditions come together today as the Tuia 250 Voyage flotilla assembles for the first time, Māori Crown Relations: Te Arawhiti Minister Kelvin Davis says. “Tuia 250 is a national commemoration and an opportunity for honest conversations ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Visit to advance trade agenda with Europe and the Commonwealth
    Minister for Trade and Export Growth David Parker leaves tomorrow for Dubai, London and Berlin for a series of meetings to advance New Zealand’s trade interests.  In Dubai he will visit New Zealand’s Pavilion at Expo 2020 where construction is underway.  There he will meet Minister of State for International Cooperation, Her ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • More cancer drugs confirmed – even more on horizon
    Confirmation that PHARMAC will fund two new cancer drugs is further evidence of the good progress the Government is making to improve the treatment of New Zealand’s leading cause of death, Health Minister David Clark says. From 1 December PHARMAC will fund alectinib (Alecensa) for ALK positive advanced non-small cell ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Boost for women in high performance sport
    An additional $2.7 million has been announced for the Government Strategy for Women and Girls in Sport and Active Recreation on the first anniversary of the strategy’s launch. Sport and Recreation Minister Grant Robertson gave the opening address to the first Sport NZ Women + Girls Summit in Wellington today, ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Parent support to help retain skilled migrants
    As part of its work to ensure businesses can get the skilled workers they need, the Coalition Government is re-opening and re-setting the Parent Category visa programme, Immigration Minister Iain Lees-Galloway says. The move will: support skilled migrants who help fill New Zealand’s skills gaps by providing a pathway for ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Senior NZDF Officer to lead Peacekeeping Mission in the Sinai Peninsula, Egypt
    Minister of Defence Ron Mark has today announced Major General Evan Williams of the New Zealand Defence Force has been selected as the commander of a significant, longstanding peacekeeping mission in the Middle East. In December, Major General Williams takes over as Force Commander for the Multinational Force and Observers ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Nurses star as Govt rebuilds health workforces
    A record number of nurses are now working to deliver health services to New Zealanders as the Government’s increased funding and new initiatives rebuild key workforces start to show results, Health Minister Dr David Clark says. •    1458 more DHB nurses since the Government took office •    106 more midwives ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New agricultural trade envoy appointed
    Farmer and former Nuffield scholar Mel Poulton has been appointed New Zealand’s Special Agricultural Trade Envoy, Minister for Trade and Export Growth, David Parker, and Minister of Agriculture, Damien O’Connor, announced today. The position supports key Government objectives, including raising the value of New Zealand agricultural goods and services. Mel is ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Pacific and Māori voyaging heritage celebrated for Tuia 250
    New Zealand’s Pacific and Māori voyaging heritage is acknowledged and celebrated today as waka of the Tuia 250 voyage flotilla arrive in Tūranga / Gisborne. “Today we celebrate Tangata Whenua, the first people of Aotearoa, and the triumphs of the voyaging tradition that brought our ancestors here from Polynesia 1000 ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Pacific languages are a root from which prosperity will grow
    “Fijian Language Week starts on Sunday and the theme reminds us how important it is that we each have something to anchor ourselves to, something that can help us pause and feel in control in a rapidly changing world,” says Minister for Pacific Peoples Aupito William Sio. “Family, culture, faith, ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • NZ Government establishes innovative, industry-focused Airspace Integration Trials Programme
    The Government is establishing an Airspace Integration Trials Programme to support the safe testing and development of advanced unmanned aircraft and accelerate their integration into the aviation system, Research, Science and Innovation Minister Megan Woods announced today. The Government will work with leading, innovative aviation industry partners to test and ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Safety upgrades and certainty for Ōtaki highway
    Transport Minister Phil Twyford today welcomed the NZ Transport Agency’s decision to fund urgent safety improvements and confirm the designation of the Ōtaki to North of Levin highway. Safety upgrades will be made along 23.4km of the existing state highway, running along SH1 from the end of the Peka Peka ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Playing our part to support refugees in our region and the world
    New Zealand playing its part in Asia-Pacific and globally are behind changes announced today to the Coalition Government’s three year refugee quota policy, Immigration Minister Iain Lees-Galloway says. “We are proud to be a welcoming and inclusive nation committed to supporting some of the world’s most vulnerable people to rebuild ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Supporting thriving inclusive communities
    Creating thriving regions and inclusive local communities is the aim of the Welcoming Communities programme being rolled out across the country, says Immigration Minister Iain Lees-Galloway today. A successful pilot of the scheme ran over the last 2 years led by Immigration New Zealand and involved ten councils across five regions ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Takahē population flying high
    Takahē may be flightless but their population is flying high with the official count reaching 418 after a record breeding season that produced an estimated 65 juveniles, the Minister of Conservation Eugenie Sage announced today. “The population reaching a high of 418 is great news for takahē which were considered ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • New Zealand makes further climate commitments
    New Zealand is today taking action to reduce the potent global warming hydrofluorocarbon (HFC) gases, Climate Minister James Shaw and Associate Environment Minister Eugenie Sage announced today. “The global agreement to reduce these potent greenhouse gases is another step in New Zealand’s commitment to reduce global warming. It is estimated ...
    BeehiveBy beehive.govt.nz
    2 weeks ago