Outage and an irritating lack of support

Written By: - Date published: 9:07 am, April 9th, 2013 - 15 comments
Categories: admin, The Standard - Tags:

As I’m sure that many of you are aware, we had an outage commencing at about 2205 NZST last night and finally getting fixed at about 0825 this morning.

I got notified about 10 minutes into the outage by my various ping systems that the system was down. The main server with the database was just completely off the net, including from web consoles. So I checked everything that I could, including requesting that the system reboot, to no avail. Meanwhile the suppliers own system had noticed that the server was inaccessible and informed me. So I sent off an email and received an automated acknowledgement of receipt.

I was  questing to find out how far the problem extended. On the network everything was there  up to the last step to the server.  It was quite clear that at least a bank of other servers at the same node had also gone dark which pointed to a structural problem that should have been making a big noise of their boards .

I figured that it was nearly midnight, most of the readers would have gone offline,  and they would know of the problem from my email at least. I’d check later in the morning to find out when it went back online. So I suppressed the server alarms from my cellphone and went to sleep.

Arggh, woke up and there was no response and no server. So I rang their polite phone support monkey service  at 0630 who said that they’d make sure someone knew about it. I finally got a response from support that arrived at 0707 about 8 hours from my first support email and about half an hour after I phoned.

Apologies for not getting back to you earlier. The underlying hosting
environment had to be reinitialized.

Why do I get the distinct impression that was the first time they’d realised that they had a problem. Don’t they read their frigging emails?  If they had then they could have rebooted eight hours earlier.

Sure, I’m a penny pinching sysop. But this server still generates several thousand dollars a year in revenue for the supplier and I only expect them to run the infrastructure. I don’t expect gold-plated support, but I do expect people to at least read the frigging emails I send them and to respond to them within hours.

Anyway, by that point I was  engaged in warming a backup system and refreshing it with the latest available data from 1700 the previous day. I’m going to have to increase the frequency of database backup deltas. I may even reimplement database replication now we’re out of the southern cross cables horrendously expensive data.

But I’m starting to look for a more support responsive primary server provider.

 

15 comments on “Outage and an irritating lack of support ”

  1. karol 1

    I understand your frustration, Lynn.

    Thanks for getting the site back up again so soon. I only noticed it was offline this morning, when I was planning to do my Thatcher post. I actually wrote up the post on a word document while the site was offline, and it came back up when I was ready to post it.

    However, such inaction from the servers’ people gives you unnecessary stress and work, and is a disruption for people wanting to comment on and/or read stuff on TS. And as you say, it’s not good business on behalf of the people who get money from this site.

    Thanks Lynn for your dedication to keeping this blog online.

  2. freedom 2

    my guess is many of us here simply suspected Big Brother took it down because of all the seedlings of change that have sprouted thanks to the tireless efforts of The Standard’s (and others’) green thumb of reality 🙂

  3. infused 3

    Should maybe not host overseas then? You kept going on how cheap it is – well now you know why.

    I’ll host it for cheap 😉

    And I was trying to get to it. Kept getting cloudflare, which I think is useless by the way. It’s generally far out of date. But I guess it’s better than nothing.

    • lprent 3.1

      The servers locally are just as cheap and have service that isn’t that much different (I had an overnight outage last year that lasted about 12 hours because of hardware failure and 8 hours before getting a support response). But only provided you do not have a lot of overseas traffic.

      I wasn’t fleeing the NZ hosts. After all our audience is 95% from NZ (which is why I moved it back here in the first place). I was fleeing the cost of having google, bing, facbook and other bots reading the site and unpredictably ratcheting up our overseas data charges at $3-1/Gb (price keeps coming down but the bots keep getting more hungry) by hundreds of dollars per month.

      If I could get a flat rate for a 600+Gb of traffic per month in NZ with someone I trust then I’d happily use it.

      Cloudflare is mostly there to serve the images, css and js as a CDN and to thereby take part of the server load.

      • infused 3.1.1

        Easy. I run local hosting here. Hardware failure shouldn’t be an issue these days, especially not a 12 hour outage.

        Not that you’d probably want to host it with me anyway, but I’d need to know exactly what resources you were paying for, and how much inbound international traffic you have.

        • lprent 3.1.1.1

          Hardware failure shouldn’t be an issue these days…

          … and it wasn’t. However simply getting the attention of the support people accounted for 9 of those twelve hours. Then there was the usual remote testing, getting someone into the rack to look at it, followed by testing there, and then replacement took less than half an hour.

          The problem is seldom getting the problem fixed. Most of the time it is getting someone to look at the frigging thing.

          Not that you’d probably want to host it with me anyway,

          Probably not.

          …but I’d need to know exactly what resources you were paying for, and how much inbound international traffic you have.

          And that is rather the point (which if you’re actually read my previous comment would have been obvious).

          I can tell you that the total international traffic will be between ~15GB/mo and ~200GB/mo because that is the range this site has had since mid 2011. The amount of inbound traffic is probably smaller than that – but probably pretty significiant. And what frigging difference does inbound make anyway? The charges are always for international traffic totals as far as I’m aware.

          The interest from overseas isn’t as steady as the local traffic which is pretty damn steady at between 50GB/mo and 100GB/mo.

          To handle the traffic spikes I need between 2 (local) and 4 (local+international bots) CPU core’s and 1-2GB of RAM (the latter is mostly to hold PHP files and most recent database queries in APC). Never bothered to look at the diskspace since I’ve never found a resource limit.

          Quite simply the amount of overseas traffic depends on how the webbots are screwing up in a particular month. There was one month when facebook had a problem with their caching of likes and every person seeing a like on every page read reloaded the whole post.

          It’d be easy enough to have a local server and a overseas server. But why bother. Easier to just move out of the range of the dumbarse monopoly pricing from Southern Cross Cable. It does mean that we push the cost indirectly onto the local readers but it does mean that we get foreseeable bills.

          • infused 3.1.1.1.1

            My point is, you shouldn’t need to get the attention of support people. That’s shit service. That’s what System Center and Ops manager are for. Obviously they are not using these tools. Having to reboot their VM Infrastructure? Sounds like they are running it on Xen or something equally as crap.

            We don’t charge for outbound international. Hence my question.

            • lprent 3.1.1.1.1.1

              I agree about the service. Wasn’t that happy about the local services I have used as well. However I’m also extremely interested in keeping costs down. I do have a couple of places that I have used before who are responsive. I suspect I will be heading back to one of them.

              Strategically the idea is that if we have to dump advertising and run completely off donations then the costs need to be *low*. This allows us to not be beholden to anyone, which allows us to write whatever we want to and put in authors who are going to piss people off. So that gives an upper limit on what level of service I’m prepared to pay for.

              We don’t charge for outbound international.

              Interesting. I searched for something like that as that would largely get rid of my issues with overseas costs traffic. Didn’t find it. All of the sites I looked at showed “international traffic” regardless of direction.

    • felix 3.2

      Cloudflare is a pain in the arse.

  4. prism 4

    Now Trade me is out – all I get is a revolting circle but finally, the site! Dah dah. Boomkeesh. No, their logo on the search line but the page says The service is unavailable. This on top of continuing problems since the last ownership change. When the rich kids were passing the Trme parcel at the party some of the goodies must have dropped out.
    Now trade me is up – I don’t know whether I’m Arthur or Martha.

  5. Mark 5

    Could be a few reasons, IMHO..

    Someone is on strike..
    Someone would prefer to be with their family than worry about your little issues..
    The irony of your not using local services became too much and it crashed due to this.
    You haven’t paid your bill
    The volume of bullshit on here overwhelmed everything.
    Dotcon’s theft of IP to sell for untaxed profit has resulted in DOS to his supporters.
    Vast RWNJ conspiracy to shut down free speech.
    You are not the world class SYSOP you think you are.
    🙂

    [lprent: Yeah right.

    And I know that you really are just myth making moronic dickhead. It takes a special kind of stupidity to attempt to put boasts in someone else’s mouth, but evidently you are the kind of pathetic little arsehole who does.

    Programming code in C++ is what I’m good at. Running this site is merely something I do in whatever spare time I have that isn’t taken up with more important tasks. ]

    • ghostrider888 5.1

      to be frank, and vernacular, ‘Mark’, why don’t you Fuck, yes, Fuck, Off back to your porn sites!

  6. xtasy 6

    I noticed there were server issues, but as it was late, yes, due to that, I decided to call it a night in front of the computer and prepare for bed.

    Such things happen, and I was relieved this morning, to see The Standard is again (and still) up and running.

    Thanks lprent

  7. lprent 7

    Damn – another 15-20 minutes of outage. Looks like the same reason.

    Update: Looking at the lack of comments it looks like the site has been inaccessible since ~1115 (pingdom says since 1119). I guess I know what I will be happening this weekend. Moving the server to somewhere a bit more stable and responsive.

Recent Comments

Recent Posts

  • Comprehensive Partnership the goal for NZ and the Philippines
    Prime Minister Christopher Luxon held a bilateral meeting today with the President of the Philippines, Ferdinand Marcos Jr.  The Prime Minister was accompanied by MP Paulo Garcia, the first Filipino to be elected to a legislature outside the Philippines. During today’s meeting, Prime Minister Luxon and President Marcos Jr discussed opportunities to ...
    BeehiveBy beehive.govt.nz
    3 hours ago
  • Government commits $20m to Westport flood protection
    The Government has announced that $20 million in funding will be made available to Westport to fund much needed flood protection around the town. This measure will significantly improve the resilience of the community, says Local Government Minister Simeon Brown. “The Westport community has already been allocated almost $3 million ...
    BeehiveBy beehive.govt.nz
    9 hours ago
  • Taupō takes pole position
    The Government is proud to support the first ever Repco Supercars Championship event in Taupō as up to 70,000 motorsport fans attend the Taupō International Motorsport Park this weekend, says Economic Development Minister Melissa Lee. “Anticipation for the ITM Taupō Super400 is huge, with tickets and accommodation selling out weeks ...
    BeehiveBy beehive.govt.nz
    10 hours ago
  • Cost of living support for low-income homeowners
    Local Government Minister Simeon Brown has announced an increase to the Rates Rebate Scheme, putting money back into the pockets of low-income homeowners.  “The coalition Government is committed to bringing down the cost of living for New Zealanders. That includes targeted support for those Kiwis who are doing things tough, such ...
    BeehiveBy beehive.govt.nz
    12 hours ago
  • Government backing mussel spat project
    The Coalition Government is investing in a project to boost survival rates of New Zealand mussels and grow the industry, Oceans and Fisheries Minister Shane Jones has announced. “This project seeks to increase the resilience of our mussels and significantly boost the sector’s productivity,” Mr Jones says. “The project - ...
    BeehiveBy beehive.govt.nz
    13 hours ago
  • Government focused on getting people into work
    Benefit figures released today underscore the importance of the Government’s plan to rebuild the economy and have 50,000 fewer people on Jobseeker Support, Social Development and Employment Minister Louise Upston says. “Benefit numbers are still significantly higher than when National was last in government, when there was about 70,000 fewer ...
    BeehiveBy beehive.govt.nz
    16 hours ago
  • Clean energy key driver to reducing emissions
    The Government’s commitment to doubling New Zealand’s renewable energy capacity is backed by new data showing that clean energy has helped the country reach its lowest annual gross emissions since 1999, Climate Change Minister Simon Watts says. New Zealand’s latest Greenhouse Gas Inventory (1990-2022) published today, shows gross emissions fell ...
    BeehiveBy beehive.govt.nz
    16 hours ago
  • Earthquake-prone buildings review brought forward
    The Government is bringing the earthquake-prone building review forward, with work to start immediately, and extending the deadline for remediations by four years, Building and Construction Minister Chris Penk says. “Our Government is focused on rebuilding the economy. A key part of our plan is to cut red tape that ...
    BeehiveBy beehive.govt.nz
    21 hours ago
  • Thailand and NZ to agree to Strategic Partnership
    Prime Minister Christopher Luxon and his Thai counterpart, Prime Minister Srettha Thavisin, have today agreed that New Zealand and the Kingdom of Thailand will upgrade the bilateral relationship to a Strategic Partnership by 2026. “New Zealand and Thailand have a lot to offer each other. We have a strong mutual desire to build ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Government consults on extending coastal permits for ports
    RMA Reform Minister Chris Bishop and Transport Minister Simeon Brown have today announced the Coalition Government’s intention to extend port coastal permits for a further 20 years, providing port operators with certainty to continue their operations. “The introduction of the Resource Management Act in 1991 required ports to obtain coastal ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Inflation coming down, but more work to do
    Today’s announcement that inflation is down to 4 per cent is encouraging news for Kiwis, but there is more work to be done - underlining the importance of the Government’s plan to get the economy back on track, acting Finance Minister Chris Bishop says. “Inflation is now at 4 per ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • School attendance restored as a priority in health advice
    Refreshed health guidance released today will help parents and schools make informed decisions about whether their child needs to be in school, addressing one of the key issues affecting school attendance, says Associate Education Minister David Seymour. In recent years, consistently across all school terms, short-term illness or medical reasons ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Unnecessary bureaucracy cut in oceans sector
    Oceans and Fisheries Minister Shane Jones is streamlining high-level oceans management while maintaining a focus on supporting the sector’s role in the export-led recovery of the economy. “I am working to realise the untapped potential of our fishing and aquaculture sector. To achieve that we need to be smarter with ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Patterson promoting NZ’s wool sector at International Congress
    Associate Agriculture Minister Mark Patterson is speaking at the International Wool Textile Organisation Congress in Adelaide, promoting New Zealand wool, and outlining the coalition Government’s support for the revitalisation the sector.    "New Zealand’s wool exports reached $400 million in the year to 30 June 2023, and the coalition Government ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Removing red tape to help early learners thrive
    The Government is making legislative changes to make it easier for new early learning services to be established, and for existing services to operate, Associate Education Minister David Seymour says. The changes involve repealing the network approval provisions that apply when someone wants to establish a new early learning service, ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • RMA changes to cut coal mining consent red tape
    Changes to the Resource Management Act will align consenting for coal mining to other forms of mining to reduce barriers that are holding back economic development, Resources Minister Shane Jones says. “The inconsistent treatment of coal mining compared with other extractive activities is burdensome red tape that fails to acknowledge ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • McClay reaffirms strong NZ-China trade relationship
    Trade, Agriculture and Forestry Minister Todd McClay has concluded productive discussions with ministerial counterparts in Beijing today, in support of the New Zealand-China trade and economic relationship. “My meeting with Commerce Minister Wang Wentao reaffirmed the complementary nature of the bilateral trade relationship, with our Free Trade Agreement at its ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Prime Minister Luxon acknowledges legacy of Singapore Prime Minister Lee
    Prime Minister Christopher Luxon today paid tribute to Singapore’s outgoing Prime Minister Lee Hsien Loong.   Meeting in Singapore today immediately before Prime Minister Lee announced he was stepping down, Prime Minister Luxon warmly acknowledged his counterpart’s almost twenty years as leader, and the enduring legacy he has left for Singapore and South East ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • PMs Luxon and Lee deepen Singapore-NZ ties
    Prime Minister Christopher Luxon held a bilateral meeting today with Singapore Prime Minister Lee Hsien Loong. While in Singapore as part of his visit to South East Asia this week, Prime Minister Luxon also met with Singapore President Tharman Shanmugaratnam and will meet with Deputy Prime Minister Lawrence Wong.  During today’s meeting, Prime Minister Luxon ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Antarctica New Zealand Board appointments
    Foreign Minister Winston Peters has made further appointments to the Board of Antarctica New Zealand as part of a continued effort to ensure the Scott Base Redevelopment project is delivered in a cost-effective and efficient manner.  The Minister has appointed Neville Harris as a new member of the Board. Mr ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Finance Minister travels to Washington DC
    Finance Minister Nicola Willis will travel to the United States on Tuesday to attend a meeting of the Five Finance Ministers group, with counterparts from Australia, the United States, Canada, and the United Kingdom.  “I am looking forward to meeting with our Five Finance partners on how we can work ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Pet bonds a win/win for renters and landlords
    The coalition Government has today announced purrfect and pawsitive changes to the Residential Tenancies Act to give tenants with pets greater choice when looking for a rental property, says Housing Minister Chris Bishop. “Pets are important members of many Kiwi families. It’s estimated that around 64 per cent of New ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Long Tunnel for SH1 Wellington being considered
    State Highway 1 (SH1) through Wellington City is heavily congested at peak times and while planning continues on the duplicate Mt Victoria Tunnel and Basin Reserve project, the Government has also asked NZ Transport Agency (NZTA) to consider and provide advice on a Long Tunnel option, Transport Minister Simeon Brown ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • New Zealand condemns Iranian strikes
    Prime Minister Christopher Luxon and Foreign Minister Winston Peters have condemned Iran’s shocking and illegal strikes against Israel.    “These attacks are a major challenge to peace and stability in a region already under enormous pressure," Mr Luxon says.    "We are deeply concerned that miscalculation on any side could ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Huge interest in Government’s infrastructure plans
    Hundreds of people in little over a week have turned out in Northland to hear Regional Development Minister Shane Jones speak about plans for boosting the regional economy through infrastructure. About 200 people from the infrastructure and associated sectors attended an event headlined by Mr Jones in Whangarei today. Last ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Health Minister thanks outgoing Health New Zealand Chair
    Health Minister Dr Shane Reti has today thanked outgoing Health New Zealand – Te Whatu Ora Chair Dame Karen Poutasi for her service on the Board.   “Dame Karen tendered her resignation as Chair and as a member of the Board today,” says Dr Reti.  “I have asked her to ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Roads of National Significance planning underway
    The NZ Transport Agency (NZTA) has signalled their proposed delivery approach for the Government’s 15 Roads of National Significance (RoNS), with the release of the State Highway Investment Proposal (SHIP) today, Transport Minister Simeon Brown says.  “Boosting economic growth and productivity is a key part of the Government’s plan to ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Navigating an unstable global environment
    New Zealand is renewing its connections with a world facing urgent challenges by pursuing an active, energetic foreign policy, Foreign Minister Winston Peters says.   “Our country faces the most unstable global environment in decades,” Mr Peters says at the conclusion of two weeks of engagements in Egypt, Europe and the United States.    “We cannot afford to sit back in splendid ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • NZ welcomes Australian Governor-General
    Prime Minister Christopher Luxon has announced the Australian Governor-General, His Excellency General The Honourable David Hurley and his wife Her Excellency Mrs Linda Hurley, will make a State visit to New Zealand from Tuesday 16 April to Thursday 18 April. The visit reciprocates the State visit of former Governor-General Dame Patsy Reddy ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Pseudoephedrine back on shelves for Winter
    Associate Health Minister David Seymour has announced that Medsafe has approved 11 cold and flu medicines containing pseudoephedrine. Pharmaceutical suppliers have indicated they may be able to supply the first products in June. “This is much earlier than the original expectation of medicines being available by 2025. The Government recognised ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • NZ and the US: an ever closer partnership
    New Zealand and the United States have recommitted to their strategic partnership in Washington DC today, pledging to work ever more closely together in support of shared values and interests, Foreign Minister Winston Peters says.    “The strategic environment that New Zealand and the United States face is considerably more ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Joint US and NZ declaration
    April 11, 2024 Joint Declaration by United States Secretary of State the Honorable Antony J. Blinken and New Zealand Minister of Foreign Affairs the Right Honourable Winston Peters We met today in Washington, D.C. to recommit to the historic partnership between our two countries and the principles that underpin it—rule ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • NZ and US to undertake further practical Pacific cooperation
    Foreign Minister Winston Peters has announced further New Zealand cooperation with the United States in the Pacific Islands region through $16.4 million in funding for initiatives in digital connectivity and oceans and fisheries research.   “New Zealand can achieve more in the Pacific if we work together more urgently and ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Government redress for Te Korowai o Wainuiārua
    The Government is continuing the bipartisan effort to restore its relationship with iwi as the Te Korowai o Wainuiārua Claims Settlement Bill passed its first reading in Parliament today, says Treaty Negotiations Minister Paul Goldsmith. “Historical grievances of Te Korowai o Wainuiārua relate to 19th century warfare, land purchased or taken ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Focus on outstanding minerals permit applications
    New Zealand Petroleum and Minerals is working to resolve almost 150 outstanding minerals permit applications by the end of the financial year, enabling valuable mining activity and signalling to the sector that New Zealand is open for business, Resources Minister Shane Jones says.  “While there are no set timeframes for ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Applications open for NZ-Ireland Research Call
    The New Zealand and Irish governments have today announced that applications for the 2024 New Zealand-Ireland Joint Research Call on Agriculture and Climate Change are now open. This is the third research call in the three-year Joint Research Initiative pilot launched in 2022 by the Ministry for Primary Industries and Ireland’s ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Tenancy rules changes to improve rental market
    The coalition Government has today announced changes to the Residential Tenancies Act to encourage landlords back to the rental property market, says Housing Minister Chris Bishop. “The previous Government waged a war on landlords. Many landlords told us this caused them to exit the rental market altogether. It caused worse ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Boosting NZ’s trade and agricultural relationship with China
    Trade and Agriculture Minister Todd McClay will visit China next week, to strengthen relationships, support Kiwi exporters and promote New Zealand businesses on the world stage. “China is one of New Zealand’s most significant trade and economic relationships and remains an important destination for New Zealand’s products, accounting for nearly 22 per cent of our good and ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Freshwater farm plan systems to be improved
    The coalition Government intends to improve freshwater farm plans so that they are more cost-effective and practical for farmers, Associate Environment Minister Andrew Hoggard and Agriculture Minister Todd McClay have announced. “A fit-for-purpose freshwater farm plan system will enable farmers and growers to find the right solutions for their farm ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Fast Track Projects advisory group named
    The coalition Government has today announced the expert advisory group who will provide independent recommendations to Ministers on projects to be included in the Fast Track Approvals Bill, say RMA Reform Minister Chris Bishop and Regional Development Minister Shane Jones. “Our Fast Track Approval process will make it easier and ...
    BeehiveBy beehive.govt.nz
    1 week ago

Page generated in The Standard by Wordpress at 2024-04-18T13:47:44+00:00