Outage and an irritating lack of support

Written By: - Date published: 9:07 am, April 9th, 2013 - 15 comments
Categories: admin, The Standard - Tags:

As I’m sure that many of you are aware, we had an outage commencing at about 2205 NZST last night and finally getting fixed at about 0825 this morning.

I got notified about 10 minutes into the outage by my various ping systems that the system was down. The main server with the database was just completely off the net, including from web consoles. So I checked everything that I could, including requesting that the system reboot, to no avail. Meanwhile the suppliers own system had noticed that the server was inaccessible and informed me. So I sent off an email and received an automated acknowledgement of receipt.

I was  questing to find out how far the problem extended. On the network everything was there  up to the last step to the server.  It was quite clear that at least a bank of other servers at the same node had also gone dark which pointed to a structural problem that should have been making a big noise of their boards .

I figured that it was nearly midnight, most of the readers would have gone offline,  and they would know of the problem from my email at least. I’d check later in the morning to find out when it went back online. So I suppressed the server alarms from my cellphone and went to sleep.

Arggh, woke up and there was no response and no server. So I rang their polite phone support monkey service  at 0630 who said that they’d make sure someone knew about it. I finally got a response from support that arrived at 0707 about 8 hours from my first support email and about half an hour after I phoned.

Apologies for not getting back to you earlier. The underlying hosting
environment had to be reinitialized.

Why do I get the distinct impression that was the first time they’d realised that they had a problem. Don’t they read their frigging emails?  If they had then they could have rebooted eight hours earlier.

Sure, I’m a penny pinching sysop. But this server still generates several thousand dollars a year in revenue for the supplier and I only expect them to run the infrastructure. I don’t expect gold-plated support, but I do expect people to at least read the frigging emails I send them and to respond to them within hours.

Anyway, by that point I was  engaged in warming a backup system and refreshing it with the latest available data from 1700 the previous day. I’m going to have to increase the frequency of database backup deltas. I may even reimplement database replication now we’re out of the southern cross cables horrendously expensive data.

But I’m starting to look for a more support responsive primary server provider.

 

15 comments on “Outage and an irritating lack of support ”

  1. karol 1

    I understand your frustration, Lynn.

    Thanks for getting the site back up again so soon. I only noticed it was offline this morning, when I was planning to do my Thatcher post. I actually wrote up the post on a word document while the site was offline, and it came back up when I was ready to post it.

    However, such inaction from the servers’ people gives you unnecessary stress and work, and is a disruption for people wanting to comment on and/or read stuff on TS. And as you say, it’s not good business on behalf of the people who get money from this site.

    Thanks Lynn for your dedication to keeping this blog online.

  2. freedom 2

    my guess is many of us here simply suspected Big Brother took it down because of all the seedlings of change that have sprouted thanks to the tireless efforts of The Standard’s (and others’) green thumb of reality 🙂

  3. infused 3

    Should maybe not host overseas then? You kept going on how cheap it is – well now you know why.

    I’ll host it for cheap 😉

    And I was trying to get to it. Kept getting cloudflare, which I think is useless by the way. It’s generally far out of date. But I guess it’s better than nothing.

    • lprent 3.1

      The servers locally are just as cheap and have service that isn’t that much different (I had an overnight outage last year that lasted about 12 hours because of hardware failure and 8 hours before getting a support response). But only provided you do not have a lot of overseas traffic.

      I wasn’t fleeing the NZ hosts. After all our audience is 95% from NZ (which is why I moved it back here in the first place). I was fleeing the cost of having google, bing, facbook and other bots reading the site and unpredictably ratcheting up our overseas data charges at $3-1/Gb (price keeps coming down but the bots keep getting more hungry) by hundreds of dollars per month.

      If I could get a flat rate for a 600+Gb of traffic per month in NZ with someone I trust then I’d happily use it.

      Cloudflare is mostly there to serve the images, css and js as a CDN and to thereby take part of the server load.

      • infused 3.1.1

        Easy. I run local hosting here. Hardware failure shouldn’t be an issue these days, especially not a 12 hour outage.

        Not that you’d probably want to host it with me anyway, but I’d need to know exactly what resources you were paying for, and how much inbound international traffic you have.

        • lprent 3.1.1.1

          Hardware failure shouldn’t be an issue these days…

          … and it wasn’t. However simply getting the attention of the support people accounted for 9 of those twelve hours. Then there was the usual remote testing, getting someone into the rack to look at it, followed by testing there, and then replacement took less than half an hour.

          The problem is seldom getting the problem fixed. Most of the time it is getting someone to look at the frigging thing.

          Not that you’d probably want to host it with me anyway,

          Probably not.

          …but I’d need to know exactly what resources you were paying for, and how much inbound international traffic you have.

          And that is rather the point (which if you’re actually read my previous comment would have been obvious).

          I can tell you that the total international traffic will be between ~15GB/mo and ~200GB/mo because that is the range this site has had since mid 2011. The amount of inbound traffic is probably smaller than that – but probably pretty significiant. And what frigging difference does inbound make anyway? The charges are always for international traffic totals as far as I’m aware.

          The interest from overseas isn’t as steady as the local traffic which is pretty damn steady at between 50GB/mo and 100GB/mo.

          To handle the traffic spikes I need between 2 (local) and 4 (local+international bots) CPU core’s and 1-2GB of RAM (the latter is mostly to hold PHP files and most recent database queries in APC). Never bothered to look at the diskspace since I’ve never found a resource limit.

          Quite simply the amount of overseas traffic depends on how the webbots are screwing up in a particular month. There was one month when facebook had a problem with their caching of likes and every person seeing a like on every page read reloaded the whole post.

          It’d be easy enough to have a local server and a overseas server. But why bother. Easier to just move out of the range of the dumbarse monopoly pricing from Southern Cross Cable. It does mean that we push the cost indirectly onto the local readers but it does mean that we get foreseeable bills.

          • infused 3.1.1.1.1

            My point is, you shouldn’t need to get the attention of support people. That’s shit service. That’s what System Center and Ops manager are for. Obviously they are not using these tools. Having to reboot their VM Infrastructure? Sounds like they are running it on Xen or something equally as crap.

            We don’t charge for outbound international. Hence my question.

            • lprent 3.1.1.1.1.1

              I agree about the service. Wasn’t that happy about the local services I have used as well. However I’m also extremely interested in keeping costs down. I do have a couple of places that I have used before who are responsive. I suspect I will be heading back to one of them.

              Strategically the idea is that if we have to dump advertising and run completely off donations then the costs need to be *low*. This allows us to not be beholden to anyone, which allows us to write whatever we want to and put in authors who are going to piss people off. So that gives an upper limit on what level of service I’m prepared to pay for.

              We don’t charge for outbound international.

              Interesting. I searched for something like that as that would largely get rid of my issues with overseas costs traffic. Didn’t find it. All of the sites I looked at showed “international traffic” regardless of direction.

    • felix 3.2

      Cloudflare is a pain in the arse.

  4. prism 4

    Now Trade me is out – all I get is a revolting circle but finally, the site! Dah dah. Boomkeesh. No, their logo on the search line but the page says The service is unavailable. This on top of continuing problems since the last ownership change. When the rich kids were passing the Trme parcel at the party some of the goodies must have dropped out.
    Now trade me is up – I don’t know whether I’m Arthur or Martha.

  5. Mark 5

    Could be a few reasons, IMHO..

    Someone is on strike..
    Someone would prefer to be with their family than worry about your little issues..
    The irony of your not using local services became too much and it crashed due to this.
    You haven’t paid your bill
    The volume of bullshit on here overwhelmed everything.
    Dotcon’s theft of IP to sell for untaxed profit has resulted in DOS to his supporters.
    Vast RWNJ conspiracy to shut down free speech.
    You are not the world class SYSOP you think you are.
    🙂

    [lprent: Yeah right.

    And I know that you really are just myth making moronic dickhead. It takes a special kind of stupidity to attempt to put boasts in someone else’s mouth, but evidently you are the kind of pathetic little arsehole who does.

    Programming code in C++ is what I’m good at. Running this site is merely something I do in whatever spare time I have that isn’t taken up with more important tasks. ]

    • ghostrider888 5.1

      to be frank, and vernacular, ‘Mark’, why don’t you Fuck, yes, Fuck, Off back to your porn sites!

  6. xtasy 6

    I noticed there were server issues, but as it was late, yes, due to that, I decided to call it a night in front of the computer and prepare for bed.

    Such things happen, and I was relieved this morning, to see The Standard is again (and still) up and running.

    Thanks lprent

  7. lprent 7

    Damn – another 15-20 minutes of outage. Looks like the same reason.

    Update: Looking at the lack of comments it looks like the site has been inaccessible since ~1115 (pingdom says since 1119). I guess I know what I will be happening this weekend. Moving the server to somewhere a bit more stable and responsive.

Recent Comments

Recent Posts

  • EV road user charges bill passes
    Transport Minister Simeon Brown has welcomed the passing of legislation to move light electric vehicles (EVs) and plug-in hybrid electric vehicles (PHEVs) into the road user charges system from 1 April.  “It was always intended that EVs and PHEVs would be exempt from road user charges until they reached two ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Bill targets illegal, unregulated fishing in international waters
    New Zealand is strengthening its ability to combat illegal fishing outside its domestic waters and beef up regulation for its own commercial fishers in international waters through a Bill which had its first reading in Parliament today. The Fisheries (International Fishing and Other Matters) Amendment Bill 2023 sets out stronger ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Reserve Bank appointments
    Economists Carl Hansen and Professor Prasanna Gai have been appointed to the Reserve Bank Monetary Policy Committee, Finance Minister Nicola Willis announced today. The Monetary Policy Committee (MPC) is the independent decision-making body that sets the Official Cash Rate which determines interest rates.  Carl Hansen, the executive director of Capital ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Stronger protections for apartment owners
    Apartment owners and buyers will soon have greater protections as further changes to the law on unit titles come into effect, Housing Minister Chris Bishop says. “The Unit Titles (Strengthening Body Corporate Governance and Other Matters) Amendment Act had already introduced some changes in December 2022 and May 2023, and ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Travel focused on traditional partners and Middle East
    Foreign Minister Winston Peters will travel to Egypt and Europe from this weekend.    “This travel will focus on a range of New Zealand’s traditional diplomatic and security partnerships while enabling broad engagement on the urgent situation in Gaza,” Mr Peters says.   Mr Peters will attend the NATO Foreign ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Keep safe on our roads this Easter
    Transport Minister Simeon Brown is encouraging all road users to stay safe, plan their journeys ahead of time, and be patient with other drivers while travelling around this Easter long weekend. “Road safety is a responsibility we all share, and with increased traffic on our roads expected this Easter we ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Cost of living support for over 1.4 million Kiwis
    About 1.4 million New Zealanders will receive cost of living relief through increased government assistance from April 1 909,000 pensioners get a boost to Superannuation, including 5000 veterans 371,000 working-age beneficiaries will get higher payments 45,000 students will see an increase in their allowance Over a quarter of New Zealanders ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Tenancy reviews for social housing restart
    Ensuring social housing is being provided to those with the greatest needs is front of mind as the Government restarts social housing tenancy reviews, Associate Housing Minister Tama Potaka says. “Our relentless focus on building a strong economy is to ensure we can deliver better public services such as social ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Kermadec Ocean Sanctuary plan halted
    The Kermadec Ocean Sanctuary will not go ahead, with Cabinet deciding to stop work on the proposed reserve and remove the Bill that would have established it from Parliament’s order paper. “The Kermadec Ocean Sanctuary Bill would have created a 620,000 sq km economic no-go zone,” Oceans and Fisheries Minister ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Cutting all that dam red tape
    Dam safety regulations are being amended so that smaller dams won’t be subject to excessive compliance costs, Minister for Building and Construction Chris Penk says. “The coalition Government is focused on reducing costs and removing unnecessary red tape so we can get the economy back on track.  “Dam safety regulations ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Drought support extended to parts of North Island
    The coalition Government is expanding the medium-scale adverse event classification to parts of the North Island as dry weather conditions persist, Agriculture Minister Todd McClay announced today. “I have made the decision to expand the medium-scale adverse event classification already in place for parts of the South Island to also cover the ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Passage of major tax bill welcomed
    The passing of legislation giving effect to coalition Government tax commitments has been welcomed by Finance Minister Nicola Willis.  “The Taxation (Annual Rates for 2023–24, Multinational Tax, and Remedial Matters) Bill will help place New Zealand on a more secure economic footing, improve outcomes for New Zealanders, and make our tax system ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Lifting economy through science, tertiary sectors
    Science, Innovation and Technology Minister Judith Collins and Tertiary Education and Skills Minister Penny Simmonds today announced plans to transform our science and university sectors to boost the economy. Two advisory groups, chaired by Professor Sir Peter Gluckman, will advise the Government on how these sectors can play a greater ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Government announces Budget priorities
    The Budget will deliver urgently-needed tax relief to hard-working New Zealanders while putting the government’s finances back on a sustainable track, Finance Minister Nicola Willis says.  The Finance Minister made the comments at the release of the Budget Policy Statement setting out the Government’s Budget objectives. “The coalition Government intends ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Government to consider accommodation solution
    The coalition Government will look at options to address a zoning issue that limits how much financial support Queenstown residents can get for accommodation. Cabinet has agreed on a response to the Petitions Committee, which had recommended the geographic information MSD uses to determine how much accommodation supplement can be ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Government approves extension to Royal Commission of Inquiry into Abuse in Care
    Cabinet has agreed to a short extension to the final reporting timeframe for the Royal Commission into Abuse in Care from 28 March 2024 to 26 June 2024, Internal Affairs Minister Brooke van Velden says.                                         “The Royal Commission wrote to me on 16 February 2024, requesting that I consider an ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • $18m boost for Kiwis travelling to health treatment
    The coalition Government is delivering an $18 million boost to New Zealanders needing to travel for specialist health treatment, Health Minister Dr Shane Reti says.   “These changes are long overdue – the National Travel Assistance (NTA) scheme saw its last increase to mileage and accommodation rates way back in 2009.  ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • PM’s Prizes for Space to showcase sector’s talent
    The Government is recognising the innovative and rising talent in New Zealand’s growing space sector, with the Prime Minister and Space Minister Judith Collins announcing the new Prime Minister’s Prizes for Space today. “New Zealand has a growing reputation as a high-value partner for space missions and research. I am ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Concerns conveyed to China over cyber activity
    Foreign Minister Winston Peters has confirmed New Zealand’s concerns about cyber activity have been conveyed directly to the Chinese Government.     “The Prime Minister and Minister Collins have expressed concerns today about malicious cyber activity, attributed to groups sponsored by the Chinese Government, targeting democratic institutions in both New ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Independent Reviewers appointed for School Property Inquiry
    Independent Reviewers appointed for School Property Inquiry Education Minister Erica Stanford today announced the appointment of three independent reviewers to lead the Ministerial Inquiry into the Ministry of Education’s School Property Function.  The Inquiry will be led by former Minister of Foreign Affairs Murray McCully. “There is a clear need ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Brynderwyns open for Easter
    State Highway 1 across the Brynderwyns will be open for Easter weekend, with work currently underway to ensure the resilience of this critical route being paused for Easter Weekend to allow holiday makers to travel north, Transport Minister Simeon Brown says. “Today I visited the Brynderwyn Hills construction site, where ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Speech to the Infrastructure Funding & Financing Conference
    Introduction Good morning to you all, and thanks for having me bright and early today. I am absolutely delighted to be the Minister for Infrastructure alongside the Minister of Housing and Resource Management Reform. I know the Prime Minister sees the three roles as closely connected and he wants me ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Parliamentary network breached by the PRC
    New Zealand stands with the United Kingdom in its condemnation of People’s Republic of China (PRC) state-backed malicious cyber activity impacting its Electoral Commission and targeting Members of the UK Parliament. “The use of cyber-enabled espionage operations to interfere with democratic institutions and processes anywhere is unacceptable,” Minister Responsible for ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • NZ to provide support for Solomon Islands election
    Foreign Minister Winston Peters and Defence Minister Judith Collins today announced New Zealand will provide logistics support for the upcoming Solomon Islands election. “We’re sending a team of New Zealand Defence Force personnel and two NH90 helicopters to provide logistics support for the election on 17 April, at the request ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • NZ-EU FTA gains Royal Assent for 1 May entry to force
    The European Union Free Trade Agreement Legislation Amendment Bill received Royal Assent today, completing the process for New Zealand’s ratification of its free trade agreement with the European Union.    “I am pleased to announce that today, in a small ceremony at the Beehive, New Zealand notified the European Union ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • COVID-19 inquiry attracts 11,000 submissions
    Public consultation on the terms of reference for the Royal Commission into COVID-19 Lessons has concluded, Internal Affairs Minister Hon Brooke van Velden says.  “I have been advised that there were over 11,000 submissions made through the Royal Commission’s online consultation portal.” Expanding the scope of the Royal Commission of ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Families to receive up to $75 a week help with ECE fees
    Hardworking families are set to benefit from a new credit to help them meet their early childcare education (ECE) costs, Finance Minister Nicola Willis says. From 1 July, parents and caregivers of young children will be supported to manage the rising cost of living with a partial reimbursement of their ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Unlocking a sustainable, low-emissions future
    A specialised Independent Technical Advisory Group (ITAG) tasked with preparing and publishing independent non-binding advice on the design of a "green" (sustainable finance) taxonomy rulebook is being established, Climate Change Minister Simon Watts says.  “Comprising experts and market participants, the ITAG's primary goal is to deliver comprehensive recommendations to the ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Chief of Army thanked for his service
    Defence Minister Judith Collins has thanked the Chief of Army, Major General John Boswell, DSD, for his service as he leaves the Army after 40 years. “I would like to thank Major General Boswell for his contribution to the Army and the wider New Zealand Defence Force, undertaking many different ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Minister to meet Australian counterparts and Manufacturing Industry Leaders
    25 March 2024 Minister to meet Australian counterparts and Manufacturing Industry Leaders Small Business, Manufacturing, Commerce and Consumer Affairs Minister Andrew Bayly will travel to Australia for a series of bi-lateral meetings and manufacturing visits. During the visit, Minister Bayly will meet with his Australian counterparts, Senator Tim Ayres, Ed ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Government commits nearly $3 million for period products in schools
    Government commits almost $3 million for period products in schools The Coalition Government has committed $2.9 million to ensure intermediate and secondary schools continue providing period products to those who need them, Minister of Education Erica Stanford announced today. “This is an issue of dignity and ensuring young women don’t ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Speech – Making it easier to build.
    Good morning, it’s great to be here.   First, I would like to acknowledge the New Zealand Institute of Building Surveyors and thank you for the opportunity to be here this morning.  I would like to use this opportunity to outline the Government’s ambitious plan and what we hope to ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Pacific youth to shine from boost to Polyfest
    Minister for Pacific Peoples Dr Shane Reti has announced the Government’s commitment to the Auckland Secondary Schools Māori and Pacific Islands Cultural Festival, more commonly known as Polyfest. “The Ministry for Pacific Peoples is a longtime supporter of Polyfest and, as it celebrates 49 years in 2024, I’m proud to ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • 2024 Ngarimu VC and 28th (Māori) Battalion Memorial Scholarships announced
    ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Speech to Breast Cancer Foundation – Insights Conference
    Before moving onto the substance of today’s address, I want to recognise the very significant and ongoing contribution the Breast Cancer Foundation makes to support the lives of New Zealand women and their families living with breast cancer. I very much enjoy working with you. I also want to recognise ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Kiwi research soars to International Space Station
    New Zealand has notched up a first with the launch of University of Canterbury research to the International Space Station, Science, Innovation and Technology and Space Minister Judith Collins says. The hardware, developed by Dr Sarah Kessans, is designed to operate autonomously in orbit, allowing scientists on Earth to study ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Speech to the New Zealand Planning Institute
    Introduction Thank you for inviting me to speak with you today and I’m sorry I can’t be there in person. Yesterday I started in Wellington for Breakfast TV, spoke to a property conference in Auckland, and finished the day speaking to local government in Christchurch, so it would have been ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Support for Northland emergency response centre
    The Coalition Government is contributing more than $1 million to support the establishment of an emergency multi-agency coordination centre in Northland. Emergency Management and Recovery Minister Mark Mitchell announced the contribution today during a visit of the Whangārei site where the facility will be constructed.  “Northland has faced a number ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Celebrating 20 years of Whakaata Māori
    New Zealanders have enjoyed a broader range of voices telling the story of Aotearoa thanks to the creation of Whakaata Māori 20 years ago, says Māori Development Minister Tama Potaka. The minister spoke at a celebration marking the national indigenous media organisation’s 20th anniversary at their studio in Auckland on ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Some commercial fishery catch limits increased
    Commercial catch limits for some fisheries have been increased following a review showing stocks are healthy and abundant, Ocean and Fisheries Minister Shane Jones says. The changes, along with some other catch limit changes and management settings, begin coming into effect from 1 April 2024. "Regular biannual reviews of fish ...
    BeehiveBy beehive.govt.nz
    1 week ago

Page generated in The Standard by Wordpress at 2024-03-29T15:20:48+00:00