web analytics

Routing issues

Written By: - Date published: 11:08 am, April 1st, 2011 - 9 comments
Categories: admin, The Standard - Tags:

We have had some routing problems this morning with several networks used by our provider in San Diego. The problem have been ongoing with the network links appearing and disappearing from the perspective of NZ and aussie (there is less of a issue from overseas networks). They’re working on it. It started just before 7am and is starting to look stable at about 11am.

Ironically, I’m just waiting for some due diligence to be done on a local supplier for the election year server – ie that they won’t just fold if a malicious complaint is made. The current warm backup system can’t handle the load (it was having problems handling the comments being fed to it), so it is currently off. The new server was due to get configured this weekend but that is looking less likely because it is unlikely to get provisioned before Monday.

And no, this isn’t an April fools joke…. But since we last had a failure almost exactly a year ago at the end of March running over April 1, I’m starting to get superstitious.

9 comments on “Routing issues ”

  1. Rich 1

    Access seems fine right now.

    Also, I use Dreamhosts. Cheap, reliable, good customer service, and a reasonable AUP. I’m not sure how robust they are to takdown complaints, but their polices seem quite firm. (http://abuse.dreamhost.com/libel/ for instance).

    Also, if you need a backup site, I may be able to provide something for free. Email me?

    • lprent 1.1

      Will do.

      These days the primary server is running on a reasonably recent dual core dedicated Xeon that started at running well under 20% normal’ peak and 40% spiking CPU during the day last March, and now runs 40% normal peak and 80% spiking. The spiking typically happens when posts are being made with the SEO notifications kicking off demand, whilst at the same time background processes like the search indexing run, and we have multiple comments happening at the same time. Unfortunately I have to do capacity planning based on the spikes rather than the normal and it is difficult to prevent spiking in software (ie it is an OR queuing problem). So I need more cores and more CPU cycles.

      The idea is to move to a much higher spec system in NZ and leave our existing US system running in the warm backup role. The latter should have enough grunt in an emergency to carry us through the election year growth (probably about another 50% page views) if I turn off some of the background processes.

      If we can’t find something suitable in NZ, then it is pretty easy to get offshore servers with more cores and more CPU cycles and where we don’t have the potential malicious legal issues we have in NZ.

  2. lprent 2

    FYI from the hosting company

    We wanted to provide you with an update about the network connectivity
    issues we were seeing today. As of approximately 3pm PT the network has
    returned to normal and we are receiving assurances from our network
    provider that the situation should remain stable. Latency will also
    improve as additional carriers are brought online.

    As to what happened, we are still not being given a clear picture. The
    engineers working on the issue will provide a detailed explanation as soon
    as the situation has been fully resolved and understood. We are more than
    happy to pass that engineering report along to you (please just let us
    know you’d like to receive it.) What we’ve been told is there was a
    routing issue with one of our carriers (XO) which propagated to our other
    carriers- we have connections from multiple backbone providers to prevent
    problems or outages with one carrier from causing a complete outage. We
    realize that has absolutely not been the case today, and we will be
    receiving an explanation and plan for how this can be avoided in the
    future.

    This type of disruption is somewhat sporadic in that traffic from some
    ISPs and networks was affected, while traffic from others was unaffected.
    Also, as the routes were restored via backup carriers, you may have
    noticed that the connection would work briefly then go down again. There
    my be some more of these hiccups as this is brought to resolution in the
    next 12-24 hours, however we expect those to me momentary (on the order of
    a few seconds.)

    Please know we will continue to monitor this situation very closely and we
    will be back in touch if further issues arise.

    Not quite as bad as the hosting companies I have had for a different operations.

    One was in Florida the day that they had a really good hurricane come through. They had multiple failures in their connection providers resulting in only a single connection staying active. The volume of traffic was such that it kept overloading. After that we always kept the servers separated in different sites and the sites were selected for their lack of expected natural disasters

    But of course we had a different hoster that had a contractor for a company down the street from them ram a metal girder straight through their high voltage underground power line. The power surge was sufficient to fry the switches that were meant to divert them to the battery backup system (and the generators that were meant to feed that). Consequently our servers in that site were out for a day while they rewired everything.

    Disaster planning is a bit of a pain when traffic keeps rising. You usually find out it doesn’t work when the systems fail. Of course you could do what I did last month. I forced a test failure and found out that the warm backup couldn’t even get close to handling normal loads.

    Anyway, I think I’d better order that server today. They may be able to provision it tomorrow. Otherwise it will be a week before I have time to do the setup.

  3. chris 3

    Use linode. Their VPS service is amazing and cheap! Steer clear of dreamhost, as a programmer I would expect nothing less than self-configured servers 😛

    • lprent 3.1

      We can’t survive on VPS’es. If you look at the ‘Online’ tab on the upper right you will see why. That is a picture of the number of readers (including the spiders) within a couple of minutes on the site. During our day that usually sits between 40-60. But it goes up as high as 150 when comment spikes. Overnight it drops down to 10-15 base load because that is when many of the spiders pick up their data steadily reading their way through the 7000 odd posts and 250,000 odd comments that are online.

      Remember that these are unique IP’s. There are many many connections within that count for all of those little graphics, CSS and JS.

      We have been booted off three VPS’es for having too much traffic so far. The last one was running purely as a warm backup storing comment and post updates. Right now we’re on the verge of having to start spreading across several servers to handle traffic, which is what I intend to set up when the new server comes online. NZ readers will read the NZ server, and overseas users (mostly spiders) will read the US server.

  4. Lanthanide 4

    “Overnight it drops down to 10-15 base load because that is when many of the spiders pick up their data steadily reading their way through the 7000 odd posts and 250,000 odd comments that are online.”

    Isn’t there some way to force the spiders not to re-cache this stuff all the time? Most of those pages and comments won’t be changing very frequently (if ever), so they should be able to index it once and then not need to come back for a couple of months, and only then just to check that the pages are still valid.

  5. BLiP 5

    What’s that cdn.topsy thing that can take f o r e v e r to load?

Recent Comments

Recent Posts

  • Clean car package to drive down emissions
    New rebates for electric and plug-in hybrid vehicles start July 1 with up to $8,625 for new vehicles and $3,450 for used. Electric vehicle chargers now available every 75km along most state highways to give Kiwis confidence. Low Emission Transport Fund will have nearly four times the funding by 2023 ...
    BeehiveBy beehive.govt.nz
    23 hours ago
  • Progress towards simpler process for changing sex on birth certificates
    The Government is taking the next step to support transgender, non-binary and intersex New Zealanders, by progressing the Births, Deaths, Marriages and Relationships Registration Bill, Minister of Internal Affairs, Jan Tinetti announced today. “This Government understands that self-identification is a significant issue for transgender, non-binary and intersex New Zealanders, and ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Crown speeds up engagement with takutai moana applicants
    The Crown is taking a new approach to takutai moana applications to give all applicants an opportunity to engage with the Crown and better support the Māori-Crown relationship, Treaty of Waitangi Negotiations Minister Andrew Little says. Following discussions with applicant groups, the Crown has reviewed the existing takutai moana application ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Alcohol and Other Drug Treatment (AODT) Court opens
    The Minister of Justice, Kris Faafoi, and the Minister for Courts, Aupito William Sio, have welcomed the opening of a new Alcohol and Other Drug Treatment (AODT) Court in Hamilton. The AODT Court (Te Whare Whakapiki Wairua) addresses situations where substance abuse and offending are intertwined. “New Zealanders have told ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • EU and UK FTAs top of list for first ministerial trip since COVID-19
    Trade and Export Growth Minister Damien O’Connor today announced details of his planned visit to the United Kingdom and European Union next week, where he will hold trade and agriculture discussions to further New Zealand’s economic recovery from COVID-19. The visit will add political weight to ongoing negotiations with both the EU ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Arihia Bennett to chair Royal Commission Ministerial Advisory Group
    Te Rūnanga o Ngāi Tahu chief executive Arihia Bennett MNZM has been appointed chair of the newly appointed Ministerial Advisory Group on the Government’s Response to the Royal Commission of Inquiry into the terrorist attack on Christchurch mosques. “Twenty-eight people from diverse backgrounds across Aotearoa have been selected for the ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Speech to the New Zealand Medical Association General Practitioners' Conference, Rotorua
    Ki ngā pou maha o te whare hauora o Aotearoa, kei te mihiTo the pillars of our health system I acknowledge/thank you Ki te ope hapai hauora o roto o tēnei rūma, kei te mihi To our health force here in the room today, I acknowledge/thank you He taura tangata, ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Karangahape Road upgrades are streets ahead
    The upgrades to Karangahape Road makes the iconic street more pedestrian and cycle-friendly, attractive and environmentally sustainable, Transport Minister Michael Wood and Auckland Mayor Phil Goff said at the formal celebration of the completion of the Karangahape Road Enhancements project. The project included widening footpaths supporting a better outdoor dining ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Speech to APEC business event
    E ngā tumu herenga waka, ākina ā ngaru, ākina ā tai ka whakatere ngā waka ki te whakapapa pounamu, otirā, ki Tamaki o ngā waka Tena koutou katoa… To the great leaders assembled, who guided your waka through turbulent times, challenging waters and you continue to navigate your respective waka ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Pause on Quarantine Free Travel with Victoria extended
    Following an assessment of the COVID-19 outbreak in greater Melbourne, New Zealand’s Quarantine Free Travel pause with Victoria will continue for a further seven days, COVID-19 Response Minister Chris Hipkins says. There are now 93 cases associated with the outbreak in greater Melbourne, spread over four clusters. Contact tracing efforts ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Supplier Diversity Aotearoa Summit: Navigate 2021
    *** Check with delivery *** A mihi to all who have contributed to making today a success – starting with you! As you have explored and navigated government procurement today you will hopefully have reflected on the journey of our people so far – and how you can make a ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Pukemiro School to close
    Pukemiro Primary School near Huntly will close following years of declining roll numbers, Education Minister Chris Hipkins announced today. “I’ve consulted with the School Commissioner, and this decision acknowledges the fact that the few remaining students from last term are now settled at other nearby schools. “I want to thank ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Govt acts to protect NZers from harmful content
    New Zealanders will be better protected from harmful or illegal content as a result of work to design a modern, flexible and coherent regulatory framework, Minister of Internal Affairs Jan Tinetti announced today. New Zealand currently has a content regulatory system that is comprised of six different arrangements covering some ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Consultation on exemption of new builds from proposed tax rules
    The Government has today confirmed new builds will be exempt from planned changes to the tax treatment of residential investment property.  Public consultation is now open on details of the proposals, which stop interest deductions being claimed for residential investment properties other than new builds.   “The Government’s goal is to ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Speech for Predator Free 2050 Conference
    Introduction E ngā mana E ngā reo E ngā iwi Tēnā koutou katoa   Ka huri ki ngā mana whenua o te rohe nei. Tēnā koutou He mihi hoki ki a tatou kua tau mai nei i raro i te kaupapa o te rā Ko Ayesha Verrall toku ingoa No ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • New stock exchange to help grow small businesses
    A new share trading market, designed as a gateway to the NZX for small-to-medium-sized enterprises (SMEs), has been granted a licence by the Government. Commerce and Consumer Affairs Minister, David Clark said Catalist Markets Ltd will provide a simpler and more affordable ‘stepping stone’ for SMEs to raise capital. “This ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Visa extensions provide certainty to employers and 10,000 visa holders
    Changes to onshore visas will provide employers and visa holders with more certainty, Immigration Minister Kris Faafoi has announced. Around 10,000 Working Holiday visas and Supplementary Seasonal Employment (SSE) work visas due to expire between 21 June 2021 and 31 December 2021 will be extended for another six months to ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Border class exceptions approved for more farm workers and vets
    The Government has approved border class exceptions for an additional 200 dairy workers and 50 veterinarians to enter New Zealand, Agriculture Minister Damien O’Connor announced today.  “It is clear from conversations with the dairy and veterinarian sectors that they are facing workforce pressures. These border exceptions will go a long ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • More freezers and South Island hub to support vaccine roll-out
    A South Island hub and 17 new ultra-low temperature freezers will help further prepare New Zealand for the ramp up of the vaccination programme in the second half of this year, COVID-19 Response Minister Chris Hipkins says. The new freezers arrived in New Zealand on 27 May. They’re currently being ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Speech at the release of Climate Change Commission's final advice
    Good morning – and thank you Prime Minister. Over the last three and half years we have been putting in place the foundations for a low-carbon Aotearoa that will be a catalyst for job creation, innovation, and prosperity for decades to come. In that future, many of our everyday tasks ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Achievable blueprint for addressing climate change released
    Report says Government making good progress on emissions reduction, but more action required Meeting climate targets achievable and affordable with existing technology Economic cost of delaying action higher than taking action now Benefits from climate action include health improvements and lower energy bills All Ministers to help meet climate targets ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Speech to release of Climate Commission final report
    A few years ago in a speech in Auckland, I compared climate change to the nuclear free movement of roughly four decades ago. And I did so for a few reasons. Firstly, because the movement of the 1980s represented a life or death situation for the Pacific, and so does ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Appointment of Judge of the High Court
    Auckland Barrister Michael Robinson has been appointed a Judge of the High Court, Attorney‑General David Parker announced today. Justice Robinson graduated with a BA and an LLB (Hons) from the University of Auckland in 1996, and commenced practice as a solicitor with Brookfields in Auckland.  In 1998 he travelled to London ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Government takes action to improve protections for subcontractors
    The Construction Contracts (Retention Money) Amendment Bill – which provides greater financial protection for subcontractors, has passed its first reading today. The Bill amends the retention provisions in the Construction Contracts Act 2002 (CCA) to provide increased confidence and transparency for subcontractors that retention money they are owed is safe. ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • 1 million more Pfizer doses to arrive in July
    Pfizer has scheduled delivery of an estimated 1 million doses of vaccine to New Zealand during July, COVID1-9 Response Minister Chris Hipkins said today. “These consignments will double the total number of Pfizer doses we have received this year to more than 1,900,000 – enough to fully vaccinate almost 1 ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Long-term home of the Independent Children’s Monitor identified
    The Independent Children’s Monitor (Te Mana Whakamaru Tamariki Motuhake), which is currently located within the Ministry of Social Development (MSD), will become its own departmental agency within Government. “Following the recommendations of several reviews, Cabinet agreed in 2019 to build a significantly expanded independent monitor for children in care,” Carmel ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Racing Integrity Board members announced
    The new Racing Integrity Board will be up and running from July 1 to ensure high standards of animal welfare, integrity and professionalism in the racing industry. Racing Minister Grant Robertson today announced the appointments to the new Board: Sir Bruce Robertson KNZM – Chair Kristy McDonald ONZM QC Penelope ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Govt crackdown on organised crime continues
    A major operation against multiple organised crime groups with international links will make a significant dent in drug harm and violent offending linked to organised crime networks, Police Minister Poto Williams says. “I want to take an opportunity to congratulate the Police for their role in Operation Trojan Shield. This ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Farm planning framework supports farmers into the future
    A new framework, agreed between Government and industry, will make it easier for farmers and growers to integrate future greenhouse gas emissions and freshwater regulatory requirements into their farm planning, Agriculture Minister Damien O’Connor said. “The Good Farm Planning Principles Guide out today, provides guidance for how farmers can organise ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Enhanced Task Force Green Approved for Canterbury
    The Government has activated Enhanced Taskforce Green (ETFG) in response to the Canterbury floods. The Minister of Social Development and Employment, Hon Carmel Sepuloni says $500,000 will be made available to help with the clean-up. The flooding in Canterbury has been a significant and adverse event damaging farmland, homes, roads ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Connecting rangatahi to the soil
    A Jobs for Nature project to raise 480,000 native plants in nurseries across South Auckland will provide work for communities disproportionately affected by the economic impacts of COVID-19, Acting Conservation Minister Ayesha Verrall says. The Mana in Kaimahi project is being run by Te Whāngai Trust Board and will establish ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Roll out of high-resolution elevation mapping begins
    The first tranche of mapping data from the Provincial Growth Fund (PGF)-LiDAR project is now available to the public from Toitū Te Whenua Land Information New Zealand. LiDAR data, which creates 3D baseline elevation information, will deliver multiple uses over the coming decades to councils and regional industries. “This mapping ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Champions of Pacific education rewarded in Queen’s Birthday Honours
    Minister for Pacific Peoples, Aupito William Sio said the Queen’s Birthday 2021 Honours list show that across Aotearoa New Zealand there were many champions of Pacific education. “Education is so vital to the success of Pacific people that it’s truly fitting that a number of educators have been honoured this ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • PM congratulates Queen’s Birthday Honours recipients
    Prime Minister Jacinda Ardern has added her congratulations to the New Zealanders recognised for their contributions to their communities and the country in the Queen’s Birthday 2021 Honours List. “This group represents decades of services across many areas, and those honoured highlight how many New Zealanders are going above and ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Change of status for Rangiriri kura
    A change of status for Te Kura o Rangiriri sees it become a designated character school within the Māori-medium network, Associate Minister of Education Kelvin Davis announced today. “This kura has been providing Māori immersion learning since 2003 in the historic town of Rangiriri, so I’m delighted that it is ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • APEC trade ministers’ unite on COVID-19 vaccine steps and rejuvenating the WTO
    APEC trade ministers today committed to speeding up the cross-border flow of vaccines and related goods to fight the COVID-19 pandemic. This followed the completion of the APEC Ministers Responsible for Trade Meeting chaired by Minister for Trade and Export Growth Damien O’Connor early this morning. “As we face the ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Further consultation for Melville schools
    Formal consultation is set to begin on specific options for the future of schooling in South West Hamilton, Education Minister Chris Hipkins says. “Recent engagement has shown that the schools and community want a change to Year 7-13 schooling for the area.  “I am now asking the Boards of Melville ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Primary schoolkids dig trees for Arbor Day
    A new Arbor Day initiative announced by Forestry Minister Stuart Nash will see thousands of primary school children get the chance to plant native trees in their communities. The initiative is open to more than 2,400 primary schools. It is a partnership between Te Uru Rākau/NZ Forest Service and the ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • NZ Upgrade Programme kept on track
    The Government is increasing its investment in the New Zealand Upgrade Programme (NZUP) to support New Zealand’s economic recovery. Over two thirds of the projects will proceed as announced despite increased costs due to COVID, with modifications being made to others. “NZUP is already supporting over 1,000 jobs with 13 ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Northern Pathway gets green light
    The Government is committed to the Northern Pathway with its preferred option being a separate structure for walking and cycling alongside the Auckland Harbour Bridge, Transport Minister Michael Wood announced today. Geotechnical investigations and testing has determined that building a structure connected to the Auckland Harbour Bridge is not possible ...
    BeehiveBy beehive.govt.nz
    1 week ago