Routing issues

Written By: - Date published: 11:08 am, April 1st, 2011 - 9 comments
Categories: admin, The Standard - Tags:

We have had some routing problems this morning with several networks used by our provider in San Diego. The problem have been ongoing with the network links appearing and disappearing from the perspective of NZ and aussie (there is less of a issue from overseas networks). They’re working on it. It started just before 7am and is starting to look stable at about 11am.

Ironically, I’m just waiting for some due diligence to be done on a local supplier for the election year server – ie that they won’t just fold if a malicious complaint is made. The current warm backup system can’t handle the load (it was having problems handling the comments being fed to it), so it is currently off. The new server was due to get configured this weekend but that is looking less likely because it is unlikely to get provisioned before Monday.

And no, this isn’t an April fools joke…. But since we last had a failure almost exactly a year ago at the end of March running over April 1, I’m starting to get superstitious.

9 comments on “Routing issues ”

  1. Rich 1

    Access seems fine right now.

    Also, I use Dreamhosts. Cheap, reliable, good customer service, and a reasonable AUP. I’m not sure how robust they are to takdown complaints, but their polices seem quite firm. (http://abuse.dreamhost.com/libel/ for instance).

    Also, if you need a backup site, I may be able to provide something for free. Email me?

    • lprent 1.1

      Will do.

      These days the primary server is running on a reasonably recent dual core dedicated Xeon that started at running well under 20% normal’ peak and 40% spiking CPU during the day last March, and now runs 40% normal peak and 80% spiking. The spiking typically happens when posts are being made with the SEO notifications kicking off demand, whilst at the same time background processes like the search indexing run, and we have multiple comments happening at the same time. Unfortunately I have to do capacity planning based on the spikes rather than the normal and it is difficult to prevent spiking in software (ie it is an OR queuing problem). So I need more cores and more CPU cycles.

      The idea is to move to a much higher spec system in NZ and leave our existing US system running in the warm backup role. The latter should have enough grunt in an emergency to carry us through the election year growth (probably about another 50% page views) if I turn off some of the background processes.

      If we can’t find something suitable in NZ, then it is pretty easy to get offshore servers with more cores and more CPU cycles and where we don’t have the potential malicious legal issues we have in NZ.

  2. lprent 2

    FYI from the hosting company

    We wanted to provide you with an update about the network connectivity
    issues we were seeing today. As of approximately 3pm PT the network has
    returned to normal and we are receiving assurances from our network
    provider that the situation should remain stable. Latency will also
    improve as additional carriers are brought online.

    As to what happened, we are still not being given a clear picture. The
    engineers working on the issue will provide a detailed explanation as soon
    as the situation has been fully resolved and understood. We are more than
    happy to pass that engineering report along to you (please just let us
    know you’d like to receive it.) What we’ve been told is there was a
    routing issue with one of our carriers (XO) which propagated to our other
    carriers- we have connections from multiple backbone providers to prevent
    problems or outages with one carrier from causing a complete outage. We
    realize that has absolutely not been the case today, and we will be
    receiving an explanation and plan for how this can be avoided in the
    future.

    This type of disruption is somewhat sporadic in that traffic from some
    ISPs and networks was affected, while traffic from others was unaffected.
    Also, as the routes were restored via backup carriers, you may have
    noticed that the connection would work briefly then go down again. There
    my be some more of these hiccups as this is brought to resolution in the
    next 12-24 hours, however we expect those to me momentary (on the order of
    a few seconds.)

    Please know we will continue to monitor this situation very closely and we
    will be back in touch if further issues arise.

    Not quite as bad as the hosting companies I have had for a different operations.

    One was in Florida the day that they had a really good hurricane come through. They had multiple failures in their connection providers resulting in only a single connection staying active. The volume of traffic was such that it kept overloading. After that we always kept the servers separated in different sites and the sites were selected for their lack of expected natural disasters

    But of course we had a different hoster that had a contractor for a company down the street from them ram a metal girder straight through their high voltage underground power line. The power surge was sufficient to fry the switches that were meant to divert them to the battery backup system (and the generators that were meant to feed that). Consequently our servers in that site were out for a day while they rewired everything.

    Disaster planning is a bit of a pain when traffic keeps rising. You usually find out it doesn’t work when the systems fail. Of course you could do what I did last month. I forced a test failure and found out that the warm backup couldn’t even get close to handling normal loads.

    Anyway, I think I’d better order that server today. They may be able to provision it tomorrow. Otherwise it will be a week before I have time to do the setup.

  3. chris 3

    Use linode. Their VPS service is amazing and cheap! Steer clear of dreamhost, as a programmer I would expect nothing less than self-configured servers 😛

    • lprent 3.1

      We can’t survive on VPS’es. If you look at the ‘Online’ tab on the upper right you will see why. That is a picture of the number of readers (including the spiders) within a couple of minutes on the site. During our day that usually sits between 40-60. But it goes up as high as 150 when comment spikes. Overnight it drops down to 10-15 base load because that is when many of the spiders pick up their data steadily reading their way through the 7000 odd posts and 250,000 odd comments that are online.

      Remember that these are unique IP’s. There are many many connections within that count for all of those little graphics, CSS and JS.

      We have been booted off three VPS’es for having too much traffic so far. The last one was running purely as a warm backup storing comment and post updates. Right now we’re on the verge of having to start spreading across several servers to handle traffic, which is what I intend to set up when the new server comes online. NZ readers will read the NZ server, and overseas users (mostly spiders) will read the US server.

  4. Lanthanide 4

    “Overnight it drops down to 10-15 base load because that is when many of the spiders pick up their data steadily reading their way through the 7000 odd posts and 250,000 odd comments that are online.”

    Isn’t there some way to force the spiders not to re-cache this stuff all the time? Most of those pages and comments won’t be changing very frequently (if ever), so they should be able to index it once and then not need to come back for a couple of months, and only then just to check that the pages are still valid.

  5. BLiP 5

    What’s that cdn.topsy thing that can take f o r e v e r to load?

Recent Comments

Recent Posts

  • Decision allows for housing growth in Western Bay of Plenty
    The Minister Responsible for RMA Reform, Chris Bishop today released his decision on four recommendations referred to him by the Western Bay of Plenty District Council, opening the door to housing growth in the area. The Council’s Plan Change 92 allows more homes to be built in existing and new ...
    BeehiveBy beehive.govt.nz
    7 hours ago
  • Speech to New Zealand China Council
    Thank you, John McKinnon and the New Zealand China Council for the invitation to speak to you today.    Thank you too, all members of the China Council. Your effort has played an essential role in helping to build, shape, and grow a balanced and resilient relationship between our two ...
    BeehiveBy beehive.govt.nz
    10 hours ago
  • Modern insurance law will protect Kiwi households
    The Government is modernising insurance law to better protect Kiwis and provide security in the event of a disaster, Commerce and Consumer Affairs Minister Andrew Bayly announced today. “These reforms are long overdue. New Zealand’s insurance law is complicated and dated, some of which is more than 100 years old. ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Government recommits to equal pay
    The coalition Government is refreshing its approach to supporting pay equity claims as time-limited funding for the Pay Equity Taskforce comes to an end, Public Service Minister Nicola Willis says.  “Three years ago, the then-government introduced changes to the Equal Pay Act to support pay equity bargaining. The changes were ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Transforming how our children learn to read
    Structured literacy will change the way New Zealand children learn to read - improving achievement and setting students up for success, Education Minister Erica Stanford says.  “Being able to read and write is a fundamental life skill that too many young people are missing out on. Recent data shows that ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • NZ not backing down in Canada dairy dispute
    Trade Minister Todd McClay says Canada’s refusal to comply in full with a CPTPP trade dispute ruling in our favour over dairy trade is cynical and New Zealand has no intention of backing down. Mr McClay said he has asked for urgent legal advice in respect of our ‘next move’ ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Stronger oversight for our most vulnerable children
    The rights of our children and young people will be enhanced by changes the coalition Government will make to strengthen oversight of the Oranga Tamariki system, including restoring a single Children’s Commissioner. “The Government is committed to delivering better public services that care for our most at-risk young people and ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Streamlining Building Consent Changes
    The Government is making it easier for minor changes to be made to a building consent so building a home is easier and more affordable, Building and Construction Minister Chris Penk says.      “The coalition Government is focused on making it easier and cheaper to build homes so we can ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Minister acknowledges passing of Sir Robert Martin (KNZM)
    New Zealand lost a true legend when internationally renowned disability advocate Sir Robert Martin (KNZM) passed away at his home in Whanganui last night, Disabilities Issues Minister Louise Upston says. “Our Government’s thoughts are with his wife Lynda, family and community, those he has worked with, the disability community in ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Speech to New Zealand Institute of International Affairs, Parliament – Annual Lecture: Challenges ...
    Good evening –   Before discussing the challenges and opportunities facing New Zealand’s foreign policy, we’d like to first acknowledge the New Zealand Institute of International Affairs. You have contributed to debates about New Zealand foreign policy over a long period of time, and we thank you for hosting us.  ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Accelerating airport security lines
    From today, passengers travelling internationally from Auckland Airport will be able to keep laptops and liquids in their carry-on bags for security screening thanks to new technology, Transport Minister Simeon Brown says. “Creating a more efficient and seamless travel experience is important for holidaymakers and businesses, enabling faster movement through ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Community hui to talk about kina barrens
    People with an interest in the health of Northland’s marine ecosystems are invited to a public meeting to discuss how to deal with kina barrens, Oceans and Fisheries Minister Shane Jones says. Mr Jones will lead the discussion, which will take place on Friday, 10 May, at Awanui Hotel in ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Kiwi exporters win as NZ-EU FTA enters into force
    Kiwi exporters are $100 million better off today with the NZ EU FTA entering into force says Trade Minister Todd McClay. “This is all part of our plan to grow the economy. New Zealand's prosperity depends on international trade, making up 60 per cent of the country’s total economic activity. ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Mining resurgence a welcome sign
    There are heartening signs that the extractive sector is once again becoming an attractive prospect for investors and a source of economic prosperity for New Zealand, Resources Minister Shane Jones says. “The beginnings of a resurgence in extractive industries are apparent in media reports of the sector in the past ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Ō-Rākau Remembrance Bill passes first reading
    The return of the historic Ō-Rākau battle site to the descendants of those who fought there moved one step closer today with the first reading of Te Pire mō Ō-Rākau, Te Pae o Maumahara / The Ō-Rākau Remembrance Bill. The Bill will entrust the 9.7-hectare battle site, five kilometres west ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Government to boost public EV charging network
    Energy Minister Simeon Brown has announced 25 new high-speed EV charging hubs along key routes between major urban centres and outlined the Government’s plan to supercharge New Zealand’s EV infrastructure.  The hubs will each have several chargers and be capable of charging at least four – and up to 10 ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Residential Property Managers Bill to not progress
    The coalition Government will not proceed with the previous Government’s plans to regulate residential property managers, Housing Minister Chris Bishop says. “I have written to the Chairperson of the Social Services and Community Committee to inform him that the Government does not intend to support the Residential Property Managers Bill ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Independent review into disability support services
    The Government has announced an independent review into the disability support system funded by the Ministry of Disabled People – Whaikaha. Disability Issues Minister Louise Upston says the review will look at what can be done to strengthen the long-term sustainability of Disability Support Services to provide disabled people and ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Justice Minister updates UN on law & order plan
    Justice Minister Paul Goldsmith has attended the Universal Periodic Review in Geneva and outlined the Government’s plan to restore law and order. “Speaking to the United Nations Human Rights Council provided us with an opportunity to present New Zealand’s human rights progress, priorities, and challenges, while responding to issues and ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Ending emergency housing motels in Rotorua
    The Government and Rotorua Lakes Council are committed to working closely together to end the use of contracted emergency housing motels in Rotorua. Associate Minister of Housing (Social Housing) Tama Potaka says the Government remains committed to ending the long-term use of contracted emergency housing motels in Rotorua by the ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Trade Minister travels to Riyadh, OECD, and Dubai
    Trade Minister Todd McClay heads overseas today for high-level trade talks in the Gulf region, and a key OECD meeting in Paris. Mr McClay will travel to Riyadh to meet with counterparts from Saudi Arabia and the Gulf Cooperation Council (GCC). “New Zealand’s goods and services exports to the Gulf region ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Education priorities focused on lifting achievement
    Education Minister Erica Stanford has outlined six education priorities to deliver a world-leading education system that sets Kiwi kids up for future success. “I’m putting ambition, achievement and outcomes at the heart of our education system. I want every child to be inspired and engaged in their learning so they ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • NZTA App first step towards digital driver licence
    The new NZ Transport Agency (NZTA) App is a secure ‘one stop shop’ to provide the services drivers need, Transport Minister Simeon Brown and Digitising Government Minister Judith Collins say.  “The NZTA App will enable an easier way for Kiwis to pay for Vehicle Registration and Road User Charges (RUC). ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Supporting whānau out of emergency housing
    Whānau with tamariki growing up in emergency housing motels will be prioritised for social housing starting this week, says Associate Housing Minister Tama Potaka. “Giving these whānau a better opportunity to build healthy stable lives for themselves and future generations is an essential part of the Government’s goal of reducing ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Tribute to Dave O'Sullivan
    Racing Minister Winston Peters has paid tribute to an icon of the industry with the recent passing of Dave O’Sullivan (OBE). “Our sympathies are with the O’Sullivan family with the sad news of Dave O’Sullivan’s recent passing,” Mr Peters says. “His contribution to racing, initially as a jockey and then ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Speech – Eid al-Fitr
    Assalaamu alaikum, greetings to you all. Eid Mubarak, everyone! I want to extend my warmest wishes to you and everyone celebrating this joyous occasion. It is a pleasure to be here. I have enjoyed Eid celebrations at Parliament before, but this is my first time joining you as the Minister ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Government saves access to medicines
    Associate Health Minister David Seymour has announced Pharmac’s largest ever budget of $6.294 billion over four years, fixing a $1.774 billion fiscal cliff.    “Access to medicines is a crucial part of many Kiwis’ lives. We’ve committed to a budget allocation of $1.774 billion over four years so Kiwis are ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Pharmac Chair appointed
    Hon Paula Bennett has been appointed as member and chair of the Pharmac board, Associate Health Minister David Seymour announced today. "Pharmac is a critical part of New Zealand's health system and plays a significant role in ensuring that Kiwis have the best possible access to medicines,” says Mr Seymour. ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Taking action on Fetal Alcohol Spectrum Disorder
    Hundreds of New Zealand families affected by Fetal Alcohol Spectrum Disorder (FASD) will benefit from a new Government focus on prevention and treatment, says Health Minister Dr Shane Reti. “We know FASD is a leading cause of preventable intellectual and neurodevelopmental disability in New Zealand,” Dr Reti says.  “Every day, ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • New sports complex opens in Kaikohe
    Regional Development Minister Shane Jones today attended the official opening of Kaikohe’s new $14.7 million sports complex. “The completion of the Kaikohe Multi Sports Complex is a fantastic achievement for the Far North,” Mr Jones says. “This facility not only fulfils a long-held dream for local athletes, but also creates ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Diplomacy needed more than ever
    Foreign Minister Winston Peters’ engagements in Türkiye this week underlined the importance of diplomacy to meet growing global challenges.    “Returning to the Gallipoli Peninsula to represent New Zealand at Anzac commemorations was a sombre reminder of the critical importance of diplomacy for de-escalating conflicts and easing tensions,” Mr Peters ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Anzac Commemorative Address, Buttes New British Cemetery Belgium
    Ambassador Millar, Burgemeester, Vandepitte, Excellencies, military representatives, distinguished guests, ladies and gentlemen – good morning and welcome to this sacred Anzac Day dawn service.  It is an honour to be here on behalf of the Government and people of New Zealand at Buttes New British Cemetery, Polygon Wood – a deeply ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Anzac Commemorative Address – NZ National Service, Chunuk Bair
    Distinguished guests -   It is an honour to return once again to this site which, as the resting place for so many of our war-dead, has become a sacred place for generations of New Zealanders.   Our presence here and at the other special spaces of Gallipoli is made ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Anzac Commemorative Address – Dawn Service, Gallipoli, Türkiye
    Mai ia tawhiti pamamao, te moana nui a Kiwa, kua tae whakaiti mai matou, ki to koutou papa whenua. No koutou te tapuwae, no matou te tapuwae, kua honoa pumautia.   Ko nga toa kua hinga nei, o te Waipounamu, o te Ika a Maui, he okioki tahi me o ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • PM announces changes to portfolios
    Paul Goldsmith will take on responsibility for the Media and Communications portfolio, while Louise Upston will pick up the Disability Issues portfolio, Prime Minister Christopher Luxon announced today. “Our Government is relentlessly focused on getting New Zealand back on track. As issues change in prominence, I plan to adjust Ministerial ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New catch limits for unique fishery areas
    Recreational catch limits will be reduced in areas of Fiordland and the Chatham Islands to help keep those fisheries healthy and sustainable, Oceans and Fisheries Minister Shane Jones says. The lower recreational daily catch limits for a range of finfish and shellfish species caught in the Fiordland Marine Area and ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Minister welcomes hydrogen milestone
    Energy Minister Simeon Brown has welcomed an important milestone in New Zealand’s hydrogen future, with the opening of the country’s first network of hydrogen refuelling stations in Wiri. “I want to congratulate the team at Hiringa Energy and its partners K one W one (K1W1), Mitsui & Co New Zealand ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Urgent changes to system through first RMA Amendment Bill
    The coalition Government is delivering on its commitment to improve resource management laws and give greater certainty to consent applicants, with a Bill to amend the Resource Management Act (RMA) expected to be introduced to Parliament next month. RMA Reform Minister Chris Bishop has today outlined the first RMA Amendment ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Overseas decommissioning models considered
    Overseas models for regulating the oil and gas sector, including their decommissioning regimes, are being carefully scrutinised as a potential template for New Zealand’s own sector, Resources Minister Shane Jones says. The Coalition Government is focused on rebuilding investor confidence in New Zealand’s energy sector as it looks to strengthen ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Release of North Island Severe Weather Event Inquiry
    Emergency Management and Recovery Minister Mark Mitchell has today released the Report of the Government Inquiry into the response to the North Island Severe Weather Events. “The report shows that New Zealand’s emergency management system is not fit-for-purpose and there are some significant gaps we need to address,” Mr Mitchell ...
    BeehiveBy beehive.govt.nz
    1 week ago

Page generated in The Standard by Wordpress at 2024-05-03T07:32:06+00:00