How this site’s webservers now work

Written By: - Date published: 4:52 pm, April 18th, 2014 - 8 comments
Categories: admin, The Standard - Tags:

Dolphin Over World 2010This site gets rather a lot of comments. Every time a comment is made the page it is on has to be regenerated, which limits the amount of static caching that can be done. So on a dynamic site like this one we require rather a lot of raw processing power to generate dynamically pages that keep changing all of the day.

There are a number of solutions for this issue from offloading the comments to an external supplier like Disqus (with the consequent issues about privacy and control) or to the excessive caching of pages when new comments come in (causing out of date pages) or to separating the comments into a dynamically loaded system (requiring extra client side javascript).

But we like to make the site as simple as possible, as up to date as possible and as secure as possible. So when comments arrive, this site updates the site.

Because of the current design of the pages (something that is always under review), this means that whenever a comment is posted, all pages with the list of recent comments is invalid. Because that is on every page to assist with navigation, the next time a page is read by anyone it is recalculated and laid out. While the server will cache that page, it will do so for a particular version of the browser that it was destined for. That is because not all pages are served up the same. It depends on what type of browser asked for it. So a page may be regenerated many times.

That requires a significiant amount of processing power when there are a lot of people commenting at the same time as there are lot of people reading the site. Outside of the middle of the night, those demand levels tend to quite unpredictable.

It’d be pretty easy to simply get bigger servers that could handle all of the possible demand. But how big is enough? We have had times when a story has been breaking where the load on the server has been for short periods of time been more than 10 times the usual load. We’ve had times when those peaks have killed the site running with a powerful 8 core system. During an election year we usually get a lot more traffic than we previously have had.

So since late last year we have been running the web facing parts of this site on cloud servers and setting up the required infrastructure to allow us to run multiple servers with common file, database, and page/query cache services.

This years we’ve been steadily getting a auto-scaling system largely triggered by CPU levels working. For a single webserver we use something that is rather like a dual core CPU laptop. Not massively fast, but capable of handling a moderate amount of traffic simultaneously. This is our quanta.

When the average processing usage for the existing web servers rises about a set value for more than a few minutes, then a new server of that type is automatically hired from Amazon, equipped with our standard disk image, and thrown on to the load balancer. If the demand keeps rising, then so do the number of servers in usage.

Conversely when the average processing usage drops below a certain value for a long period, then a server is dropped. This keeps happening until the servers are working productively. They go off to do some other type of task for someone else.

This kind of on-demand system is damn hard on people because they really don’t like chopping and changing all of the time as it is intensely unproductive to keep retraining people for different tasks. But computers are machines. You give them a new filesystem and they are trained. The hard bit is creatively figuring out what should be on that filesystem – a task that humans are usually pretty good at.

Anyway, I thought I’d share what our webservers have been doing over the last couple of weeks. Bear in mind that the times on this graph are in UTC (Universal time or Greenwich time) so they are currently about 12 hours off NZ time.

Web server loadings

You can see the strong diurnal cycle (click through the image for a higher resolution one if you cannot). The servers drop down to two or even one overnight. You can also see the weekends and public holidays where the servers during the day run a three and sometimes four. During the week days we’re usually running at four and sometimes five servers.

Each server costs about 2.8c per hour to have (with variable costs like bandwidth as extra), so while it doesn’t cost much, it adds up over a month to save the overnight costs. With a voluntary site like this all money saved is pretty useful.

But you can also see peaks for short bursts where we are running at six, seven or even eight servers when a story goes active and everyone piles into the site. And this fortnight has been a quiet period news wise.

I’ve currently capped the maximum numbers of web servers at 35 based on the potential loads on the database and file servers. Hopefully that will be enough to cope with the upcoming demand. Besides, I can always start autoscaling those if I really have to…

8 comments on “How this site’s webservers now work ”

  1. andy (the other one) 1

    Sorry, some times I hit F5 too much. mostly when felix or Sanc are trolling Syrlands or Pete G.

    Its my online crack habit…

    Its irresistable.

    • lprent 1.1

      I don’t care. It is literally what the system was designed for.

      BUT don’t do it more often than 15 times in a minute. If you do then you will find the site abruptly slows down on you for the next two hours because Wordfence now treats you as a pain-in-the-arse. I find that this seems to induce the change in behaviour that I’m after.

      • George 1.1.1

        Great work Lynn. And thank goodness I don’t have to use Disqus – which simultaneously destroys the small measure of privacy I use and slows comments to a crawl.

  2. tjp 2

    So if I read the above correctly it sounds as if the majority of your problems in terms of scalability are due to the frequent cache invalidations caused by new comments? I would be interested in hearing more about why you don’t want to load them dynamically – the additional client coding sounds like a relatively minor tradeoff. It wouldn’t solve the problem of caching multiple versions of the page for different clients but you would be able to hold them in cache longer.

    • lprent 2.1

      Oh I agree that it’d appear to make the site simpler to run. However the coding/testing complexity of doing it is a pain when you look at the detail. And it wouldn’t make much difference in terms of actual operations because of the site layout. We’d have to do a lot more coding to make it work.

      Depends what we’re talking about of course….

      But if I dynamically loaded the comments on the right tab then I’d get benefits for the pages that weren’t getting comments added to them and weren’t the front page.

      However at any given time between a quarter and a half of our page views are the front page. A prominent feature on the front page is the comment count. Every comment causes that page to regenerate. Of course that count could also become dynamic – but it has just doubled the dynamic code to test and maintain.

      At any given time between a quarter and a half of our page views are usually on a single post. This is usually because that particular post is getting a lot of comments, a lot of reads by people commenting, and a lot of reads by the substantial lurker population who read comments closely.

      During the day, about a fifth of the page views are from search engines plowing their way through the nearly 14k posts and 700k comments. We help them as much as we can by telling them where the site is active. But they are a very important source of new readers to the site.

      This means that most of the pages being read through the day and evening are on the most dynamic pages anyway. Usually inside a full day something like 75% of the page views will fall in on less than 10 pages. These will also be the pages that have the most comments being made (or in the case of the front page, comments being reported)

      To be effective, we’d have to do the same code changes amongst ALL of the browsers that access this site. As well as the browsers for the desktop versions of windows, mac and linux, it also includes the ever increasing numbers of variants being used on the mobiles (14% of page views). Just testing those on every one of the frequent code updates for wordpress and plugins beggars belief. The only real way to do it would be to write automated testers – which would be a hell of a coding chore.

      And of course wordpress directly has no support for dynamic comments, which means that we’d be on our own for testing updates across all of those targets platforms. The testing would have to be done with every update on wordpress.

      There are a number of possible alternatives system that do get their own testing done (Disqus for instance).

      But this is a site that gets quite a lot of comments, many of them large and just as interesting as the posts. There is a considerable SEO benefit in optimizing for them – which we have done. So we’d also have to do something about how the search engines access the comments. For instance I ran your first sentence through google and got an immediate hit. If you do the same on a disqus site like whaleoil – the same will not happen. As far as I can tell (a notoriously difficult thing to measure) about a third of the retained new readers come in after querying something that got a strike on a comment.

      There are a whole lot of other issues as well (like the ever shifting jQuery/Ajax systems). Suffice it to say that doing it with servers appeared to be a lot simpler than doing it with code because the returns off doing it with code are a lot less than you’d expect, and the cost of the code is whole lot higher and less transportable.

      Roll on HTML5 which will make some of this easier to do inside wordpress itself.

  3. lprent 3

    Interesting looking at the Shane Jones effect today on servers

    Click image for a larger view. The times are in UTC, so they are 12 hours out. So the Shane Jones story went up just after 06:00

    The load peaked at approximately 3.5x the usual human average maximum weekday middle of the day load at about 7pm and started falling back to under twice the usual load at about 9ish, then spiked again with the evening news. Then dropped rapidly after that.

    No page errors reported from the client to load balancer, but the latency was somewhat spiky as web servers were being added.

    I need a bigger story to really test the spike loading on the site. It didn’t really touch the database or file server. Comments kept pouring in without any problem apart from some moronic trolls unaware of the first comment policy…

    I’m happy with the performance. Especially for a peak loading at about 17-18c per hour…

    • lprent 3.1

      The traffic is certainly heating up.

      The direct traffic out last month for webpages (excluding the static images, javascript, and CSS) was 309GB (short month feb was 319.996 GB). So far this month as at the 22nd, we’re at 338GB. I guess election year is underway.

      Good to see that the changes I made at the end of last month are reducing all of the other costs down.

Links to post

Recent Comments

Recent Posts

  • Taking action on Fetal Alcohol Spectrum Disorder
    Hundreds of New Zealand families affected by Fetal Alcohol Spectrum Disorder (FASD) will benefit from a new Government focus on prevention and treatment, says Health Minister Dr Shane Reti. “We know FASD is a leading cause of preventable intellectual and neurodevelopmental disability in New Zealand,” Dr Reti says.  “Every day, ...
    BeehiveBy beehive.govt.nz
    11 hours ago
  • New sports complex opens in Kaikohe
    Regional Development Minister Shane Jones today attended the official opening of Kaikohe’s new $14.7 million sports complex. “The completion of the Kaikohe Multi Sports Complex is a fantastic achievement for the Far North,” Mr Jones says. “This facility not only fulfils a long-held dream for local athletes, but also creates ...
    BeehiveBy beehive.govt.nz
    13 hours ago
  • Diplomacy needed more than ever
    Foreign Minister Winston Peters’ engagements in Türkiye this week underlined the importance of diplomacy to meet growing global challenges.    “Returning to the Gallipoli Peninsula to represent New Zealand at Anzac commemorations was a sombre reminder of the critical importance of diplomacy for de-escalating conflicts and easing tensions,” Mr Peters ...
    BeehiveBy beehive.govt.nz
    16 hours ago
  • Anzac Commemorative Address, Buttes New British Cemetery Belgium
    Ambassador Millar, Burgemeester, Vandepitte, Excellencies, military representatives, distinguished guests, ladies and gentlemen – good morning and welcome to this sacred Anzac Day dawn service.  It is an honour to be here on behalf of the Government and people of New Zealand at Buttes New British Cemetery, Polygon Wood – a deeply ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Anzac Commemorative Address – NZ National Service, Chunuk Bair
    Distinguished guests -   It is an honour to return once again to this site which, as the resting place for so many of our war-dead, has become a sacred place for generations of New Zealanders.   Our presence here and at the other special spaces of Gallipoli is made ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Anzac Commemorative Address – Dawn Service, Gallipoli, Türkiye
    Mai ia tawhiti pamamao, te moana nui a Kiwa, kua tae whakaiti mai matou, ki to koutou papa whenua. No koutou te tapuwae, no matou te tapuwae, kua honoa pumautia.   Ko nga toa kua hinga nei, o te Waipounamu, o te Ika a Maui, he okioki tahi me o ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • PM announces changes to portfolios
    Paul Goldsmith will take on responsibility for the Media and Communications portfolio, while Louise Upston will pick up the Disability Issues portfolio, Prime Minister Christopher Luxon announced today. “Our Government is relentlessly focused on getting New Zealand back on track. As issues change in prominence, I plan to adjust Ministerial ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • New catch limits for unique fishery areas
    Recreational catch limits will be reduced in areas of Fiordland and the Chatham Islands to help keep those fisheries healthy and sustainable, Oceans and Fisheries Minister Shane Jones says. The lower recreational daily catch limits for a range of finfish and shellfish species caught in the Fiordland Marine Area and ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Minister welcomes hydrogen milestone
    Energy Minister Simeon Brown has welcomed an important milestone in New Zealand’s hydrogen future, with the opening of the country’s first network of hydrogen refuelling stations in Wiri. “I want to congratulate the team at Hiringa Energy and its partners K one W one (K1W1), Mitsui & Co New Zealand ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Urgent changes to system through first RMA Amendment Bill
    The coalition Government is delivering on its commitment to improve resource management laws and give greater certainty to consent applicants, with a Bill to amend the Resource Management Act (RMA) expected to be introduced to Parliament next month. RMA Reform Minister Chris Bishop has today outlined the first RMA Amendment ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Overseas decommissioning models considered
    Overseas models for regulating the oil and gas sector, including their decommissioning regimes, are being carefully scrutinised as a potential template for New Zealand’s own sector, Resources Minister Shane Jones says. The Coalition Government is focused on rebuilding investor confidence in New Zealand’s energy sector as it looks to strengthen ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Release of North Island Severe Weather Event Inquiry
    Emergency Management and Recovery Minister Mark Mitchell has today released the Report of the Government Inquiry into the response to the North Island Severe Weather Events. “The report shows that New Zealand’s emergency management system is not fit-for-purpose and there are some significant gaps we need to address,” Mr Mitchell ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Justice Minister to attend Human Rights Council
    Justice Minister Paul Goldsmith is today travelling to Europe where he’ll update the United Nations Human Rights Council on the Government’s work to restore law and order.  “Attending the Universal Periodic Review in Geneva provides us with an opportunity to present New Zealand’s human rights progress, priorities, and challenges, while ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Patterson reopens world’s largest wool scouring facility
    Associate Agriculture Minister, Mark Patterson, formally reopened the world’s largest wool processing facility today in Awatoto, Napier, following a $50 million rebuild and refurbishment project. “The reopening of this facility will significantly lift the economic opportunities available to New Zealand’s wool sector, which already accounts for 20 per cent of ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Speech to the Southland Otago Regional Engineering Collective Summit, 18 April 2024
    Hon Andrew Bayly, Minister for Small Business and Manufacturing  At the Southland Otago Regional Engineering Collective (SOREC) Summit, 18 April, Dunedin    Ngā mihi nui, Ko Andrew Bayly aho, Ko Whanganui aho    Good Afternoon and thank you for inviting me to open your summit today.    I am delighted ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Government to introduce revised Three Strikes law
    The Government is delivering on its commitment to bring back the Three Strikes legislation, Associate Justice Minister Nicole McKee announced today. “Our Government is committed to restoring law and order and enforcing appropriate consequences on criminals. We are making it clear that repeat serious violent or sexual offending is not ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • New diplomatic appointments
    Foreign Minister Winston Peters has today announced four new diplomatic appointments for New Zealand’s overseas missions.   “Our diplomats have a vital role in maintaining and protecting New Zealand’s interests around the world,” Mr Peters says.    “I am pleased to announce the appointment of these senior diplomats from the ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Humanitarian support for Ethiopia and Somalia
    New Zealand is contributing NZ$7 million to support communities affected by severe food insecurity and other urgent humanitarian needs in Ethiopia and Somalia, Foreign Minister Rt Hon Winston Peters announced today.   “Over 21 million people are in need of humanitarian assistance across Ethiopia, with a further 6.9 million people ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Arts Minister congratulates Mataaho Collective
    Minister for Arts, Culture and Heritage Paul Goldsmith is congratulating Mataaho Collective for winning the Golden Lion for best participant in the main exhibition at the Venice Biennale. "Congratulations to the Mataaho Collective for winning one of the world's most prestigious art prizes at the Venice Biennale.  “It is good ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Supporting better financial outcomes for Kiwis
    The Government is reforming financial services to improve access to home loans and other lending, and strengthen customer protections, Commerce and Consumer Affairs Minister Andrew Bayly and Housing Minister Chris Bishop announced today. “Our coalition Government is committed to rebuilding the economy and making life simpler by cutting red tape. We are ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Trade relationship with China remains strong
    “China remains a strong commercial opportunity for Kiwi exporters as Chinese businesses and consumers continue to value our high-quality safe produce,” Trade and Agriculture Minister Todd McClay says.   Mr McClay has returned to New Zealand following visits to Beijing, Harbin and Shanghai where he met ministers, governors and mayors and engaged in trade and agricultural events with the New ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • PM’s South East Asia mission does the business
    Prime Minister Christopher Luxon has completed a successful trip to Singapore, Thailand and the Philippines, deepening relationships and capitalising on opportunities. Mr Luxon was accompanied by a business delegation and says the choice of countries represents the priority the New Zealand Government places on South East Asia, and our relationships in ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • $41m to support clean energy in South East Asia
    New Zealand is demonstrating its commitment to reducing global greenhouse emissions, and supporting clean energy transition in South East Asia, through a contribution of NZ$41 million (US$25 million) in climate finance to the Asian Development Bank (ADB)-led Energy Transition Mechanism (ETM). Prime Minister Christopher Luxon and Climate Change Minister Simon Watts announced ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Minister releases Fast-track stakeholder list
    The Government is today releasing a list of organisations who received letters about the Fast-track applications process, says RMA Reform Minister Chris Bishop. “Recently Ministers and agencies have received a series of OIA requests for a list of organisations to whom I wrote with information on applying to have a ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Judicial appointments announced
    Attorney-General Judith Collins today announced the appointment of Wellington Barrister David Jonathan Boldt as a Judge of the High Court, and the Honourable Justice Matthew Palmer as a Judge of the Court of Appeal. Justice Boldt graduated with an LLB from Victoria University of Wellington in 1990, and also holds ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Education Minister heads to major teaching summit in Singapore
    Education Minister Erica Stanford will lead the New Zealand delegation at the 2024 International Summit on the Teaching Profession (ISTP) held in Singapore. The delegation includes representatives from the Post Primary Teachers’ Association (PPTA) Te Wehengarua and the New Zealand Educational Institute (NZEI) Te Riu Roa.  The summit is co-hosted ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Value of stopbank project proven during cyclone
    A stopbank upgrade project in Tairawhiti partly funded by the Government has increased flood resilience for around 7000ha of residential and horticultural land so far, Regional Development Minister Shane Jones says. Mr Jones today attended a dawn service in Gisborne to mark the end of the first stage of the ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Anzac commemorations, Türkiye relationship focus of visit
    Foreign Affairs Minister Winston Peters will represent the Government at Anzac Day commemorations on the Gallipoli Peninsula next week and engage with senior representatives of the Turkish government in Istanbul.    “The Gallipoli campaign is a defining event in our history. It will be a privilege to share the occasion ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Minister to Europe for OECD meeting, Anzac Day
    Science, Innovation and Technology and Defence Minister Judith Collins will next week attend the OECD Science and Technology Ministerial conference in Paris and Anzac Day commemorations in Belgium. “Science, innovation and technology have a major role to play in rebuilding our economy and achieving better health, environmental and social outcomes ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Comprehensive Partnership the goal for NZ and the Philippines
    Prime Minister Christopher Luxon held a bilateral meeting today with the President of the Philippines, Ferdinand Marcos Jr.  The Prime Minister was accompanied by MP Paulo Garcia, the first Filipino to be elected to a legislature outside the Philippines. During today’s meeting, Prime Minister Luxon and President Marcos Jr discussed opportunities to ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government commits $20m to Westport flood protection
    The Government has announced that $20 million in funding will be made available to Westport to fund much needed flood protection around the town. This measure will significantly improve the resilience of the community, says Local Government Minister Simeon Brown. “The Westport community has already been allocated almost $3 million ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Taupō takes pole position
    The Government is proud to support the first ever Repco Supercars Championship event in Taupō as up to 70,000 motorsport fans attend the Taupō International Motorsport Park this weekend, says Economic Development Minister Melissa Lee. “Anticipation for the ITM Taupō Super400 is huge, with tickets and accommodation selling out weeks ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Cost of living support for low-income homeowners
    Local Government Minister Simeon Brown has announced an increase to the Rates Rebate Scheme, putting money back into the pockets of low-income homeowners.  “The coalition Government is committed to bringing down the cost of living for New Zealanders. That includes targeted support for those Kiwis who are doing things tough, such ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government backing mussel spat project
    The Coalition Government is investing in a project to boost survival rates of New Zealand mussels and grow the industry, Oceans and Fisheries Minister Shane Jones has announced. “This project seeks to increase the resilience of our mussels and significantly boost the sector’s productivity,” Mr Jones says. “The project - ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government focused on getting people into work
    Benefit figures released today underscore the importance of the Government’s plan to rebuild the economy and have 50,000 fewer people on Jobseeker Support, Social Development and Employment Minister Louise Upston says. “Benefit numbers are still significantly higher than when National was last in government, when there was about 70,000 fewer ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Clean energy key driver to reducing emissions
    The Government’s commitment to doubling New Zealand’s renewable energy capacity is backed by new data showing that clean energy has helped the country reach its lowest annual gross emissions since 1999, Climate Change Minister Simon Watts says. New Zealand’s latest Greenhouse Gas Inventory (1990-2022) published today, shows gross emissions fell ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Earthquake-prone buildings review brought forward
    The Government is bringing the earthquake-prone building review forward, with work to start immediately, and extending the deadline for remediations by four years, Building and Construction Minister Chris Penk says. “Our Government is focused on rebuilding the economy. A key part of our plan is to cut red tape that ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Thailand and NZ to agree to Strategic Partnership
    Prime Minister Christopher Luxon and his Thai counterpart, Prime Minister Srettha Thavisin, have today agreed that New Zealand and the Kingdom of Thailand will upgrade the bilateral relationship to a Strategic Partnership by 2026. “New Zealand and Thailand have a lot to offer each other. We have a strong mutual desire to build ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government consults on extending coastal permits for ports
    RMA Reform Minister Chris Bishop and Transport Minister Simeon Brown have today announced the Coalition Government’s intention to extend port coastal permits for a further 20 years, providing port operators with certainty to continue their operations. “The introduction of the Resource Management Act in 1991 required ports to obtain coastal ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Inflation coming down, but more work to do
    Today’s announcement that inflation is down to 4 per cent is encouraging news for Kiwis, but there is more work to be done - underlining the importance of the Government’s plan to get the economy back on track, acting Finance Minister Chris Bishop says. “Inflation is now at 4 per ...
    BeehiveBy beehive.govt.nz
    1 week ago

Page generated in The Standard by Wordpress at 2024-04-27T10:11:47+00:00