How this site’s webservers now work

Written By: - Date published: 4:52 pm, April 18th, 2014 - 8 comments
Categories: admin, The Standard - Tags:

Dolphin Over World 2010This site gets rather a lot of comments. Every time a comment is made the page it is on has to be regenerated, which limits the amount of static caching that can be done. So on a dynamic site like this one we require rather a lot of raw processing power to generate dynamically pages that keep changing all of the day.

There are a number of solutions for this issue from offloading the comments to an external supplier like Disqus (with the consequent issues about privacy and control) or to the excessive caching of pages when new comments come in (causing out of date pages) or to separating the comments into a dynamically loaded system (requiring extra client side javascript).

But we like to make the site as simple as possible, as up to date as possible and as secure as possible. So when comments arrive, this site updates the site.

Because of the current design of the pages (something that is always under review), this means that whenever a comment is posted, all pages with the list of recent comments is invalid. Because that is on every page to assist with navigation, the next time a page is read by anyone it is recalculated and laid out. While the server will cache that page, it will do so for a particular version of the browser that it was destined for. That is because not all pages are served up the same. It depends on what type of browser asked for it. So a page may be regenerated many times.

That requires a significiant amount of processing power when there are a lot of people commenting at the same time as there are lot of people reading the site. Outside of the middle of the night, those demand levels tend to quite unpredictable.

It’d be pretty easy to simply get bigger servers that could handle all of the possible demand. But how big is enough? We have had times when a story has been breaking where the load on the server has been for short periods of time been more than 10 times the usual load. We’ve had times when those peaks have killed the site running with a powerful 8 core system. During an election year we usually get a lot more traffic than we previously have had.

So since late last year we have been running the web facing parts of this site on cloud servers and setting up the required infrastructure to allow us to run multiple servers with common file, database, and page/query cache services.

This years we’ve been steadily getting a auto-scaling system largely triggered by CPU levels working. For a single webserver we use something that is rather like a dual core CPU laptop. Not massively fast, but capable of handling a moderate amount of traffic simultaneously. This is our quanta.

When the average processing usage for the existing web servers rises about a set value for more than a few minutes, then a new server of that type is automatically hired from Amazon, equipped with our standard disk image, and thrown on to the load balancer. If the demand keeps rising, then so do the number of servers in usage.

Conversely when the average processing usage drops below a certain value for a long period, then a server is dropped. This keeps happening until the servers are working productively. They go off to do some other type of task for someone else.

This kind of on-demand system is damn hard on people because they really don’t like chopping and changing all of the time as it is intensely unproductive to keep retraining people for different tasks. But computers are machines. You give them a new filesystem and they are trained. The hard bit is creatively figuring out what should be on that filesystem – a task that humans are usually pretty good at.

Anyway, I thought I’d share what our webservers have been doing over the last couple of weeks. Bear in mind that the times on this graph are in UTC (Universal time or Greenwich time) so they are currently about 12 hours off NZ time.

Web server loadings

You can see the strong diurnal cycle (click through the image for a higher resolution one if you cannot). The servers drop down to two or even one overnight. You can also see the weekends and public holidays where the servers during the day run a three and sometimes four. During the week days we’re usually running at four and sometimes five servers.

Each server costs about 2.8c per hour to have (with variable costs like bandwidth as extra), so while it doesn’t cost much, it adds up over a month to save the overnight costs. With a voluntary site like this all money saved is pretty useful.

But you can also see peaks for short bursts where we are running at six, seven or even eight servers when a story goes active and everyone piles into the site. And this fortnight has been a quiet period news wise.

I’ve currently capped the maximum numbers of web servers at 35 based on the potential loads on the database and file servers. Hopefully that will be enough to cope with the upcoming demand. Besides, I can always start autoscaling those if I really have to…

8 comments on “How this site’s webservers now work ”

  1. andy (the other one) 1

    Sorry, some times I hit F5 too much. mostly when felix or Sanc are trolling Syrlands or Pete G.

    Its my online crack habit…

    Its irresistable.

    • lprent 1.1

      I don’t care. It is literally what the system was designed for.

      BUT don’t do it more often than 15 times in a minute. If you do then you will find the site abruptly slows down on you for the next two hours because Wordfence now treats you as a pain-in-the-arse. I find that this seems to induce the change in behaviour that I’m after.

      • George 1.1.1

        Great work Lynn. And thank goodness I don’t have to use Disqus – which simultaneously destroys the small measure of privacy I use and slows comments to a crawl.

  2. tjp 2

    So if I read the above correctly it sounds as if the majority of your problems in terms of scalability are due to the frequent cache invalidations caused by new comments? I would be interested in hearing more about why you don’t want to load them dynamically – the additional client coding sounds like a relatively minor tradeoff. It wouldn’t solve the problem of caching multiple versions of the page for different clients but you would be able to hold them in cache longer.

    • lprent 2.1

      Oh I agree that it’d appear to make the site simpler to run. However the coding/testing complexity of doing it is a pain when you look at the detail. And it wouldn’t make much difference in terms of actual operations because of the site layout. We’d have to do a lot more coding to make it work.

      Depends what we’re talking about of course….

      But if I dynamically loaded the comments on the right tab then I’d get benefits for the pages that weren’t getting comments added to them and weren’t the front page.

      However at any given time between a quarter and a half of our page views are the front page. A prominent feature on the front page is the comment count. Every comment causes that page to regenerate. Of course that count could also become dynamic – but it has just doubled the dynamic code to test and maintain.

      At any given time between a quarter and a half of our page views are usually on a single post. This is usually because that particular post is getting a lot of comments, a lot of reads by people commenting, and a lot of reads by the substantial lurker population who read comments closely.

      During the day, about a fifth of the page views are from search engines plowing their way through the nearly 14k posts and 700k comments. We help them as much as we can by telling them where the site is active. But they are a very important source of new readers to the site.

      This means that most of the pages being read through the day and evening are on the most dynamic pages anyway. Usually inside a full day something like 75% of the page views will fall in on less than 10 pages. These will also be the pages that have the most comments being made (or in the case of the front page, comments being reported)

      To be effective, we’d have to do the same code changes amongst ALL of the browsers that access this site. As well as the browsers for the desktop versions of windows, mac and linux, it also includes the ever increasing numbers of variants being used on the mobiles (14% of page views). Just testing those on every one of the frequent code updates for wordpress and plugins beggars belief. The only real way to do it would be to write automated testers – which would be a hell of a coding chore.

      And of course wordpress directly has no support for dynamic comments, which means that we’d be on our own for testing updates across all of those targets platforms. The testing would have to be done with every update on wordpress.

      There are a number of possible alternatives system that do get their own testing done (Disqus for instance).

      But this is a site that gets quite a lot of comments, many of them large and just as interesting as the posts. There is a considerable SEO benefit in optimizing for them – which we have done. So we’d also have to do something about how the search engines access the comments. For instance I ran your first sentence through google and got an immediate hit. If you do the same on a disqus site like whaleoil – the same will not happen. As far as I can tell (a notoriously difficult thing to measure) about a third of the retained new readers come in after querying something that got a strike on a comment.

      There are a whole lot of other issues as well (like the ever shifting jQuery/Ajax systems). Suffice it to say that doing it with servers appeared to be a lot simpler than doing it with code because the returns off doing it with code are a lot less than you’d expect, and the cost of the code is whole lot higher and less transportable.

      Roll on HTML5 which will make some of this easier to do inside wordpress itself.

  3. lprent 3

    Interesting looking at the Shane Jones effect today on servers

    Click image for a larger view. The times are in UTC, so they are 12 hours out. So the Shane Jones story went up just after 06:00

    The load peaked at approximately 3.5x the usual human average maximum weekday middle of the day load at about 7pm and started falling back to under twice the usual load at about 9ish, then spiked again with the evening news. Then dropped rapidly after that.

    No page errors reported from the client to load balancer, but the latency was somewhat spiky as web servers were being added.

    I need a bigger story to really test the spike loading on the site. It didn’t really touch the database or file server. Comments kept pouring in without any problem apart from some moronic trolls unaware of the first comment policy…

    I’m happy with the performance. Especially for a peak loading at about 17-18c per hour…

    • lprent 3.1

      The traffic is certainly heating up.

      The direct traffic out last month for webpages (excluding the static images, javascript, and CSS) was 309GB (short month feb was 319.996 GB). So far this month as at the 22nd, we’re at 338GB. I guess election year is underway.

      Good to see that the changes I made at the end of last month are reducing all of the other costs down.

Links to post

Recent Comments

Recent Posts

  • EV road user charges bill passes
    Transport Minister Simeon Brown has welcomed the passing of legislation to move light electric vehicles (EVs) and plug-in hybrid electric vehicles (PHEVs) into the road user charges system from 1 April.  “It was always intended that EVs and PHEVs would be exempt from road user charges until they reached two ...
    BeehiveBy beehive.govt.nz
    4 hours ago
  • Bill targets illegal, unregulated fishing in international waters
    New Zealand is strengthening its ability to combat illegal fishing outside its domestic waters and beef up regulation for its own commercial fishers in international waters through a Bill which had its first reading in Parliament today. The Fisheries (International Fishing and Other Matters) Amendment Bill 2023 sets out stronger ...
    BeehiveBy beehive.govt.nz
    5 hours ago
  • Reserve Bank appointments
    Economists Carl Hansen and Professor Prasanna Gai have been appointed to the Reserve Bank Monetary Policy Committee, Finance Minister Nicola Willis announced today. The Monetary Policy Committee (MPC) is the independent decision-making body that sets the Official Cash Rate which determines interest rates.  Carl Hansen, the executive director of Capital ...
    BeehiveBy beehive.govt.nz
    6 hours ago
  • Stronger protections for apartment owners
    Apartment owners and buyers will soon have greater protections as further changes to the law on unit titles come into effect, Housing Minister Chris Bishop says. “The Unit Titles (Strengthening Body Corporate Governance and Other Matters) Amendment Act had already introduced some changes in December 2022 and May 2023, and ...
    BeehiveBy beehive.govt.nz
    8 hours ago
  • Travel focused on traditional partners and Middle East
    Foreign Minister Winston Peters will travel to Egypt and Europe from this weekend.    “This travel will focus on a range of New Zealand’s traditional diplomatic and security partnerships while enabling broad engagement on the urgent situation in Gaza,” Mr Peters says.   Mr Peters will attend the NATO Foreign ...
    BeehiveBy beehive.govt.nz
    8 hours ago
  • Keep safe on our roads this Easter
    Transport Minister Simeon Brown is encouraging all road users to stay safe, plan their journeys ahead of time, and be patient with other drivers while travelling around this Easter long weekend. “Road safety is a responsibility we all share, and with increased traffic on our roads expected this Easter we ...
    BeehiveBy beehive.govt.nz
    11 hours ago
  • Cost of living support for over 1.4 million Kiwis
    About 1.4 million New Zealanders will receive cost of living relief through increased government assistance from April 1 909,000 pensioners get a boost to Superannuation, including 5000 veterans 371,000 working-age beneficiaries will get higher payments 45,000 students will see an increase in their allowance Over a quarter of New Zealanders ...
    BeehiveBy beehive.govt.nz
    11 hours ago
  • Tenancy reviews for social housing restart
    Ensuring social housing is being provided to those with the greatest needs is front of mind as the Government restarts social housing tenancy reviews, Associate Housing Minister Tama Potaka says. “Our relentless focus on building a strong economy is to ensure we can deliver better public services such as social ...
    BeehiveBy beehive.govt.nz
    12 hours ago
  • Kermadec Ocean Sanctuary plan halted
    The Kermadec Ocean Sanctuary will not go ahead, with Cabinet deciding to stop work on the proposed reserve and remove the Bill that would have established it from Parliament’s order paper. “The Kermadec Ocean Sanctuary Bill would have created a 620,000 sq km economic no-go zone,” Oceans and Fisheries Minister ...
    BeehiveBy beehive.govt.nz
    12 hours ago
  • Cutting all that dam red tape
    Dam safety regulations are being amended so that smaller dams won’t be subject to excessive compliance costs, Minister for Building and Construction Chris Penk says. “The coalition Government is focused on reducing costs and removing unnecessary red tape so we can get the economy back on track.  “Dam safety regulations ...
    BeehiveBy beehive.govt.nz
    13 hours ago
  • Drought support extended to parts of North Island
    The coalition Government is expanding the medium-scale adverse event classification to parts of the North Island as dry weather conditions persist, Agriculture Minister Todd McClay announced today. “I have made the decision to expand the medium-scale adverse event classification already in place for parts of the South Island to also cover the ...
    BeehiveBy beehive.govt.nz
    14 hours ago
  • Passage of major tax bill welcomed
    The passing of legislation giving effect to coalition Government tax commitments has been welcomed by Finance Minister Nicola Willis.  “The Taxation (Annual Rates for 2023–24, Multinational Tax, and Remedial Matters) Bill will help place New Zealand on a more secure economic footing, improve outcomes for New Zealanders, and make our tax system ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Lifting economy through science, tertiary sectors
    Science, Innovation and Technology Minister Judith Collins and Tertiary Education and Skills Minister Penny Simmonds today announced plans to transform our science and university sectors to boost the economy. Two advisory groups, chaired by Professor Sir Peter Gluckman, will advise the Government on how these sectors can play a greater ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Government announces Budget priorities
    The Budget will deliver urgently-needed tax relief to hard-working New Zealanders while putting the government’s finances back on a sustainable track, Finance Minister Nicola Willis says.  The Finance Minister made the comments at the release of the Budget Policy Statement setting out the Government’s Budget objectives. “The coalition Government intends ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Government to consider accommodation solution
    The coalition Government will look at options to address a zoning issue that limits how much financial support Queenstown residents can get for accommodation. Cabinet has agreed on a response to the Petitions Committee, which had recommended the geographic information MSD uses to determine how much accommodation supplement can be ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Government approves extension to Royal Commission of Inquiry into Abuse in Care
    Cabinet has agreed to a short extension to the final reporting timeframe for the Royal Commission into Abuse in Care from 28 March 2024 to 26 June 2024, Internal Affairs Minister Brooke van Velden says.                                         “The Royal Commission wrote to me on 16 February 2024, requesting that I consider an ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • $18m boost for Kiwis travelling to health treatment
    The coalition Government is delivering an $18 million boost to New Zealanders needing to travel for specialist health treatment, Health Minister Dr Shane Reti says.   “These changes are long overdue – the National Travel Assistance (NTA) scheme saw its last increase to mileage and accommodation rates way back in 2009.  ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • PM’s Prizes for Space to showcase sector’s talent
    The Government is recognising the innovative and rising talent in New Zealand’s growing space sector, with the Prime Minister and Space Minister Judith Collins announcing the new Prime Minister’s Prizes for Space today. “New Zealand has a growing reputation as a high-value partner for space missions and research. I am ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Concerns conveyed to China over cyber activity
    Foreign Minister Winston Peters has confirmed New Zealand’s concerns about cyber activity have been conveyed directly to the Chinese Government.     “The Prime Minister and Minister Collins have expressed concerns today about malicious cyber activity, attributed to groups sponsored by the Chinese Government, targeting democratic institutions in both New ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Independent Reviewers appointed for School Property Inquiry
    Independent Reviewers appointed for School Property Inquiry Education Minister Erica Stanford today announced the appointment of three independent reviewers to lead the Ministerial Inquiry into the Ministry of Education’s School Property Function.  The Inquiry will be led by former Minister of Foreign Affairs Murray McCully. “There is a clear need ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Brynderwyns open for Easter
    State Highway 1 across the Brynderwyns will be open for Easter weekend, with work currently underway to ensure the resilience of this critical route being paused for Easter Weekend to allow holiday makers to travel north, Transport Minister Simeon Brown says. “Today I visited the Brynderwyn Hills construction site, where ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Speech to the Infrastructure Funding & Financing Conference
    Introduction Good morning to you all, and thanks for having me bright and early today. I am absolutely delighted to be the Minister for Infrastructure alongside the Minister of Housing and Resource Management Reform. I know the Prime Minister sees the three roles as closely connected and he wants me ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Parliamentary network breached by the PRC
    New Zealand stands with the United Kingdom in its condemnation of People’s Republic of China (PRC) state-backed malicious cyber activity impacting its Electoral Commission and targeting Members of the UK Parliament. “The use of cyber-enabled espionage operations to interfere with democratic institutions and processes anywhere is unacceptable,” Minister Responsible for ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • NZ to provide support for Solomon Islands election
    Foreign Minister Winston Peters and Defence Minister Judith Collins today announced New Zealand will provide logistics support for the upcoming Solomon Islands election. “We’re sending a team of New Zealand Defence Force personnel and two NH90 helicopters to provide logistics support for the election on 17 April, at the request ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • NZ-EU FTA gains Royal Assent for 1 May entry to force
    The European Union Free Trade Agreement Legislation Amendment Bill received Royal Assent today, completing the process for New Zealand’s ratification of its free trade agreement with the European Union.    “I am pleased to announce that today, in a small ceremony at the Beehive, New Zealand notified the European Union ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • COVID-19 inquiry attracts 11,000 submissions
    Public consultation on the terms of reference for the Royal Commission into COVID-19 Lessons has concluded, Internal Affairs Minister Hon Brooke van Velden says.  “I have been advised that there were over 11,000 submissions made through the Royal Commission’s online consultation portal.” Expanding the scope of the Royal Commission of ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Families to receive up to $75 a week help with ECE fees
    Hardworking families are set to benefit from a new credit to help them meet their early childcare education (ECE) costs, Finance Minister Nicola Willis says. From 1 July, parents and caregivers of young children will be supported to manage the rising cost of living with a partial reimbursement of their ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Unlocking a sustainable, low-emissions future
    A specialised Independent Technical Advisory Group (ITAG) tasked with preparing and publishing independent non-binding advice on the design of a "green" (sustainable finance) taxonomy rulebook is being established, Climate Change Minister Simon Watts says.  “Comprising experts and market participants, the ITAG's primary goal is to deliver comprehensive recommendations to the ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Chief of Army thanked for his service
    Defence Minister Judith Collins has thanked the Chief of Army, Major General John Boswell, DSD, for his service as he leaves the Army after 40 years. “I would like to thank Major General Boswell for his contribution to the Army and the wider New Zealand Defence Force, undertaking many different ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Minister to meet Australian counterparts and Manufacturing Industry Leaders
    25 March 2024 Minister to meet Australian counterparts and Manufacturing Industry Leaders Small Business, Manufacturing, Commerce and Consumer Affairs Minister Andrew Bayly will travel to Australia for a series of bi-lateral meetings and manufacturing visits. During the visit, Minister Bayly will meet with his Australian counterparts, Senator Tim Ayres, Ed ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Government commits nearly $3 million for period products in schools
    Government commits almost $3 million for period products in schools The Coalition Government has committed $2.9 million to ensure intermediate and secondary schools continue providing period products to those who need them, Minister of Education Erica Stanford announced today. “This is an issue of dignity and ensuring young women don’t ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Speech – Making it easier to build.
    Good morning, it’s great to be here.   First, I would like to acknowledge the New Zealand Institute of Building Surveyors and thank you for the opportunity to be here this morning.  I would like to use this opportunity to outline the Government’s ambitious plan and what we hope to ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Pacific youth to shine from boost to Polyfest
    Minister for Pacific Peoples Dr Shane Reti has announced the Government’s commitment to the Auckland Secondary Schools Māori and Pacific Islands Cultural Festival, more commonly known as Polyfest. “The Ministry for Pacific Peoples is a longtime supporter of Polyfest and, as it celebrates 49 years in 2024, I’m proud to ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • 2024 Ngarimu VC and 28th (Māori) Battalion Memorial Scholarships announced
    ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Speech to Breast Cancer Foundation – Insights Conference
    Before moving onto the substance of today’s address, I want to recognise the very significant and ongoing contribution the Breast Cancer Foundation makes to support the lives of New Zealand women and their families living with breast cancer. I very much enjoy working with you. I also want to recognise ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Kiwi research soars to International Space Station
    New Zealand has notched up a first with the launch of University of Canterbury research to the International Space Station, Science, Innovation and Technology and Space Minister Judith Collins says. The hardware, developed by Dr Sarah Kessans, is designed to operate autonomously in orbit, allowing scientists on Earth to study ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Speech to the New Zealand Planning Institute
    Introduction Thank you for inviting me to speak with you today and I’m sorry I can’t be there in person. Yesterday I started in Wellington for Breakfast TV, spoke to a property conference in Auckland, and finished the day speaking to local government in Christchurch, so it would have been ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Support for Northland emergency response centre
    The Coalition Government is contributing more than $1 million to support the establishment of an emergency multi-agency coordination centre in Northland. Emergency Management and Recovery Minister Mark Mitchell announced the contribution today during a visit of the Whangārei site where the facility will be constructed.  “Northland has faced a number ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Celebrating 20 years of Whakaata Māori
    New Zealanders have enjoyed a broader range of voices telling the story of Aotearoa thanks to the creation of Whakaata Māori 20 years ago, says Māori Development Minister Tama Potaka. The minister spoke at a celebration marking the national indigenous media organisation’s 20th anniversary at their studio in Auckland on ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Some commercial fishery catch limits increased
    Commercial catch limits for some fisheries have been increased following a review showing stocks are healthy and abundant, Ocean and Fisheries Minister Shane Jones says. The changes, along with some other catch limit changes and management settings, begin coming into effect from 1 April 2024. "Regular biannual reviews of fish ...
    BeehiveBy beehive.govt.nz
    6 days ago

Page generated in The Standard by Wordpress at 2024-03-28T08:38:01+00:00