How this site’s webservers now work

Written By: - Date published: 4:52 pm, April 18th, 2014 - 8 comments
Categories: admin, The Standard - Tags:

Dolphin Over World 2010This site gets rather a lot of comments. Every time a comment is made the page it is on has to be regenerated, which limits the amount of static caching that can be done. So on a dynamic site like this one we require rather a lot of raw processing power to generate dynamically pages that keep changing all of the day.

There are a number of solutions for this issue from offloading the comments to an external supplier like Disqus (with the consequent issues about privacy and control) or to the excessive caching of pages when new comments come in (causing out of date pages) or to separating the comments into a dynamically loaded system (requiring extra client side javascript).

But we like to make the site as simple as possible, as up to date as possible and as secure as possible. So when comments arrive, this site updates the site.

Because of the current design of the pages (something that is always under review), this means that whenever a comment is posted, all pages with the list of recent comments is invalid. Because that is on every page to assist with navigation, the next time a page is read by anyone it is recalculated and laid out. While the server will cache that page, it will do so for a particular version of the browser that it was destined for. That is because not all pages are served up the same. It depends on what type of browser asked for it. So a page may be regenerated many times.

That requires a significiant amount of processing power when there are a lot of people commenting at the same time as there are lot of people reading the site. Outside of the middle of the night, those demand levels tend to quite unpredictable.

It’d be pretty easy to simply get bigger servers that could handle all of the possible demand. But how big is enough? We have had times when a story has been breaking where the load on the server has been for short periods of time been more than 10 times the usual load. We’ve had times when those peaks have killed the site running with a powerful 8 core system. During an election year we usually get a lot more traffic than we previously have had.

So since late last year we have been running the web facing parts of this site on cloud servers and setting up the required infrastructure to allow us to run multiple servers with common file, database, and page/query cache services.

This years we’ve been steadily getting a auto-scaling system largely triggered by CPU levels working. For a single webserver we use something that is rather like a dual core CPU laptop. Not massively fast, but capable of handling a moderate amount of traffic simultaneously. This is our quanta.

When the average processing usage for the existing web servers rises about a set value for more than a few minutes, then a new server of that type is automatically hired from Amazon, equipped with our standard disk image, and thrown on to the load balancer. If the demand keeps rising, then so do the number of servers in usage.

Conversely when the average processing usage drops below a certain value for a long period, then a server is dropped. This keeps happening until the servers are working productively. They go off to do some other type of task for someone else.

This kind of on-demand system is damn hard on people because they really don’t like chopping and changing all of the time as it is intensely unproductive to keep retraining people for different tasks. But computers are machines. You give them a new filesystem and they are trained. The hard bit is creatively figuring out what should be on that filesystem – a task that humans are usually pretty good at.

Anyway, I thought I’d share what our webservers have been doing over the last couple of weeks. Bear in mind that the times on this graph are in UTC (Universal time or Greenwich time) so they are currently about 12 hours off NZ time.

Web server loadings

You can see the strong diurnal cycle (click through the image for a higher resolution one if you cannot). The servers drop down to two or even one overnight. You can also see the weekends and public holidays where the servers during the day run a three and sometimes four. During the week days we’re usually running at four and sometimes five servers.

Each server costs about 2.8c per hour to have (with variable costs like bandwidth as extra), so while it doesn’t cost much, it adds up over a month to save the overnight costs. With a voluntary site like this all money saved is pretty useful.

But you can also see peaks for short bursts where we are running at six, seven or even eight servers when a story goes active and everyone piles into the site. And this fortnight has been a quiet period news wise.

I’ve currently capped the maximum numbers of web servers at 35 based on the potential loads on the database and file servers. Hopefully that will be enough to cope with the upcoming demand. Besides, I can always start autoscaling those if I really have to…

8 comments on “How this site’s webservers now work ”

  1. andy (the other one) 1

    Sorry, some times I hit F5 too much. mostly when felix or Sanc are trolling Syrlands or Pete G.

    Its my online crack habit…

    Its irresistable.

    • lprent 1.1

      I don’t care. It is literally what the system was designed for.

      BUT don’t do it more often than 15 times in a minute. If you do then you will find the site abruptly slows down on you for the next two hours because Wordfence now treats you as a pain-in-the-arse. I find that this seems to induce the change in behaviour that I’m after.

      • George 1.1.1

        Great work Lynn. And thank goodness I don’t have to use Disqus – which simultaneously destroys the small measure of privacy I use and slows comments to a crawl.

  2. tjp 2

    So if I read the above correctly it sounds as if the majority of your problems in terms of scalability are due to the frequent cache invalidations caused by new comments? I would be interested in hearing more about why you don’t want to load them dynamically – the additional client coding sounds like a relatively minor tradeoff. It wouldn’t solve the problem of caching multiple versions of the page for different clients but you would be able to hold them in cache longer.

    • lprent 2.1

      Oh I agree that it’d appear to make the site simpler to run. However the coding/testing complexity of doing it is a pain when you look at the detail. And it wouldn’t make much difference in terms of actual operations because of the site layout. We’d have to do a lot more coding to make it work.

      Depends what we’re talking about of course….

      But if I dynamically loaded the comments on the right tab then I’d get benefits for the pages that weren’t getting comments added to them and weren’t the front page.

      However at any given time between a quarter and a half of our page views are the front page. A prominent feature on the front page is the comment count. Every comment causes that page to regenerate. Of course that count could also become dynamic – but it has just doubled the dynamic code to test and maintain.

      At any given time between a quarter and a half of our page views are usually on a single post. This is usually because that particular post is getting a lot of comments, a lot of reads by people commenting, and a lot of reads by the substantial lurker population who read comments closely.

      During the day, about a fifth of the page views are from search engines plowing their way through the nearly 14k posts and 700k comments. We help them as much as we can by telling them where the site is active. But they are a very important source of new readers to the site.

      This means that most of the pages being read through the day and evening are on the most dynamic pages anyway. Usually inside a full day something like 75% of the page views will fall in on less than 10 pages. These will also be the pages that have the most comments being made (or in the case of the front page, comments being reported)

      To be effective, we’d have to do the same code changes amongst ALL of the browsers that access this site. As well as the browsers for the desktop versions of windows, mac and linux, it also includes the ever increasing numbers of variants being used on the mobiles (14% of page views). Just testing those on every one of the frequent code updates for wordpress and plugins beggars belief. The only real way to do it would be to write automated testers – which would be a hell of a coding chore.

      And of course wordpress directly has no support for dynamic comments, which means that we’d be on our own for testing updates across all of those targets platforms. The testing would have to be done with every update on wordpress.

      There are a number of possible alternatives system that do get their own testing done (Disqus for instance).

      But this is a site that gets quite a lot of comments, many of them large and just as interesting as the posts. There is a considerable SEO benefit in optimizing for them – which we have done. So we’d also have to do something about how the search engines access the comments. For instance I ran your first sentence through google and got an immediate hit. If you do the same on a disqus site like whaleoil – the same will not happen. As far as I can tell (a notoriously difficult thing to measure) about a third of the retained new readers come in after querying something that got a strike on a comment.

      There are a whole lot of other issues as well (like the ever shifting jQuery/Ajax systems). Suffice it to say that doing it with servers appeared to be a lot simpler than doing it with code because the returns off doing it with code are a lot less than you’d expect, and the cost of the code is whole lot higher and less transportable.

      Roll on HTML5 which will make some of this easier to do inside wordpress itself.

  3. lprent 3

    Interesting looking at the Shane Jones effect today on servers

    Click image for a larger view. The times are in UTC, so they are 12 hours out. So the Shane Jones story went up just after 06:00

    The load peaked at approximately 3.5x the usual human average maximum weekday middle of the day load at about 7pm and started falling back to under twice the usual load at about 9ish, then spiked again with the evening news. Then dropped rapidly after that.

    No page errors reported from the client to load balancer, but the latency was somewhat spiky as web servers were being added.

    I need a bigger story to really test the spike loading on the site. It didn’t really touch the database or file server. Comments kept pouring in without any problem apart from some moronic trolls unaware of the first comment policy…

    I’m happy with the performance. Especially for a peak loading at about 17-18c per hour…

    • lprent 3.1

      The traffic is certainly heating up.

      The direct traffic out last month for webpages (excluding the static images, javascript, and CSS) was 309GB (short month feb was 319.996 GB). So far this month as at the 22nd, we’re at 338GB. I guess election year is underway.

      Good to see that the changes I made at the end of last month are reducing all of the other costs down.

Links to post

Recent Comments

Recent Posts

  • NZ – UAE trade agreement consultation begins
    Following a meeting with UAE Trade Minister Dr. Thani bin Ahmed Al Zeyoudi at the WTO Ministerial Conference in Abu Dhabi, Trade Minister Todd McClay has launched public consultation for a trade agreement between New Zealand and the United Arab Emirates (UAE).   “The UAE is a top-20 export market for ...
    BeehiveBy beehive.govt.nz
    10 hours ago
  • Minister thanks Public Service Commissioner
    Public Service Minister Nicola Willis has thanked retiring Public Service Commissioner Peter Hughes for his 43 years of service. Mr Hughes retires today, after serving eight years as Public Service Commissioner.  “Peter Hughes is an outstanding public servant who has served many governments, regardless of their political leaning, with professionalism and ...
    BeehiveBy beehive.govt.nz
    13 hours ago
  • Tourism data shows determination of sector
    New tourism data out today shows the continued importance of tourism to the New Zealand economy as tourism steps up to become our second-biggest export earner, Tourism Minister Matt Doocey says. “The Tourism Satellite Account shows how strongly tourism rebounded post-pandemic with total tourism expenditure in New Zealand of $37.7b ...
    BeehiveBy beehive.govt.nz
    14 hours ago
  • Housing Minister thanks outgoing Kāinga Ora Chair
    Housing Minister Chris Bishop has today thanked outgoing Kāinga Ora – Homes & Communities Chair Vui Mark Gosche for his many years of public service. “Mr Gosche tendered his resignation as Chair yesterday evening. He will remain a member of the Board until the end of March,” says Housing Minister ...
    BeehiveBy beehive.govt.nz
    15 hours ago
  • New sanctions package against Russia
    Foreign Minister Winston Peters today announced a new package of sanctions as part of the ongoing international sanction response to Russia’s illegal invasion of Ukraine in February 2022.   The new sanctions are:   Implementation of the G7-plus price cap on Russian-origin oil; making explicit the prohibition on exporting restricted ...
    BeehiveBy beehive.govt.nz
    15 hours ago
  • Travel bans on extremist Israeli settlers
    Prime Minister Christopher Luxon and Foreign Minister Winston Peters have announced travel bans on a number of extremist Israeli settlers who have committed violent attacks against Palestinians in the West Bank.   “New Zealand is seriously concerned by the significant increase in extremist violence perpetrated by Israeli settlers against Palestinian populations in recent months. This is ...
    BeehiveBy beehive.govt.nz
    15 hours ago
  • NZ designates entirety of Hamas as terrorist entity
    Prime Minister Christopher Luxon and Foreign Minister Winston Peters have announced today the designation of Hamas in its entirety as a terrorist entity.   “The terrorist attacks by Hamas in October 2023 were brutal and we have unequivocally condemned them,” Mr Luxon says.    Following these attacks, then Prime Minister Chris Hipkins commissioned advice from officials about designating the ...
    BeehiveBy beehive.govt.nz
    15 hours ago
  • Government announces independent review of forestry ETS costs
    Forestry Minister Todd McClay has today announced an independent review into the forestry component of the Emissions Trading Scheme (ETS) Register to ensure it is efficient and cost-effective. “Up and down the country forestry owners have been raising concerns about the excessive costs that have been imposed upon them by ...
    BeehiveBy beehive.govt.nz
    19 hours ago
  • Access barriers to PET-CT scans removed
    New Zealanders now have the same access to PET-CT scans no matter where they live, Health Minister Dr Shane Reti says. Health New Zealand - Te Whatu Ora has approved funding an updated national set of criteria that will allow for about 1,000 more PET-CT scans a year to be ...
    BeehiveBy beehive.govt.nz
    20 hours ago
  • Air New Zealand and Singapore Airlines’ alliance extended
    Associate Transport Minister Matt Doocey announced today that the Government has extended Air New Zealand and Singapore Airlines’ strategic alliance for another five years. “Reauthorising this strategic partnership means that passengers flying in and out of New Zealand will continue to have access to a wide range of flights and destinations,” ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Health system reforms need further action
    Health Minister Dr Shane Reti says the latest report into New Zealand’s health reforms shows a few benefits, but overall once again demonstrates a lack of leadership by the previous Labour government.  The Ministerial Advisory Committee (MAC) report released today was commissioned by the previous government to provide an independent ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Parallel assessment means new medicines assessed sooner
    Pharmac is changing its process so it can assess a funding application at the same time Medsafe is assessing the application for regulatory approval. This means that medicines will be able to be considered for funding sooner in New Zealand. “Access to medicines is a crucial part of many Kiwis’ ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Smokefree Amendment Bill Introduced
    The Government has today introduced an Amendment Bill that will repeal three parts of the previous Government’s planned changes to regulate smoked tobacco. “The Coalition Government is committed to the Smokefree 2025 goal, but we are taking a different regulatory approach to reducing smoking rates and the harm from smoking,” ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Targeted support for young people
    Recently allocated Ministry of Youth Development funding will support more than 6700 young people to receive targeted youth development support to remain in education or transition to further training or employment and improve their wellbeing, Youth Minister Matt Doocey says.  Funding of $10.69 million will be allocated to 34 community-based ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Reshaping the health system to bring Māori health closer to home
    Legislation that will disestablish the Māori Health Authority will be introduced in Parliament today, heralding the start of a new vision for Māori health says Minister of Health Dr Shane Reti.  “We have said we will bring healthcare for all New Zealanders closer to the home and closer to the ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Speech to the Wellington Chamber of Commerce
    Acknowledgements Good morning. Can I start by acknowledging Simon and the team at the Chamber. Thanks for the invitation to be here today. Introduction In October last year New Zealanders voted for change. The Coalition government was elected with a clear mandate to rebuild the economy and reduce the cost ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • NZ welcomes Australia and Brazil to agreements
    New Zealand has welcomed Australia to the Inclusive Trade Action Group (ITAG) and Australia and Brazil to the Global Trade and Gender Arrangement (GTAGA) Minister for Trade Todd McClay says.  As the current chair of ITAG and GTAGA, Minister McClay hosted the signing ceremony and issued the Abu Dhabi Joint ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Inquiry announced into school property
    The Government will conduct a Ministerial Inquiry to address problems with the school property system where the scope of property works planned was unrealistic and unaffordable. “The coalition Government has inherited a school property system bordering on crisis,” Education Minister Erica Stanford says. “There have been a number of cost escalations ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • New Chair for Guardians of NZ Superannuation
    Company director and investor John Williamson has been appointed as the new Chair of the Guardians of New Zealand Superannuation, the Crown entity that oversees the NZ Super Fund and the Elevate NZ Venture Capital Fund, Finance Minister Nicola Willis announced today.  Mr Williamson will take up his new position ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Northland open for business as critical works to repair SH1 Brynderwyn Hills begin
    The Government is encouraging New Zealanders to support, visit, and explore Northland, as the closure and detour of SH1 at the Bryderwyn Hills begins, and critical repair work by the NZ Transport Agency (NZTA) gets underway, Transport Minister Simeon Brown says. “Many regions across the country suffered extensive and devastating ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Government backs police to crackdown on gangs
    The coalition Government is restoring law and order by providing police new tools to crack down on criminal gangs, says Justice Minister Paul Goldsmith and Police Minister Mark Mitchell.  “Over the last five years gangs have recruited more than 3000 members, a 51 per cent increase. At the same time, ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Northland’s new Kāeo Bridge officially open
    Transport Minister Simeon Brown has welcomed the official opening of the new State Highway 10 (SH10) Kāeo Bridge, which will improve safety and traffic flow for people heading to and from the Far North. “This is an important piece of infrastructure for the Northland region that will help members of ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Government grants $6.6 million to clean up old landfill sites
    The Government has granted $6.6 million to clean up four historic New Zealand landfill and dump sites vulnerable to extreme weather events and coastal erosion. At the BlueGreens Forum in Paihia today Environment Minister Penny Simmonds said that the Contaminated Sites Remediation Fund grants will go towards fixing former landfills ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Dry weather triggers extra support for farmers and growers across the top of the South Island
    The coalition Government is providing support for farmers and growers as dry conditions worsen across the top of the South Island. “Conditions on the ground across the Marlborough, Tasman, and Nelson districts are now extremely dry and likely to get worse in the coming months,” Agriculture Minister Todd McClay said. ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Trade Minister heads to Abu Dhabi for key WTO negotiations
    Trade Minister Todd McClay travels to Abu Dhabi in the United Arab Emirates for the 13th Ministerial Conference of the World Trade Organisation (WTO) today, to take up his role as Vice Chair of the negotiations. The Ministerial Conference is the highest decision-making body within the WTO and meets every ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Appointment round for King’s Counsel announced
    Attorney-General Judith Collins today announced an appointment round for King’s Counsel will take place in 2024. Appointments of King’s Counsel are made by the Governor-General on the recommendation of the Attorney-General and with the concurrence of the Chief Justice. The Governor-General retains the discretion to appoint King’s Counsel in recognition ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Retiring Chief of Navy thanked for his service
    Defence Minister Judith Collins has thanked the Chief of Navy, Rear Admiral David Proctor, for his service as he retires from the Royal New Zealand Navy after 37 years. Rear Admiral Proctor will retire on 16 May to take up an employment opportunity in Australia.  “I would like to thank ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Indonesian Vice President to visit New Zealand
    Indonesia’s Vice President Ma’ruf Amin will visit New Zealand next week, the first here by an Indonesian leader since 2018, Deputy Prime Minister Winston Peters has announced. “New Zealand and Indonesia have a strong partnership,” Mr Peters says.  “The Vice President’s visit is an opportunity to discuss how we can strengthen ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Government boost to fight against caulerpa
    The battle to contain the fast-spreading exotic caulerpa seaweed has today received a $5 million boost to accelerate the development of removal techniques, says Biosecurity Minister Andrew Hoggard.  “The time is now to really lean in and build on the work of Biosecurity New Zealand, mana whenua, communities and local ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Minister attending Australian data, digital meeting
    Minister for Digitising Government Judith Collins is in Sydney to attend the first Data and Digital Ministers’ Meeting of 2024.  “This is a great opportunity to connect with our Australian counterparts and identify how we can work together on digital transformation,” Ms Collins says.   “Both our nations are looking into ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Appointments to Antarctica New Zealand Board
    Foreign Minister Winston Peters has appointed Leon Grice and Heather Simpson to serve on the Antarctica New Zealand board.  “Since taking office, the Coalition Government has become concerned about the direction of the Scott Base Redevelopment Project,” Mr Peters says.  “It is vital that Antarctica New Zealand has the right ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Strengthening the Single Economic Market
    Finance Minister Nicola Willis has met with Australian Treasurer Jim Chalmers to discuss the opportunities to lower business costs and increase the ease with which businesses and people can operate across the Tasman.     “I have met with Treasurer Chalmers and shared our new Government’s ambitious economic goals, our plans ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government to address business payment practices
    The Government will repeal the Business Payment Practices Act 2023, Small Business and Manufacturing Minister Andrew Bayly announced today. “There is a major problem with large market players imposing long payment terms and routinely paying invoices late. “However, the Business Payment Practices Act is not an effective solution and would ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Greater focus on work will reduce child poverty
    Worsening child poverty rates support the Coalition Government’s focus on reducing the cost of living and getting people into work, Child Poverty Reduction Minister Louise Upston says. Figures released by Stats NZ today show child poverty rates have increased, with the rising cost of living, driven by inflation, making it ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • NZ announces new support for Ukraine
    Foreign Minister Winston Peters and Defence Minister Judith Collins have marked two years since Russia’s invasion of Ukraine by announcing further support and sanctions, and extending our military assistance. “Russia launched its illegal, full-scale invasion of Ukraine, in blatant violation of international law, including the UN Charter,” Mr Peters says. ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Finalists of Ahuwhenua Trophy announced
    Māori Development Minister Tama Potaka announced the two finalists for this year’s Ahuwhenua Trophy at Parliament yesterday.  “I am pleased to see such a high calibre of Māori dairy farms featured as finalists this year,” Mr Potaka says. The finalists for 2024 are: Wairarapa Moana ki Pouakani Whakatōhea Māori Trust ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Finance Minister to meet Australian Treasurer
    Finance Minister Nicola Willis will travel to Australia today to meet her Australian counterpart, Treasurer Jim Chalmers.    “New Zealand and Australia have an incredibly strong trade and investment relationship. The Closer Economic Relations and Single Economic Market are powerful engines for growth on both sides of the Tasman.     “I will ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • PM shocked and saddened at death of Efeso Collins
    “I am truly shocked and saddened at the news of Efeso Collins’ sudden death,” Prime Minister Christopher Luxon says. “Efeso was a good man, always friendly and kind, and a true champion and advocate for his Samoan and South Auckland communities. “Our thoughts and deepest sympathies go to his family, ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Greater support for social workers
    The Coalition Government is enhancing the professionalism of the social work sector and supporting the vulnerable people who rely on them, Social Development and Employment Minister Louise Upston says.  The Social Workers Registration Legislation Amendment Bill passed its third reading in Parliament today. It amends the Social Workers Registration Legislation ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government delivers greater freedom and choice for sick New Zealanders
    The coalition government is delivering on its commitment to making principled decisions by getting rid of red tape that doesn’t make sense and allowing sick New Zealanders greater freedom and choice to purchase effective cold and flu medicines. A bill amending the Misuse of Drugs Act 1975 is being introduced, and changes to the Medicines ...
    BeehiveBy beehive.govt.nz
    1 week ago

Page generated in The Standard by Wordpress at 2024-02-29T16:22:35+00:00