web analytics

How this site’s webservers now work

Written By: - Date published: 4:52 pm, April 18th, 2014 - 7 comments
Categories: admin, The Standard - Tags:

Dolphin Over World 2010This site gets rather a lot of comments. Every time a comment is made the page it is on has to be regenerated, which limits the amount of static caching that can be done. So on a dynamic site like this one we require rather a lot of raw processing power to generate dynamically pages that keep changing all of the day.

There are a number of solutions for this issue from offloading the comments to an external supplier like Disqus (with the consequent issues about privacy and control) or to the excessive caching of pages when new comments come in (causing out of date pages) or to separating the comments into a dynamically loaded system (requiring extra client side javascript).

But we like to make the site as simple as possible, as up to date as possible and as secure as possible. So when comments arrive, this site updates the site.

Because of the current design of the pages (something that is always under review), this means that whenever a comment is posted, all pages with the list of recent comments is invalid. Because that is on every page to assist with navigation, the next time a page is read by anyone it is recalculated and laid out. While the server will cache that page, it will do so for a particular version of the browser that it was destined for. That is because not all pages are served up the same. It depends on what type of browser asked for it. So a page may be regenerated many times.

That requires a significiant amount of processing power when there are a lot of people commenting at the same time as there are lot of people reading the site. Outside of the middle of the night, those demand levels tend to quite unpredictable.

It’d be pretty easy to simply get bigger servers that could handle all of the possible demand. But how big is enough? We have had times when a story has been breaking where the load on the server has been for short periods of time been more than 10 times the usual load. We’ve had times when those peaks have killed the site running with a powerful 8 core system. During an election year we usually get a lot more traffic than we previously have had.

So since late last year we have been running the web facing parts of this site on cloud servers and setting up the required infrastructure to allow us to run multiple servers with common file, database, and page/query cache services.

This years we’ve been steadily getting a auto-scaling system largely triggered by CPU levels working. For a single webserver we use something that is rather like a dual core CPU laptop. Not massively fast, but capable of handling a moderate amount of traffic simultaneously. This is our quanta.

When the average processing usage for the existing web servers rises about a set value for more than a few minutes, then a new server of that type is automatically hired from Amazon, equipped with our standard disk image, and thrown on to the load balancer. If the demand keeps rising, then so do the number of servers in usage.

Conversely when the average processing usage drops below a certain value for a long period, then a server is dropped. This keeps happening until the servers are working productively. They go off to do some other type of task for someone else.

This kind of on-demand system is damn hard on people because they really don’t like chopping and changing all of the time as it is intensely unproductive to keep retraining people for different tasks. But computers are machines. You give them a new filesystem and they are trained. The hard bit is creatively figuring out what should be on that filesystem – a task that humans are usually pretty good at.

Anyway, I thought I’d share what our webservers have been doing over the last couple of weeks. Bear in mind that the times on this graph are in UTC (Universal time or Greenwich time) so they are currently about 12 hours off NZ time.

Web server loadings

You can see the strong diurnal cycle (click through the image for a higher resolution one if you cannot). The servers drop down to two or even one overnight. You can also see the weekends and public holidays where the servers during the day run a three and sometimes four. During the week days we’re usually running at four and sometimes five servers.

Each server costs about 2.8c per hour to have (with variable costs like bandwidth as extra), so while it doesn’t cost much, it adds up over a month to save the overnight costs. With a voluntary site like this all money saved is pretty useful.

But you can also see peaks for short bursts where we are running at six, seven or even eight servers when a story goes active and everyone piles into the site. And this fortnight has been a quiet period news wise.

I’ve currently capped the maximum numbers of web servers at 35 based on the potential loads on the database and file servers. Hopefully that will be enough to cope with the upcoming demand. Besides, I can always start autoscaling those if I really have to…

7 comments on “How this site’s webservers now work”

  1. andy (the other one) 1

    Sorry, some times I hit F5 too much. mostly when felix or Sanc are trolling Syrlands or Pete G.

    Its my online crack habit…

    Its irresistable.

    • lprent 1.1

      I don’t care. It is literally what the system was designed for.

      BUT don’t do it more often than 15 times in a minute. If you do then you will find the site abruptly slows down on you for the next two hours because Wordfence now treats you as a pain-in-the-arse. I find that this seems to induce the change in behaviour that I’m after.

      • George 1.1.1

        Great work Lynn. And thank goodness I don’t have to use Disqus – which simultaneously destroys the small measure of privacy I use and slows comments to a crawl.

  2. tjp 2

    So if I read the above correctly it sounds as if the majority of your problems in terms of scalability are due to the frequent cache invalidations caused by new comments? I would be interested in hearing more about why you don’t want to load them dynamically – the additional client coding sounds like a relatively minor tradeoff. It wouldn’t solve the problem of caching multiple versions of the page for different clients but you would be able to hold them in cache longer.

    • lprent 2.1

      Oh I agree that it’d appear to make the site simpler to run. However the coding/testing complexity of doing it is a pain when you look at the detail. And it wouldn’t make much difference in terms of actual operations because of the site layout. We’d have to do a lot more coding to make it work.

      Depends what we’re talking about of course….

      But if I dynamically loaded the comments on the right tab then I’d get benefits for the pages that weren’t getting comments added to them and weren’t the front page.

      However at any given time between a quarter and a half of our page views are the front page. A prominent feature on the front page is the comment count. Every comment causes that page to regenerate. Of course that count could also become dynamic – but it has just doubled the dynamic code to test and maintain.

      At any given time between a quarter and a half of our page views are usually on a single post. This is usually because that particular post is getting a lot of comments, a lot of reads by people commenting, and a lot of reads by the substantial lurker population who read comments closely.

      During the day, about a fifth of the page views are from search engines plowing their way through the nearly 14k posts and 700k comments. We help them as much as we can by telling them where the site is active. But they are a very important source of new readers to the site.

      This means that most of the pages being read through the day and evening are on the most dynamic pages anyway. Usually inside a full day something like 75% of the page views will fall in on less than 10 pages. These will also be the pages that have the most comments being made (or in the case of the front page, comments being reported)

      To be effective, we’d have to do the same code changes amongst ALL of the browsers that access this site. As well as the browsers for the desktop versions of windows, mac and linux, it also includes the ever increasing numbers of variants being used on the mobiles (14% of page views). Just testing those on every one of the frequent code updates for wordpress and plugins beggars belief. The only real way to do it would be to write automated testers – which would be a hell of a coding chore.

      And of course wordpress directly has no support for dynamic comments, which means that we’d be on our own for testing updates across all of those targets platforms. The testing would have to be done with every update on wordpress.

      There are a number of possible alternatives system that do get their own testing done (Disqus for instance).

      But this is a site that gets quite a lot of comments, many of them large and just as interesting as the posts. There is a considerable SEO benefit in optimizing for them – which we have done. So we’d also have to do something about how the search engines access the comments. For instance I ran your first sentence through google and got an immediate hit. If you do the same on a disqus site like whaleoil – the same will not happen. As far as I can tell (a notoriously difficult thing to measure) about a third of the retained new readers come in after querying something that got a strike on a comment.

      There are a whole lot of other issues as well (like the ever shifting jQuery/Ajax systems). Suffice it to say that doing it with servers appeared to be a lot simpler than doing it with code because the returns off doing it with code are a lot less than you’d expect, and the cost of the code is whole lot higher and less transportable.

      Roll on HTML5 which will make some of this easier to do inside wordpress itself.

  3. lprent 3

    Interesting looking at the Shane Jones effect today on servers

    Click image for a larger view. The times are in UTC, so they are 12 hours out. So the Shane Jones story went up just after 06:00

    The load peaked at approximately 3.5x the usual human average maximum weekday middle of the day load at about 7pm and started falling back to under twice the usual load at about 9ish, then spiked again with the evening news. Then dropped rapidly after that.

    No page errors reported from the client to load balancer, but the latency was somewhat spiky as web servers were being added.

    I need a bigger story to really test the spike loading on the site. It didn’t really touch the database or file server. Comments kept pouring in without any problem apart from some moronic trolls unaware of the first comment policy…

    I’m happy with the performance. Especially for a peak loading at about 17-18c per hour…

    • lprent 3.1

      The traffic is certainly heating up.

      The direct traffic out last month for webpages (excluding the static images, javascript, and CSS) was 309GB (short month feb was 319.996 GB). So far this month as at the 22nd, we’re at 338GB. I guess election year is underway.

      Good to see that the changes I made at the end of last month are reducing all of the other costs down.

Links to post

Recent Comments

Recent Posts

  • Police partnership programme with Fiji launched
    A new partnership programme between the New Zealand Police and Fiji Police will focus on combatting transnational organised crime and enhancing investigative skills, Prime Minister Jacinda Ardern announced on the first day of her visit to Fiji. The programme will see: ·       New Zealand Institute of Environmental Science and Research ...
    BeehiveBy beehive.govt.nz
    14 hours ago
  • Joint statement from Prime Minister Ardern and Prime Minister Bainimarama
    Prime Minister Jacinda Ardern and Prime Minister Josaia Voreqe Bainimarama met today in Suva, and renewed their commitment to continue to strengthen Fiji-New Zealand relations on a foundation of shared values and equal partnership. The Prime Ministers acknowledged the kinship between Fijians and New Zealanders, one that has endured over ...
    BeehiveBy beehive.govt.nz
    15 hours ago
  • $19.9 million from PGF for Kawerau
    A $19.9 million investment from the Provincial Growth Fund will help develop essential infrastructure for an industrial hub in the Bay of Plenty town of Kawerau, Regional Economic Development Minister Shane Jones says. “The funding will go to three projects to further develop the Putauaki Trust Industrial Hub, an industrial ...
    BeehiveBy beehive.govt.nz
    19 hours ago
  • PGF funds Mahia roading package
    The Provincial Growth Fund is investing $8.3 million on a roading package for Mahia that will lead to greater and wider economic benefits for the region and beyond, Under Secretary for Regional Economic Development Fletcher Tabuteau announced at an event in Mahia today. The $8.3 million announced consists of: $7 ...
    BeehiveBy beehive.govt.nz
    21 hours ago
  • 18,400 children lifted out of poverty
    Prime Minister Jacinda Ardern has welcomed new reporting showing the Coalition Government is on track to meet its child poverty targets, with 18,400 children lifted out of poverty as a result of the Families Package.   Stats NZ has released the first set of comprehensive child poverty statistics since the Government ...
    BeehiveBy beehive.govt.nz
    21 hours ago
  • 20,000 more Kiwi kids on bikes
    Associate Minister of Transport Julie Anne Genter today announced that Bikes in Schools facilities have been rolled out to 20,000 more kiwi kids under this Government. She made the announcement at the opening of a new bike track at Henderson North School in Auckland. “Bikes in Schools facilities give kids ...
    BeehiveBy beehive.govt.nz
    23 hours ago
  • Benefit settings rise in line with wages as of 1 April
    Benefit settings rise in line with wages as of 1 April   Main benefits will increase by over 3 percent, instead of 1.66 percent, on 1 April with the Government’s decision to annually adjust benefit rates to increases in the average wage. The Minister for Social Development, Carmel Sepuloni, said ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Foreign and Trade Ministers to lead business delegation to India
    Strengthening New Zealand’s political and business ties with India will be the focus of Foreign Affairs Minister Winston Peters’ and Trade and Export Growth Minister David Parker’s visit to India this week. The Ministers are co-leading a high level business delegation to India to support increased people and economic engagement ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Minister champions more Pacific in STEM – Toloa Awards
    The Minister for Pacific Peoples, Aupito William Sio continues to champion for greater Pacific participation in STEM (Science, Technology, Engineering and Mathematics) careers with the announcement of the Toloa Awards, with 8 recipients of the Toloa Community Fund and 13 Toloa Tertiary Scholarships. “The Toloa Programme encourages more Pacific peoples ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Submission period for whitebait consultation extended
    Conservation Minister Eugenie Sage has extended the date for people to have their say on proposed changes to improve management of whitebait across New Zealand.   Submissions were due to close on 2 March 2020 but will now remain open until 9am on Monday 16 March 2020.   “I have ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • New international protection for frequent fliers
    The endangered toroa/Antipodean albatross has new international protection for its 100,000km annual migration, thanks to collaborative efforts led by New Zealand, Australia and Chile.   Today, 130 countries agreed to strictly protect Antipodean albatross at the Conference of Parties on the Convention on the Conservation of Migratory Species of Wild ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Government to regulate vaping
      No sales to under-18-year-olds No advertising and sponsorship of vaping products and e-cigarettes No vaping or smokeless tobacco in smokefree areas Regulates vaping product safety comprehensively, - including devices, flavours and ingredients Ensure vaping products are available for those who want to quit smoking   Vaping regulation that balances ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Justice Minister represents New Zealand at Berlin nuclear disarmament summit
    Justice Minister Andrew Little will travel to Berlin tomorrow to represent New Zealand at a high-level summit on nuclear disarmament. This year, the Nuclear Non-Proliferation Treaty (NPT) celebrates 50 years since it entered into force. “New Zealand’s proud record and leadership on nuclear disarmament is unwavering, so it’s important we are present ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Prime Minister to visit Fiji and Australia
    Prime Minister Jacinda Ardern will visit two of New Zealand’s most important Pacific partners, Fiji and Australia, next week. The visit to Fiji will be the first by a New Zealand Prime Minister in four years and comes during the 50th anniversary of Fijian independence and diplomatic relations between our ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Next steps in Criminal Cases Review Commission announced
    Justice Minister Andrew Little and New Zealand First MP Darroch Ball, have today announced the appointment of the Chief Commissioner of the Criminal Cases Review Commission (CCRC), the location, and the membership of the Establishment Advisory Group. Colin Carruthers QC has been appointed Chief Commissioner of the CCRC for an ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Horticultural Ahuwhenua Trophy finalists announced
    Māori Development Minister Hon Nanaia Mahuta and Agriculture Minister Hon Damien O’Connor co-announced the first horticultural finalists for the Ahuwhenua Trophy celebrating excellence in the Māori agricultural sector.  The three finalists are Ngai Tukairangi Trust from Mt Maunganui, Otama Marere Trust from Tauranga, and Hineora Orchard Te Kaha 15B Ahuwhenua ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • New support for students with dyslexia
    A new kete of resources to strengthen support for students with dyslexia will provide extra tools for the new Learning Support Coordinators (LSCs) as they start in schools, Associate Education Minister Tracey Martin announced today. The Minister launched the kete in Wellington this morning, at the first of three induction ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Rental reforms progress to select committee stage
    The Government continues to make progress on a fairer and more secure rental market for renters and landlords with the First Reading of the Residential Tenancies Amendment Bill and its referral to the Social Services and Community Select Committee.  “Now is the opportunity for landlords, tenants and others who want ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Papua New Guinea Prime Minister to visit New Zealand
    Papua New Guinea’s Prime Minister Hon James Marape will visit New Zealand from 21-25 February, Prime Minister Jacinda Ardern announced today. “New Zealand and Papua New Guinea have a warm and friendly relationship. I look forward to welcoming Prime Minister Marape here and strengthening the relationship between our two countries,” ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Free school lunches served up to thousands
    Thousands of children have begun receiving a free lunch on every day of the school week, Prime Minister Jacinda Ardern said. The Government’s free and healthy school lunch programme is under way for 7,000 students at 31 schools in Hawke’s Bay / Tairāwhiti and Bay of Plenty / Waiariki, extending ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Social Wellbeing Agency replaces Social Investment Agency with new approach
    The Minister for Social Development Carmel Sepuloni today announced a new approach that continues to broaden the Government’s social sector focus from a narrow, investment approach to one centred on people and wellbeing. Minister Sepuloni said redefining the previous approach to social investment by combining science, data and lived experience ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Government to strengthen protections for whistleblowers
    The Government is strengthening the Protected Disclosures Act to provide better protection for whistle blowers, Minister of State Services Chris Hipkins said today. “The Protected Disclosures Act is meant to encourage people to speak up about serious wrongdoing in the workplace and protect them from losing their jobs or being ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • PM speech at Parliamentary Chinese New Year celebration 2020
    Nǐn hǎo (Hello in Mandarin). Xīn Nián Kuài Lè (Happy New Year in Mandarin) Néi Hóu (Hello in Cantonese). Sun Nin Fai Lok (Happy New Year in Cantonese) Tēnā koutou, tēnā koutou, tēnā koutou katoa. Thank you for your invitation to attend this celebration today. I would like to acknowledge ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • IPANZ Annual Address
    Kia ora. Ngā mihi nui ki a koutou katoa. Nau mai haere mai ki te Whare Pāremata. E ngā mana whenua ki tēnei rohe Taranaki Whānui, Te Upoko o Te Ika, Ngāti Toa Rangatira, Ngāti Raukawa – kei te mihi, kei te mihi, kei te mihi. E ngā mana, e ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • 2020 IPANZ Annual Address
    Kia ora. Ngā mihi nui ki a koutou katoa. Nau mai haere mai ki te Whare Pāremata. E ngā mana whenua ki tēnei rohe Taranaki Whānui, Te Upoko o Te Ika, Ngāti Toa Rangatira, Ngāti Raukawa – kei te mihi, kei te mihi, kei te mihi. E ngā mana, e ngā ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Tougher penalties for gun crime a step closer
    Tougher penalties for gun crime are a step closer with the passage of firearms reform legislation through another stage in Parliament. The Arms Legislation Bill has tonight passed its Second Reading. “The changes have one objective - to prevent firearms falling into the wrong hands,” said Police Minister Stuart Nash. ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Arms Legislation Bill: Second Reading
    Introduction Mr Speaker We all know why we are here today. It has been a long journey. The journey did not actually begin on 15 March 2019. It began on 30 June 1997. Almost 23 years ago, Justice Sir Thomas Thorp told us what was wrong with our firearms legislation. ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • New era for vocational education
    The Government’s work to put trades and vocational education back on the agenda took another major step forward today with the passing of the Education (Vocational Education and Training Reform) Amendment Bill, Education Minister Chris Hipkins said. “This is a watershed day for trades and vocational education. These law changes formalise ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Bill to Amend the Greater Christchurch Regeneration Act
    Speeding up the return of Christchurch regeneration activities to local leadership is behind the Greater Christchurch Regeneration Amendment Bill introduced to Parliament today by Minister Megan Woods. “As we approach nine years since the February 2011 earthquake in Canterbury, and with the transition to local leadership well underway, the time ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Milford Track to partly reopen after storm damage
    Hundreds of New Zealanders and international visitors will be able to get back out into nature with the Milford Track partially reopening next week, after extensive assessments and repairs, the Minister of Conservation Eugenie Sage announced today. The popular Great Walk has been closed since 3 February after an extreme ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Government drives low-emissions transport momentum
    Up to 110 new EV chargers nationwide in cities and regions 50 electric vehicles for ride-sharing The Government is helping deliver more infrastructure and options for low emissions transport through new projects, Energy and Resources Minister Dr Megan Woods says. Tauranga, Nelson, Levin, New Plymouth and Oamaru are just some ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Kiwis better off under Coalition Government
    New Zealanders are increasingly better off under this Government as wages rise and families have more disposable income, Finance Minister Grant Robertson says. Stats NZ reported today that average household disposable incomes after housing costs rose 4.9% in 2019. This was the highest rise in four years and came as ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Another step towards restoring rights for screen production workers
    All New Zealanders need to have their voices heard at work to ensure we have an inclusive and productive economy. Today we introduce a Bill to do this for workers in the New Zealand screen industry, Workplace Relations and Safety Minister Iain Lees-Galloway says. “The Screen Industry Workers Bill will ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Enhanced Taskforce Green for Southland and South Otago
    The Government has announced further help for the Southland and Otago regions to speed up recovery efforts from the floods.  “I’ve approved Enhanced Taskforce Green (ETFG), making $500,000 available to help with the clean-up in Fiordland, Southland, and the Clutha district in Otago,” Minister for Social Development Carmel Sepuloni said.  ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Employers and Industry take the lead to connect students to vocational education
    Following the announcement that more than 340 schools will be funded to run events promoting vocational education, the Government has announced it will fund a further 257 events to be run by employers and industry. “These industry-run events will allow more than 30,000 students to connect with more than 2,000 ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Rental reforms a step closer with the introduction of the Residential Tenancies Amendment Bill
    Today the Government is making progress on a fairer and more secure rental market for renters and landlords with the introduction of the Residential Tenancies Amendment Bill in Parliament.  “This Bill includes a series of reforms to improve the wellbeing of the 609,700 households that live in rented homes, and ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Biosecurity Minister announces world first eradication of pea weevil
    A Government programme to wipe out pea weevil has achieved a world first, with Biosecurity Minister Damien O’Connor today announcing the successful eradication of the noxious pest from Wairarapa. This means the nearly four-year ban on pea plants and pea straw was lifted today. Commercial and home gardeners can again grow ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Temporary Accommodation Service activated for Southland flooding
    Southland residents hit by flooding caused by heavy rainfall can now access help finding temporary accommodation with the Government activating the Temporary Accommodation Service, Minister of Civil Defence Peeni Henare announced today. “The Temporary Accommodation Service (TAS) has been activated by the Ministry of Business, Innovation and Employment to help ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Bridges: Over-hyped and under-delivered
    “Is that it?” That’s Finance Minister Grant Robertson’s response to Simon Bridges’ much-hyped economic speech today. “Simon Bridges just gave the most over-hyped and under-delivered speech that I can remember during my time in politics,” Grant Robertson said. “It’s not surprising. Simon Bridges literally said on the radio this morning ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Police to trial eye in the sky in Christchurch
    A trial deployment of the Police Eagle helicopter in Christchurch will test whether the aircraft would make a significant difference to crime prevention and community safety. “The Bell 429 helicopter will be based in Christchurch for five weeks, from 17 February to 20 March,” said Police Minister Stuart Nash. “The ...
    BeehiveBy beehive.govt.nz
    1 week ago