web analytics

How this site’s webservers now work

Written By: - Date published: 4:52 pm, April 18th, 2014 - 8 comments
Categories: admin, The Standard - Tags:

Dolphin Over World 2010This site gets rather a lot of comments. Every time a comment is made the page it is on has to be regenerated, which limits the amount of static caching that can be done. So on a dynamic site like this one we require rather a lot of raw processing power to generate dynamically pages that keep changing all of the day.

There are a number of solutions for this issue from offloading the comments to an external supplier like Disqus (with the consequent issues about privacy and control) or to the excessive caching of pages when new comments come in (causing out of date pages) or to separating the comments into a dynamically loaded system (requiring extra client side javascript).

But we like to make the site as simple as possible, as up to date as possible and as secure as possible. So when comments arrive, this site updates the site.

Because of the current design of the pages (something that is always under review), this means that whenever a comment is posted, all pages with the list of recent comments is invalid. Because that is on every page to assist with navigation, the next time a page is read by anyone it is recalculated and laid out. While the server will cache that page, it will do so for a particular version of the browser that it was destined for. That is because not all pages are served up the same. It depends on what type of browser asked for it. So a page may be regenerated many times.

That requires a significiant amount of processing power when there are a lot of people commenting at the same time as there are lot of people reading the site. Outside of the middle of the night, those demand levels tend to quite unpredictable.

It’d be pretty easy to simply get bigger servers that could handle all of the possible demand. But how big is enough? We have had times when a story has been breaking where the load on the server has been for short periods of time been more than 10 times the usual load. We’ve had times when those peaks have killed the site running with a powerful 8 core system. During an election year we usually get a lot more traffic than we previously have had.

So since late last year we have been running the web facing parts of this site on cloud servers and setting up the required infrastructure to allow us to run multiple servers with common file, database, and page/query cache services.

This years we’ve been steadily getting a auto-scaling system largely triggered by CPU levels working. For a single webserver we use something that is rather like a dual core CPU laptop. Not massively fast, but capable of handling a moderate amount of traffic simultaneously. This is our quanta.

When the average processing usage for the existing web servers rises about a set value for more than a few minutes, then a new server of that type is automatically hired from Amazon, equipped with our standard disk image, and thrown on to the load balancer. If the demand keeps rising, then so do the number of servers in usage.

Conversely when the average processing usage drops below a certain value for a long period, then a server is dropped. This keeps happening until the servers are working productively. They go off to do some other type of task for someone else.

This kind of on-demand system is damn hard on people because they really don’t like chopping and changing all of the time as it is intensely unproductive to keep retraining people for different tasks. But computers are machines. You give them a new filesystem and they are trained. The hard bit is creatively figuring out what should be on that filesystem – a task that humans are usually pretty good at.

Anyway, I thought I’d share what our webservers have been doing over the last couple of weeks. Bear in mind that the times on this graph are in UTC (Universal time or Greenwich time) so they are currently about 12 hours off NZ time.

Web server loadings

You can see the strong diurnal cycle (click through the image for a higher resolution one if you cannot). The servers drop down to two or even one overnight. You can also see the weekends and public holidays where the servers during the day run a three and sometimes four. During the week days we’re usually running at four and sometimes five servers.

Each server costs about 2.8c per hour to have (with variable costs like bandwidth as extra), so while it doesn’t cost much, it adds up over a month to save the overnight costs. With a voluntary site like this all money saved is pretty useful.

But you can also see peaks for short bursts where we are running at six, seven or even eight servers when a story goes active and everyone piles into the site. And this fortnight has been a quiet period news wise.

I’ve currently capped the maximum numbers of web servers at 35 based on the potential loads on the database and file servers. Hopefully that will be enough to cope with the upcoming demand. Besides, I can always start autoscaling those if I really have to…

8 comments on “How this site’s webservers now work ”

  1. andy (the other one) 1

    Sorry, some times I hit F5 too much. mostly when felix or Sanc are trolling Syrlands or Pete G.

    Its my online crack habit…

    Its irresistable.

    • lprent 1.1

      I don’t care. It is literally what the system was designed for.

      BUT don’t do it more often than 15 times in a minute. If you do then you will find the site abruptly slows down on you for the next two hours because Wordfence now treats you as a pain-in-the-arse. I find that this seems to induce the change in behaviour that I’m after.

      • George 1.1.1

        Great work Lynn. And thank goodness I don’t have to use Disqus – which simultaneously destroys the small measure of privacy I use and slows comments to a crawl.

  2. tjp 2

    So if I read the above correctly it sounds as if the majority of your problems in terms of scalability are due to the frequent cache invalidations caused by new comments? I would be interested in hearing more about why you don’t want to load them dynamically – the additional client coding sounds like a relatively minor tradeoff. It wouldn’t solve the problem of caching multiple versions of the page for different clients but you would be able to hold them in cache longer.

    • lprent 2.1

      Oh I agree that it’d appear to make the site simpler to run. However the coding/testing complexity of doing it is a pain when you look at the detail. And it wouldn’t make much difference in terms of actual operations because of the site layout. We’d have to do a lot more coding to make it work.

      Depends what we’re talking about of course….

      But if I dynamically loaded the comments on the right tab then I’d get benefits for the pages that weren’t getting comments added to them and weren’t the front page.

      However at any given time between a quarter and a half of our page views are the front page. A prominent feature on the front page is the comment count. Every comment causes that page to regenerate. Of course that count could also become dynamic – but it has just doubled the dynamic code to test and maintain.

      At any given time between a quarter and a half of our page views are usually on a single post. This is usually because that particular post is getting a lot of comments, a lot of reads by people commenting, and a lot of reads by the substantial lurker population who read comments closely.

      During the day, about a fifth of the page views are from search engines plowing their way through the nearly 14k posts and 700k comments. We help them as much as we can by telling them where the site is active. But they are a very important source of new readers to the site.

      This means that most of the pages being read through the day and evening are on the most dynamic pages anyway. Usually inside a full day something like 75% of the page views will fall in on less than 10 pages. These will also be the pages that have the most comments being made (or in the case of the front page, comments being reported)

      To be effective, we’d have to do the same code changes amongst ALL of the browsers that access this site. As well as the browsers for the desktop versions of windows, mac and linux, it also includes the ever increasing numbers of variants being used on the mobiles (14% of page views). Just testing those on every one of the frequent code updates for wordpress and plugins beggars belief. The only real way to do it would be to write automated testers – which would be a hell of a coding chore.

      And of course wordpress directly has no support for dynamic comments, which means that we’d be on our own for testing updates across all of those targets platforms. The testing would have to be done with every update on wordpress.

      There are a number of possible alternatives system that do get their own testing done (Disqus for instance).

      But this is a site that gets quite a lot of comments, many of them large and just as interesting as the posts. There is a considerable SEO benefit in optimizing for them – which we have done. So we’d also have to do something about how the search engines access the comments. For instance I ran your first sentence through google and got an immediate hit. If you do the same on a disqus site like whaleoil – the same will not happen. As far as I can tell (a notoriously difficult thing to measure) about a third of the retained new readers come in after querying something that got a strike on a comment.

      There are a whole lot of other issues as well (like the ever shifting jQuery/Ajax systems). Suffice it to say that doing it with servers appeared to be a lot simpler than doing it with code because the returns off doing it with code are a lot less than you’d expect, and the cost of the code is whole lot higher and less transportable.

      Roll on HTML5 which will make some of this easier to do inside wordpress itself.

  3. lprent 3

    Interesting looking at the Shane Jones effect today on servers

    Click image for a larger view. The times are in UTC, so they are 12 hours out. So the Shane Jones story went up just after 06:00

    The load peaked at approximately 3.5x the usual human average maximum weekday middle of the day load at about 7pm and started falling back to under twice the usual load at about 9ish, then spiked again with the evening news. Then dropped rapidly after that.

    No page errors reported from the client to load balancer, but the latency was somewhat spiky as web servers were being added.

    I need a bigger story to really test the spike loading on the site. It didn’t really touch the database or file server. Comments kept pouring in without any problem apart from some moronic trolls unaware of the first comment policy…

    I’m happy with the performance. Especially for a peak loading at about 17-18c per hour…

    • lprent 3.1

      The traffic is certainly heating up.

      The direct traffic out last month for webpages (excluding the static images, javascript, and CSS) was 309GB (short month feb was 319.996 GB). So far this month as at the 22nd, we’re at 338GB. I guess election year is underway.

      Good to see that the changes I made at the end of last month are reducing all of the other costs down.

Links to post

Recent Comments

Recent Posts

  • More young Kiwis supported with mental health and addiction services
    Nearly quarter of a million more young New Zealanders will have access to mental health and addiction support in their communities as the Government’s youth mental health programme gathers pace. New contracts to expand youth-specific services across the Northland, Waitematā and Auckland District Health Board areas have been confirmed, providing ...
    BeehiveBy beehive.govt.nz
    17 hours ago
  • New hospital facilities mean fewer trips to Auckland for Northlanders
    Northlanders will no longer automatically have to go to Auckland for lifesaving heart procedures like angiograms, angioplasty and the insertion of pacemakers, thanks to new operating theatres and a cardiac catheter laboratory opened at Whangārei Hospital by Health Minister Andrew Little today. The two projects – along with a new ...
    BeehiveBy beehive.govt.nz
    19 hours ago
  • Fair Pay Agreements to improve pay and conditions for essential workers
    The Government is delivering on its pre-election commitment to implement Fair Pay Agreements which will improve wages and conditions, as well as help support our economic recovery, Workplace Relations and Safety Minister Michael Wood announced today. Fair Pay Agreements will set minimum standards for all employees and employers in an ...
    BeehiveBy beehive.govt.nz
    19 hours ago
  • Establishment of the new Māori Health Authority takes first big step
    Sir Mason Durie will lead a Steering Group to provide advice to the Transition Unit on governance arrangements and initial appointments to an interim board to oversee the establishment of the Māori Health Authority. This Group will ensure that Māori shape a vital element of our future health system, Minister ...
    BeehiveBy beehive.govt.nz
    20 hours ago
  • Cycle trails move up a gear in Central
    Work on new and upgraded cycle trails in Queenstown, Arrowtown and Central Otago is moving up a gear as two significant projects pass further milestones today. Tourism Minister Stuart Nash has announced new funding for the Queenstown Trails Project, and will also formally open the Lake Dunstan Trail at Bannockburn ...
    BeehiveBy beehive.govt.nz
    20 hours ago
  • Picton ferry terminal upgrade consent fast-tracked
    The planned upgrade of the Waitohi Picton Ferry terminal has been approved under the fast-track consenting process.  Environment Minister David Parker today welcomed the decision by the expert consenting panel to approve the Waitohi Picton Ferry Precinct Redevelopment Project.    The project will provide a significant upgrade to the ferry facilities ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Quarantine Free Travel with New South Wales paused
    COVID-19 Response Minister Chris Hipkins has announced his intention to pause Quarantine Free Travel from New South Wales to New Zealand while the source of infection of the two cases announced in Sydney in the last two days is investigated.  Whole genome sequencing has linked the case yesterday to a recent ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Covid-19 immigration powers to be extended
    The passing of a bill to extend temporary COVID-19 immigration powers means continued flexibility to support migrants, manage the border, and help industries facing labour shortages, Immigration Minister Kris Faafoi said. “Over the past year, we’ve made rapid decisions to extend visas, vary visa conditions and waive some application requirements ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • “Supporting a Trade-Led Economic Recovery”
    Trade Policy Road Show SpeechManukau, Auckland   Kia ora koutou – nau mai, haere mai ki Manukau, ki Tāmaki.   Good morning everyone, and thank you for this opportunity to discuss with you current global challenges, opportunities and the Government’s strategy in support of a trade-led recovery from the economic ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Building consent numbers at an all-time high
    A record 41,028 new homes have been consented in the year ended March 2021 March 2021 consent numbers the highest since the 1940s Record number of new homes consented in Auckland The number of new homes consented is at an all-time high, showing a strong and increasing pipeline of demand ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Whānau-centred support for parents and tamariki
    Up to 60 whānau in Counties Manukau will be supported through the first three years of their parenthood by a new whānau-centred model of care, said Associate Health Minister, Hon Aupito William Sio. “Providing this support to young parents is something we have to get right. It’s a priority both ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • NZ backs moves to improve global access to COVID vaccines
    New Zealand welcomes and strongly supports the announcement made by the United States Trade Representative to work for a waiver of IP protections on COVID-19 vaccines at the WTO, Trade Minister Damien O’Connor said. “New Zealand supports equitable access to COVID vaccines for all. No one is safe from the ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Tourism communities: support, recovery and re-set plan
    TIHEI MAURI ORA Tuia te whakapono Tuia te tumanako Tuia te aroha Tuia te hunga ora Ki te hunga ora Tihei Mauri ora Ka nui te mihi ki a koutou Tena koutou, tena koutou, tena koutou katoa. Thank you, Hilary and thank you, Chris, and everyone at TIA for this ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Support, recovery and re-set plan for tourism communities
    Five South Island tourist communities targeted for specialist support Pressure on Māori tourism operators and Conservation facilities recognised Domestic and international-facing tourism agencies put on more secure footing Long-term plan to re-set tourism with a focus on sustainability, industry standards and regional economic diversification A plan to ensure the immediate ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Speech on NZ Rail Plan
    Check against delivery E ngā mana, e ngā reo, e ngā karanga maha o te wa, tēnā koutou, tēnā koutou, tēnā tātou katoa. Ki ngā mana whenua o Taranaki Whānui anō nei aku mihi ki a koutou. Nōku te hōnore kia haere mai ki te whakanuia tēnei huihuinga whakahirahira. Nō ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Government hits massive milestone in Violence Prevention & Elimination
    Minister for Family and Sexual Violence Marama Davidson announced a major milestone at a hui in South Auckland today, with the launch of the national engagement process on the prevention and elimination of family and sexual violence. “There is no room for violence in our lives – there is no ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Fee waiver extended for conservation tourism businesses
    Tourism businesses operating on public conservation land will have another six months of fees waived to help them adjust to the downturn in international visitors in the wake of the COVID-19 pandemic, Acting Minister of Conservation Dr Ayesha Verrall has announced. "We acknowledge it has been a difficult year for ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • ‘Lua Wave’ to future-proof Pasifika Festivals in Aotearoa
    Pasifika festival organisers will receive additional support to adapt to the COVID-19 environment thanks to the Government’s newly launched ‘Lua Wave’ component of the Pasifika Festivals Initiative, Minister for Arts, Culture and Heritage Carmel Sepuloni announced today. “This initiative has not only been to support festival organisers to recover from ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Crown accounts show confidence in Govt economic plan
    The Government’s financial accounts continue to reflect the resilience of the economy and confidence in the Government’s economic recovery plan. The Crown accounts for the nine months to the end of March 2021 show both OBEGAL and the operating balance remain better than forecast in the Half Year Economic and ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Energy Trusts of NZ Autumn Conference
    It’s a pleasure to be here today. Thank you Karen [Sherry] for the introduction and thanks to the Energy Trusts Executive for inviting me to speak at tonight’s event. It is an exciting time to come to speak to trustees of distribution companies. For many decades the electricity industry was ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • New partnership to grow Māori success in STEM
    A new partnership with the Pūhoro STEM Academy will support thousands more rangatahi Māori to participate and succeed in the fields of science, technology, and innovation, Associate Education Minister Kelvin Davis announced today. Since 2016, Pūhoro has worked with Māori students to build their capability and create pathways to employment ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Rail builds platform for economic recovery
    Transport Minister Michael Wood and State-Owned Enterprises Minister Dr David Clark today released the Government’s long term vision for a sustainable rail network that supports our economic recovery. New Zealand Rail Plan lays out how the Government is building a resilient, reliable and safe network, as well as the indicative ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • NZ and UK agree to lift the pace of free trade talks
    New Zealand and the United Kingdom have agreed to rapidly lift the tempo of talks, as the two countries enter a new phase in free trade negotiations, Trade and Export Growth Minister Damien O’Connor announced today. “UK Secretary of State for International Trade, Liz Truss, and I spoke today about ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Counter-Terrorism Legislation Bill passes first reading
    The Counter-Terrorism Legislation Bill has passed its first reading and will now be considered by Parliament’s Justice select committee. “The Bill updates and improves New Zealand’s counter-terrorism legislation and ensures that the right legislative tools are available to intervene early and prevent harm,” Minister of Justice Kris Faafoi said. “The ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Statement on The Speaker and Annual Review Debate
    “The serious issue of alleged sexual assault and harassment at Parliament was poorly managed and inappropriately politicised last night. The tone of the debate did not reflect well on Parliament as a whole,” Prime Minister Jacinda Ardern said. “Any investigation of claims of sexual assault should be in a manner ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Govt motoring towards zero-carbon buses and protecting drivers’ conditions
    Transport Minister Michael Wood is seeking feedback on options for the next phase of the Public Transport Operating Model (PTOM) review to better protect bus drivers’ pay conditions, and also achieving the Government’s target of fully decarbonising the public transport bus fleet by 2035. Michael Wood said investing in our ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Drop in unemployment shows Govt economic plan is working
    The Government’s economic recovery plan continues to be reflected in the labour market, with more people in work and unemployment falling. Stats NZ figures show employment rose by 15,000 in the March quarter, with 14,000 more women in work. The unemployment rate fell from 4.9 percent to 4.7 percent. This ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Government sets pay and workforce expectations for the Public Sector
    The Government’s Workforce Policy Statement issued today sets out its expectations for pay and employment relations in the Public Sector, the Minister of Finance and Minister for the Public Service say. “New Zealand has had an exceptionally successful health and economic response to COVID-19. This has been supported by the ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Author Ben Brown is New Zealand’s first Te Awhi Rito Reading Ambassador
    Lyttleton writer Ben Brown (Ngāti Mahuta, Ngāti Koroki, Ngāti Paoa) will be New Zealand’s first Te Awhi Rito Reading Ambassador, promoting the value of reading for children and young people, Internal Affairs Minister Jan Tinetti announced today. A poet and award-winning author, Ben Brown writes books, non-fiction and short stories ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Celebrating New Zealand’s firefighters this International Firefighters’ day
    With two fire stations already complete, and building underway on 16 fire stations around the country, today we celebrate International Firefighters’ Day for the important role firefighters have in keeping communities across the country safe, says Minister of Internal Affairs, Jan Tinetti. The work is progressing due to Government funding ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Ron Brierley knighthood to go
    Ron Brierley has written to the Clerk of the Executive Council to tender his resignation as a Knight Bachelor. The Queen has been informed. The forfeiture follows the Prime Minister initiating the process to remove his Knighthood. The Clerk of the Executive Council wrote to him on 6 April 2021 ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Employment boost for rural communities
    The Government is continuing to create opportunities for at-risk rangatahi overcome barriers to employment, education or training with the next tranche of He Poutama Rangatahi programmes, Minister for Social Development and Employment Carmel Sepuloni announced today. “We’re focused on supporting rangatahi to get what they need to progress in the ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Pre-Budget speech to Wellington Chamber of Commerce
    Thank you for the invitation to speak today, it is great to be here.  I mean that both sincerely and literally. For this equivalent speech last year I took part virtually, beaming in from the Beehive Theatrette with only a socially distanced press gallery bearing silent witness. You are a ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Budget 2021 reprioritises nearly $1 billion
    The Government’s strong pandemic response and the better than expected economic recovery means not all the money allocated in the COVID-19 Response and Recovery Fund has been spent, Grant Robertson said in his annual pre-Budget speech to the Wellington Chamber of Commerce this morning. “As part of Budget preparation I ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Speech on Digital Identity Trust Framework
    I'd like to start by thanking Graeme, David and Ben from NZTech and Digital Identity New Zealand for inviting me to speak to you. I’m so sorry I can’t be there in person, but I want to acknowledge those of you who are, including some of this country’s top tech ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • NZ Cook Islands travel bubble significant step in COVID-19 recovery
    New Zealand Prime Minister Jacinda Ardern and Cook Islands Prime Minister Mark Brown have today announced that, pending final confirmation by New Zealand’s Director-General of Health and the Cook Islands Secretary of Health, two-way quarantine-free travel will commence between the two countries on 17 May (NZT). “Two way quarantine-free travel ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Minister for State Owned Enterprises saddened by passing of KiwiRail Chair
    Minister for State Owned Enterprises, David Clark is deeply saddened to hear about the passing of KiwiRail Chairman, Brian Corban. “I know Brian was seen as a transformative leader within KiwiRail, well respected for his wisdom, honesty and sense of humour,” said David Clark. Mr Corban served as Chair of ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Speech to the China Business Summit by the Minister for Trade and Export Growth
      Distinguished guests, ladies and gentlemen.  Tena koutou katoa.  Good morning. It is my pleasure to join you today for this China Business Summit – my first as Minister for Trade and Export Growth as well as Minister of Agriculture – and to have the opportunity to speak to you ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Productivity Commission inquiry into immigration settings
    The Productivity Commission will hold an inquiry into immigration settings to ensure New Zealand’s long term prosperity and wellbeing, Grant Robertson and Kris Faafoi say. This inquiry, the first under the new Productivity Commission chair, Dr Ganesh Nana, will focus on immigration policy as a means of improving productivity in ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Speech to China Business Summit
    Ladies and gentlemen, distinguished guests, ate marie, tena koutou katoa, good morning. I am very happy to be here with you once again at my fourth China Business Summit. Thanks again to you, Fran, for being the driving force behind this event.  As ever, I’m very pleased to see such ...
    BeehiveBy beehive.govt.nz
    5 days ago