web analytics

Meet The Standard

Written By: - Date published: 4:39 pm, September 7th, 2014 - 30 comments
Categories: admin, The Standard - Tags:

2014-09-07 15.27.01

The communications (click for a larger image)

Some of you will be wondering what the announcement at the top of the screen about an outage tonight is about. I have to power down the system to put in uninterruptible power supplies (UPS) to handle possible power outages. The reason why is that after 5 years on server farms, The Standard has moved home1.

The Standard has been running on a home server running fibre connection2 since August 3rd, which has massively reduced the cost of running The Standard. With the exception of an outage for a network card that became unreliable, it has been running perfectly.

While originally envisaged as a backup system for the cloud based servers at Amazon in Sydney, it has proved fast enough on the NZ network to provide a significiant drop in latency for our mainly NZ audience.

So this evening the system will get the UPS that should allow it to handle any reasonably short power outages. You can see the new 650 VA UPS in the front of the photo waiting to get plugged in. Mounted on the wall are the Chorus fibre end point, Zyxel switch, and our main VOIP phone (whose main user is Lyn’s mother calling us).

The fibre connection is running at 100MBits/sec down and 50MBits/sec up, and has been easily able to handle the load including the spikes of 250 users per minute online after the launch of Dirty Politics.

The average traffic load has doubled during the month as we head into the election. In July according to Statcounter3 we had 444 thousand page views from humans. In August it was 717 thousand. Base don previous elections, I’m expecting that this month it will be a lot closer to a million page views.

2014-09-07 15.27.57

The processors (click for a larger image)

It runs on a quiet watercooled 8 core AMD box with 24GB of RAM and 256GB of solid state drive. Next to it is a dual core AMD server which has a 8TB RAID6 array and 16GB RAM. It acts as the immediate backup server for the site, and feeds the offshore ‘warm’ backup servers in Sydney that I set up a few weeks ago.

These two systems are usually my home development systems and are rather notorious for being a messy mass of cables and keyboards (night and wired4). On top of the servers is a rarely used printer scanner with a bluray player that acts as the digital gateway to the wall mounted TV above it and out of the picture.

The new 1600VA UPS is sitting in front of the servers waiting to go in tonight.

So now when you see me saying that you can’t treat The Standard as a person with opinions because it is a machine. You’ll now know what I’m talking about….

But the reality is that this is just the hardware. The Standard is the sum of the loosely cooperative authors and their mixture of different viewpoints, and the commenters who have fun analyzing everything to death.

It appears this blog runs on quite a different model to those on the right that have been  exposed in Dirty Politics. It is far closer to the bulletin boards and usenet that I played on when I was a somewhat younger person in my 20s and 30s. The hardware even looks the same, just black rather than IBM beige..


 

  1. The Standard started on a PC. It moved to a donated share on ISP hosted servers in 2008. These systems turned out to have been donated by the ISP to the NZLP and run by an activist. To avoid conflict of interest questions when this was discovered, after a week it got moved to a virtual private server. Since then it has toured the world in server farms in Colorado, San Diego, Auckland and Sydney as it overflowed the server bounds due to growth or costs (NZ servers are damn expensive). The reason it is able to come home again is because of the widening of the bandwidth that the UFB provides. I tried running it at home briefly in 2009 after a server provide booted the site off for traffic volumes. But an ADSL connection just couldn’t handle the peak loads. Since it moved home on August 3rd, google analytics reckons that 89 thousand unique people have read the site.
  2. This was a UFB connection that I had started to sign up for in July 2012. The plan was that the UFB would be in my street in May 2013. It actually finished in June 2013. And then the fun started. It turned out that Steven Joyce had forgotten or left a few crucial details about strata title apartments when negotiating the UFB rollout back in 2009. It took Amy Adams to resolve these in May 2013. Between the time for Chorus to get organised with the body corporate to install the gear, it was June 2014 to be able to install the gear for the block. Then another couple of months to get it linked to my apartment. Completely sloppy bearing in mind that most areas that it has been installed in Auckland have high concentrations of apartment blocks.
  3. Statcounter is used to provide public access stats for Open Parachute. I mostly read the google analytics statistics which are a lot less variable and quite a lot more detailed than Statcounter, Sitemeter, and most of the other analysis tools. I also read the apache logs which measure the actual load as it doesn’t filter out the bots.
  4. You have to have keyboards to run a computer. If you are like me you have specialised keyboards for different tasks.I usually use a Logitech K750 solar powered keyboard and I have three of these. One at work, one at home, and one to take with me when I travel (they are really thin and light). This means that for normal coding tasks my ‘finger memory’ is the same in all locations. This saves me from cursing as my fingers slip on to the wrong keys because of spacing and height differences. I write somewhere between half a million and a million lines of code per year, and those computers are damn finicky!But when I wind up working on issue after lights out (ie when Lyn goes to sleep), I use a wireless illuminated Logitech keyboard (left in the photo above). When I am setting machines up and the USB isn’t working, then I use the little cabled PS2 keyboard (on top of the servers).

30 comments on “Meet The Standard ”

    • lprent 1.1

      I like google analytics for the depth of information I get off it for free. Recently though I have been starting to purchase more data from Statcounter because of certain bits of very useful information. For instance this one.

      That our readers spend a lot of time on the site was well known. But until I saw this, I didn’t realise just how long a lot of them did stay onsite. They would have had to have clicked on a page at least once within every half hour and not clicked out to get that “longer than an hour” statistic.

      The less than 5 seconds is where people came on for a single page and immediately left. Usually from a search engine.

      • ianmac 1.1.1

        KDC said that Whaleoil had a huge percentage of visits from outside countries without name. Only 42% seemed to be NZ visitors. Do you think he artificially inflates his numbers. I think KDC wrote about bots whatever they are.

        • lprent 1.1.1.1

          The problem is that KDC gave no information on where he got that data from. That makes it meaningless.

          I suspect that he is right. However it isn’t anywhere near proof that he is right. It is just a silly distraction.

  1. Draco T Bastard 2

    While originally envisaged as a backup system for the cloud based servers at Amazon in Sydney, it has proved fast enough on the NZ network to provide a significiant drop in latency for our mainly NZ audience.

    The fibre connection is running at 100MBits/sec down and 50MBits/sec up, and has been easily able to handle the load including the spikes of 250 users per minute online after the launch of Dirty Politics.

    Which is why we need FttH across the country and, AFAIC, it needs to be synchronous and around 1gbps. It really would allow for a massive expansion of collaboration and thus innovation across the country.

    • lprent 2.1

      Yes, and I have to say that Steven Joyce has certainly muffed it badly.

      Virtually everyone I have talked to (businesses and individuals) has had major hassles getting fibre in because of his really terrible decisions.

      Amy Adams has done a far better job on getting the detail in place that makes it possible to get connected more easily. But that took 5 years to do.

      • Lanthanide 2.1.1

        Amy Adams seems like a very competent up-and-comer to me. Good to have at least one woman in the cabinet who is for more than just show.

  2. ianmac 3

    Those of us who wake up to read the Standard every day and every evening, salute you Lynn. I hope we don’t take your commitment for granted just because it is reliable and omnipresent. Thanks.

    • lprent 3.1

      Oh it is definitely not omnipresent. After all I have to go to work each week day and sometimes during the weekends. That just mens that I lug a cellphone, tablet, and laptop where ever I go.

      Mostly what happens is that the main timewasters (ie trolls) tend to find out that I don’t like having my time wasted and tend to adjust their behaviour over time. I don’t care particularly about what people say, but I do care about peoples behaviour – and I force it to be adjusted within our purview. A bit of primitively effective social engineering.

  3. Hami Shearlie 4

    We are all very grateful to you LPRENT!!!

  4. CnrJoe 5

    Muchos gracias – where would we be…?

  5. Tracey 6

    yay you mr prentice

  6. Ad 7

    I have no idea what you said except “shutdown tonight”, but if you ever wanted a fundraiser event, let me know and I will happily do one for you.

    Nice big fat dinner, so we can invite all the nom de plumes who want to.

    Great to see the traffic going though the roof LPrent.

    • lprent 7.1

      Cost last month was $185 for UFB and $166 for Amazon AWS. It dropped from about $660 the previous month. Should be down further this month as I eliminated a excessive search bot issue from Baidu spider.

    • lprent 7.2

      I’d have to say that the AWS (amazon web services) was great until the site traffic went through the roof. It peaked at just under 900GB for a month.

      Then it started to cost as much as the site that I’d shifted from where it was running at 250GB and the 120GB of overseas traffic was the cost. The issue was that it was bloody hard to get a good remote analysis of what was going wrong apart from it being a bot problem. I found it within days after I shifted it back here. It was a problem on the site that the baidu spider was too dumb to “see” so it kept fetching the same page under a different URL.

      We actually get more enough from the minimal advertising we put on site provided that the advertising money turns up. But my MBA was in operations and I come from a family that tends to do production and operations. I just can’t resist figuring out how to do more with a lot less.

      This last round of productivity/cost means that I could do an enormous improvement in local performance by siting in the NZ network where 89% of our readers were from. I now use AWS for the graphics, server routing, and backup/warm server cluster.

      I’ve just been looking at the last weeks traffic stats. The overseas traffic is still way too high. But from the pattern it is generally the requests that is over the top. That is what I am working on now. I’m going through the site removing requests for small graphics and either making it CSS or out of a consolidated graphic.

  7. yeshe 8

    You are pure gold, Lprent .. thank you 🙂

  8. philj 9

    A true public service. Thank you.

  9. Harry Holland 10

    “The Standard is the sum of the loosely cooperative authors and their mixture of different viewpoints, and the commenters who have fun analyzing everything to death.”

    Very like the old days of the greenviews mailing list back in the 90’s, hosted by Sigi Kirchmair, but eventually shut down as GPANZ sought to take things in-house and exert control and security on contentious discussion.

    Not being ‘owned’ by any political party is a great strength of The Standard.

  10. infused 11

    Zyxel router. Poor you. You should change that piece of shit out straight away. If you are on fibre, put mikrotik/cisco/somethinghalfdecent in.

    That zyxel won’t even push 100/100 with the firewall on.

    • lprent 11.1

      Never had a problem with a Zyxel. However I have had a Cisco high end fail. And I avoid Cisco’s linksys like the plague as I have never had one reach the end of its life with me. Basically I regard the company as crap.

      However, I’m only on 100/50 and the testing shows 94/48 depending on what is on the other end. Good enough for the price I’m paying for it.

      • infused 11.1.1

        The Zyxel firmware is as buggy as it comes. I sold these things for like 5 years before so much failure made me go elsewhere.

        Since you like fiddling with things, look at something like this.

        http://routerboard.com/RB2011UiAS-2HnD-IN

        Would be good for your custom rules, like banning bots etc. Easy to make some honeypots using that to catch a lot of crap that comes in. I run them everywhere now.

        • lprent 11.1.1.1

          I usually don’t trap them in device firewalls. All of them are not fast, they aren’t smart, aren’t dynamic, and all have useless interfaces.

          I don’t expect a router to be smart and I find that the simplier I configure them, the less aggravation I have.

          I pinhole to a gateway server handling the service(s). These days that runs (on linux) iptable or ufw rules set up by one of many tools that writes those rules. Some of those are pretty smart and all of the tools have a lot of CPU running open source code that is debugged and widely known.

          It sounds to me like you were expecting too much of a simple device. After writing embedded code for large chunks of the last 8 years I know from the inside how limited their processors are. I usually use them well within design limits. They aren’t PCs or servers (which I have been writing for over the last 30 years). Their processors tend to be more like a 8051 than a 80286.

        • lprent 11.1.1.2

          Your router board looks interesting if I ran windows servers on the net. But these days I don’t. The ones at work that are outward facing are run by the IT department. The ones that I work on are on isolated subnets, and I’m interested in their application code rather than their security.

          • infused 11.1.1.2.1

            They are not windows based fyi. RouterOS is based on linux. There is a Windows winbox program, but also ssh, telnet and http ways of configuring them.

            I probably do expect too much from them. After all SPI is the thing these days. Your static firewall is dead.

            Trapping them on the router is far more efficient use of the routers resources instead of offloading it on to the servers etc.

            You should grab a $50 one just to play 😛

  11. sophie 12

    I salute Lyn for putting up with that monstrosity in what looks like the corner of the kitchen!

    Thanks Lyn and, of course, thanks LPrent.

  12. yeshe 13

    So much faster this morning — what a delight … thx again Lynn.

  13. veutoviper 14

    My thanks also for all your hard work and dedication to keeping TS available and well maintained, Lprent.

  14. raa 15

    “It is far closer to the bulletin boards and usenet that I played on when I was a somewhat younger person in my 20s and 30s.”

    It still has that feel .. long may it last – but
    depending only on one person may limit
    its longevity.

Links to post

Recent Comments

Recent Posts

  • Twenty highlights of 2020
    As we welcome in the new year, our focus is on continuing to keep New Zealanders safe and moving forward with our economic recovery. There’s a lot to get on with, but before we say a final goodbye to 2020, here’s a quick look back at some of the milestones ...
    2 weeks ago

  • Cook Islanders to resume travel to New Zealand
    The Prime Minister of New Zealand Jacinda Ardern and the Prime Minister of the Cook Islands Mark Brown have announced passengers from the Cook Islands can resume quarantine-free travel into New Zealand from 21 January, enabling access to essential services such as health. “Following confirmation of the Cook Islands’ COVID ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Supporting communities and landowners to grow employment opportunities
    Jobs for Nature funding is being made available to conservation groups and landowners to employ staff and contractors in a move aimed at boosting local biodiversity-focused projects, Conservation Minister Kiritapu Allan has announced. It is estimated some 400-plus jobs will be created with employment opportunities in ecology, restoration, trapping, ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Border exception for some returning international tertiary students
    The Government has approved an exception class for 1000 international tertiary students, degree level and above, who began their study in New Zealand but were caught offshore when border restrictions began. The exception will allow students to return to New Zealand in stages from April 2021. “Our top priority continues ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Tiwai deal gives time for managed transition
    Today’s deal between Meridian and Rio Tinto for the Tiwai smelter to remain open another four years provides time for a managed transition for Southland. “The deal provides welcome certainty to the Southland community by protecting jobs and incomes as the region plans for the future. The Government is committed ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • New member for APEC Business Advisory Council
    Prime Minister Jacinda Ardern has appointed Anna Curzon to the APEC Business Advisory Council (ABAC). The leader of each APEC economy appoints three private sector representatives to ABAC. ABAC provides advice to leaders annually on business priorities. “ABAC helps ensure that APEC’s work programme is informed by business community perspectives ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Govt’s careful economic management recognised
    The Government’s prudent fiscal management and strong policy programme in the face of the COVID-19 global pandemic have been acknowledged by the credit rating agency Fitch. Fitch has today affirmed New Zealand’s local currency rating at AA+ with a stable outlook and foreign currency rating at AA with a positive ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Additional actions to keep COVID-19 out of NZ
    The Government is putting in place a suite of additional actions to protect New Zealand from COVID-19, including new emerging variants, COVID-19 Response Minister Chris Hipkins said today. “Given the high rates of infection in many countries and evidence of the global spread of more transmissible variants, it’s clear that ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • 19 projects will clean up and protect waterways
    $36 million of Government funding alongside councils and others for 19 projects Investment will clean up and protect waterways and create local jobs Boots on the ground expected in Q2 of 2021 Funding part of the Jobs for Nature policy package A package of 19 projects will help clean up ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Zealand Government acknowledges 175th anniversary of Battle of Ruapekapeka
    The commemoration of the 175th anniversary of the Battle of Ruapekapeka represents an opportunity for all New Zealanders to reflect on the role these conflicts have had in creating our modern nation, says Associate Minister for Arts, Culture and Heritage Kiri Allan. “The Battle at Te Ruapekapeka Pā, which took ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Better care for babies with tongue-tie
    Babies born with tongue-tie will be assessed and treated consistently under new guidelines released by the Ministry of Health, Associate Minister of Health Dr Ayesha Verrall announced today. Around 5% to 10% of babies are born with a tongue-tie, or ankyloglossia, in New Zealand each year. At least half can ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Prisoner disorder event at Waikeria Prison over
    The prisoner disorder event at Waikeria Prison is over, with all remaining prisoners now safely and securely detained, Corrections Minister Kelvin Davis says. The majority of those involved in the event are members of the Mongols and Comancheros. Five of the men are deportees from Australia, with three subject to ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Pre-departure COVID-19 test for travellers from the UK and the US from 15 January
    Travellers from the United Kingdom or the United States bound for New Zealand will be required to get a negative test result for COVID-19 before departing, and work is underway to extend the requirement to other long haul flights to New Zealand, COVID-19 Response Minister Chris Hipkins confirmed today. “The new PCR test requirement, foreshadowed last ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • PM congratulates New Year Honour recipients
    Prime Minister Jacinda Ardern has added her warm congratulations to the New Zealanders recognised for their contributions to their communities and the country in the New Year 2021 Honours List. “The past year has been one that few of us could have imagined. In spite of all the things that ...
    BeehiveBy beehive.govt.nz
    3 weeks ago
  • David Parker congratulates New Year 2021 Honours recipients
    Attorney-General and Minister for the Environment David Parker has congratulated two retired judges who have had their contributions to the country and their communities recognised in the New Year 2021 Honours list. The Hon Tony Randerson QC has been appointed a Companion of the New Zealand Order of Merit for ...
    BeehiveBy beehive.govt.nz
    3 weeks ago
  • New Year’s Honours highlights outstanding Pacific leadership through challenging year
    Minister for Pacific Peoples Aupito William Sio says the New Year’s Honours List 2021 highlights again the outstanding contribution made by Pacific people across Aotearoa. “We are acknowledging the work of 13 Pacific leaders in the New Year’s Honours, representing a number of sectors including health, education, community, sports, the ...
    BeehiveBy beehive.govt.nz
    3 weeks ago
  • Supporting seniors to embrace technology
    The Government’s investment in digital literacy training for seniors has led to more than 250 people participating so far, helping them stay connected. “COVID-19 has meant older New Zealanders are showing more interest in learning how to use technology like Zoom and Skype so they can to keep in touch ...
    BeehiveBy beehive.govt.nz
    3 weeks ago