Meet The Standard

Written By: - Date published: 4:39 pm, September 7th, 2014 - 30 comments
Categories: admin, The Standard - Tags:

2014-09-07 15.27.01

The communications (click for a larger image)

Some of you will be wondering what the announcement at the top of the screen about an outage tonight is about. I have to power down the system to put in uninterruptible power supplies (UPS) to handle possible power outages. The reason why is that after 5 years on server farms, The Standard has moved home1.

The Standard has been running on a home server running fibre connection2 since August 3rd, which has massively reduced the cost of running The Standard. With the exception of an outage for a network card that became unreliable, it has been running perfectly.

While originally envisaged as a backup system for the cloud based servers at Amazon in Sydney, it has proved fast enough on the NZ network to provide a significiant drop in latency for our mainly NZ audience.

So this evening the system will get the UPS that should allow it to handle any reasonably short power outages. You can see the new 650 VA UPS in the front of the photo waiting to get plugged in. Mounted on the wall are the Chorus fibre end point, Zyxel switch, and our main VOIP phone (whose main user is Lyn’s mother calling us).

The fibre connection is running at 100MBits/sec down and 50MBits/sec up, and has been easily able to handle the load including the spikes of 250 users per minute online after the launch of Dirty Politics.

The average traffic load has doubled during the month as we head into the election. In July according to Statcounter3 we had 444 thousand page views from humans. In August it was 717 thousand. Base don previous elections, I’m expecting that this month it will be a lot closer to a million page views.

2014-09-07 15.27.57

The processors (click for a larger image)

It runs on a quiet watercooled 8 core AMD box with 24GB of RAM and 256GB of solid state drive. Next to it is a dual core AMD server which has a 8TB RAID6 array and 16GB RAM. It acts as the immediate backup server for the site, and feeds the offshore ‘warm’ backup servers in Sydney that I set up a few weeks ago.

These two systems are usually my home development systems and are rather notorious for being a messy mass of cables and keyboards (night and wired4). On top of the servers is a rarely used printer scanner with a bluray player that acts as the digital gateway to the wall mounted TV above it and out of the picture.

The new 1600VA UPS is sitting in front of the servers waiting to go in tonight.

So now when you see me saying that you can’t treat The Standard as a person with opinions because it is a machine. You’ll now know what I’m talking about….

But the reality is that this is just the hardware. The Standard is the sum of the loosely cooperative authors and their mixture of different viewpoints, and the commenters who have fun analyzing everything to death.

It appears this blog runs on quite a different model to those on the right that have been  exposed in Dirty Politics. It is far closer to the bulletin boards and usenet that I played on when I was a somewhat younger person in my 20s and 30s. The hardware even looks the same, just black rather than IBM beige..

 

  1. The Standard started on a PC. It moved to a donated share on ISP hosted servers in 2008. These systems turned out to have been donated by the ISP to the NZLP and run by an activist. To avoid conflict of interest questions when this was discovered, after a week it got moved to a virtual private server. Since then it has toured the world in server farms in Colorado, San Diego, Auckland and Sydney as it overflowed the server bounds due to growth or costs (NZ servers are damn expensive). The reason it is able to come home again is because of the widening of the bandwidth that the UFB provides. I tried running it at home briefly in 2009 after a server provide booted the site off for traffic volumes. But an ADSL connection just couldn’t handle the peak loads. Since it moved home on August 3rd, google analytics reckons that 89 thousand unique people have read the site.
  2. This was a UFB connection that I had started to sign up for in July 2012. The plan was that the UFB would be in my street in May 2013. It actually finished in June 2013. And then the fun started. It turned out that Steven Joyce had forgotten or left a few crucial details about strata title apartments when negotiating the UFB rollout back in 2009. It took Amy Adams to resolve these in May 2013. Between the time for Chorus to get organised with the body corporate to install the gear, it was June 2014 to be able to install the gear for the block. Then another couple of months to get it linked to my apartment. Completely sloppy bearing in mind that most areas that it has been installed in Auckland have high concentrations of apartment blocks.
  3. Statcounter is used to provide public access stats for Open Parachute. I mostly read the google analytics statistics which are a lot less variable and quite a lot more detailed than Statcounter, Sitemeter, and most of the other analysis tools. I also read the apache logs which measure the actual load as it doesn’t filter out the bots.
  4. You have to have keyboards to run a computer. If you are like me you have specialised keyboards for different tasks.I usually use a Logitech K750 solar powered keyboard and I have three of these. One at work, one at home, and one to take with me when I travel (they are really thin and light). This means that for normal coding tasks my ‘finger memory’ is the same in all locations. This saves me from cursing as my fingers slip on to the wrong keys because of spacing and height differences. I write somewhere between half a million and a million lines of code per year, and those computers are damn finicky!But when I wind up working on issue after lights out (ie when Lyn goes to sleep), I use a wireless illuminated Logitech keyboard (left in the photo above). When I am setting machines up and the USB isn’t working, then I use the little cabled PS2 keyboard (on top of the servers).

30 comments on “Meet The Standard ”

  • Comments are now closed

    • lprent 1.1
      7 September 2014 at 5:02 pm

      I like google analytics for the depth of information I get off it for free. Recently though I have been starting to purchase more data from Statcounter because of certain bits of very useful information. For instance this one.

      That our readers spend a lot of time on the site was well known. But until I saw this, I didn’t realise just how long a lot of them did stay onsite. They would have had to have clicked on a page at least once within every half hour and not clicked out to get that “longer than an hour” statistic.

      The less than 5 seconds is where people came on for a single page and immediately left. Usually from a search engine.

      • ianmac 1.1.1
        7 September 2014 at 5:10 pm

        KDC said that Whaleoil had a huge percentage of visits from outside countries without name. Only 42% seemed to be NZ visitors. Do you think he artificially inflates his numbers. I think KDC wrote about bots whatever they are.

        • lprent 1.1.1.1
          7 September 2014 at 5:56 pm

          The problem is that KDC gave no information on where he got that data from. That makes it meaningless.

          I suspect that he is right. However it isn’t anywhere near proof that he is right. It is just a silly distraction.

  2. Draco T Bastard 2
    7 September 2014 at 4:51 pm

    While originally envisaged as a backup system for the cloud based servers at Amazon in Sydney, it has proved fast enough on the NZ network to provide a significiant drop in latency for our mainly NZ audience.

    The fibre connection is running at 100MBits/sec down and 50MBits/sec up, and has been easily able to handle the load including the spikes of 250 users per minute online after the launch of Dirty Politics.

    Which is why we need FttH across the country and, AFAIC, it needs to be synchronous and around 1gbps. It really would allow for a massive expansion of collaboration and thus innovation across the country.

    • lprent 2.1
      7 September 2014 at 5:05 pm

      Yes, and I have to say that Steven Joyce has certainly muffed it badly.

      Virtually everyone I have talked to (businesses and individuals) has had major hassles getting fibre in because of his really terrible decisions.

      Amy Adams has done a far better job on getting the detail in place that makes it possible to get connected more easily. But that took 5 years to do.

      • Lanthanide 2.1.1
        7 September 2014 at 6:44 pm

        Amy Adams seems like a very competent up-and-comer to me. Good to have at least one woman in the cabinet who is for more than just show.

  3. ianmac 3
    7 September 2014 at 5:05 pm

    Those of us who wake up to read the Standard every day and every evening, salute you Lynn. I hope we don’t take your commitment for granted just because it is reliable and omnipresent. Thanks.

    • lprent 3.1
      7 September 2014 at 5:11 pm

      Oh it is definitely not omnipresent. After all I have to go to work each week day and sometimes during the weekends. That just mens that I lug a cellphone, tablet, and laptop where ever I go.

      Mostly what happens is that the main timewasters (ie trolls) tend to find out that I don’t like having my time wasted and tend to adjust their behaviour over time. I don’t care particularly about what people say, but I do care about peoples behaviour – and I force it to be adjusted within our purview. A bit of primitively effective social engineering.

  4. Hami Shearlie 4
    7 September 2014 at 5:47 pm

    We are all very grateful to you LPRENT!!!

  5. CnrJoe 5
    7 September 2014 at 6:04 pm

    Muchos gracias – where would we be…?

  6. Tracey 6
    7 September 2014 at 6:19 pm

    yay you mr prentice

  7. Ad 7
    7 September 2014 at 6:25 pm

    I have no idea what you said except “shutdown tonight”, but if you ever wanted a fundraiser event, let me know and I will happily do one for you.

    Nice big fat dinner, so we can invite all the nom de plumes who want to.

    Great to see the traffic going though the roof LPrent.

    • lprent 7.1
      7 September 2014 at 6:36 pm

      Cost last month was $185 for UFB and $166 for Amazon AWS. It dropped from about $660 the previous month. Should be down further this month as I eliminated a excessive search bot issue from Baidu spider.

    • lprent 7.2
      7 September 2014 at 7:35 pm

      I’d have to say that the AWS (amazon web services) was great until the site traffic went through the roof. It peaked at just under 900GB for a month.

      Then it started to cost as much as the site that I’d shifted from where it was running at 250GB and the 120GB of overseas traffic was the cost. The issue was that it was bloody hard to get a good remote analysis of what was going wrong apart from it being a bot problem. I found it within days after I shifted it back here. It was a problem on the site that the baidu spider was too dumb to “see” so it kept fetching the same page under a different URL.

      We actually get more enough from the minimal advertising we put on site provided that the advertising money turns up. But my MBA was in operations and I come from a family that tends to do production and operations. I just can’t resist figuring out how to do more with a lot less.

      This last round of productivity/cost means that I could do an enormous improvement in local performance by siting in the NZ network where 89% of our readers were from. I now use AWS for the graphics, server routing, and backup/warm server cluster.

      I’ve just been looking at the last weeks traffic stats. The overseas traffic is still way too high. But from the pattern it is generally the requests that is over the top. That is what I am working on now. I’m going through the site removing requests for small graphics and either making it CSS or out of a consolidated graphic.

  8. yeshe 8
    7 September 2014 at 7:05 pm

    You are pure gold, Lprent .. thank you 🙂

  9. philj 9
    7 September 2014 at 7:18 pm

    A true public service. Thank you.

  10. Harry Holland 10
    7 September 2014 at 9:02 pm

    “The Standard is the sum of the loosely cooperative authors and their mixture of different viewpoints, and the commenters who have fun analyzing everything to death.”

    Very like the old days of the greenviews mailing list back in the 90’s, hosted by Sigi Kirchmair, but eventually shut down as GPANZ sought to take things in-house and exert control and security on contentious discussion.

    Not being ‘owned’ by any political party is a great strength of The Standard.

  11. infused 11
    7 September 2014 at 10:17 pm

    Zyxel router. Poor you. You should change that piece of shit out straight away. If you are on fibre, put mikrotik/cisco/somethinghalfdecent in.

    That zyxel won’t even push 100/100 with the firewall on.

    • lprent 11.1
      7 September 2014 at 11:26 pm

      Never had a problem with a Zyxel. However I have had a Cisco high end fail. And I avoid Cisco’s linksys like the plague as I have never had one reach the end of its life with me. Basically I regard the company as crap.

      However, I’m only on 100/50 and the testing shows 94/48 depending on what is on the other end. Good enough for the price I’m paying for it.

      • infused 11.1.1
        8 September 2014 at 12:07 am

        The Zyxel firmware is as buggy as it comes. I sold these things for like 5 years before so much failure made me go elsewhere.

        Since you like fiddling with things, look at something like this.

        http://routerboard.com/RB2011UiAS-2HnD-IN

        Would be good for your custom rules, like banning bots etc. Easy to make some honeypots using that to catch a lot of crap that comes in. I run them everywhere now.

        • lprent 11.1.1.1
          8 September 2014 at 1:54 am

          I usually don’t trap them in device firewalls. All of them are not fast, they aren’t smart, aren’t dynamic, and all have useless interfaces.

          I don’t expect a router to be smart and I find that the simplier I configure them, the less aggravation I have.

          I pinhole to a gateway server handling the service(s). These days that runs (on linux) iptable or ufw rules set up by one of many tools that writes those rules. Some of those are pretty smart and all of the tools have a lot of CPU running open source code that is debugged and widely known.

          It sounds to me like you were expecting too much of a simple device. After writing embedded code for large chunks of the last 8 years I know from the inside how limited their processors are. I usually use them well within design limits. They aren’t PCs or servers (which I have been writing for over the last 30 years). Their processors tend to be more like a 8051 than a 80286.

        • lprent 11.1.1.2
          8 September 2014 at 2:05 am

          Your router board looks interesting if I ran windows servers on the net. But these days I don’t. The ones at work that are outward facing are run by the IT department. The ones that I work on are on isolated subnets, and I’m interested in their application code rather than their security.

          • infused 11.1.1.2.1
            8 September 2014 at 9:00 am

            They are not windows based fyi. RouterOS is based on linux. There is a Windows winbox program, but also ssh, telnet and http ways of configuring them.

            I probably do expect too much from them. After all SPI is the thing these days. Your static firewall is dead.

            Trapping them on the router is far more efficient use of the routers resources instead of offloading it on to the servers etc.

            You should grab a $50 one just to play 😛

  12. sophie 12
    8 September 2014 at 8:31 am

    I salute Lyn for putting up with that monstrosity in what looks like the corner of the kitchen!

    Thanks Lyn and, of course, thanks LPrent.

  13. yeshe 13
    8 September 2014 at 9:14 am

    So much faster this morning — what a delight … thx again Lynn.

  14. veutoviper 14
    8 September 2014 at 9:47 am

    My thanks also for all your hard work and dedication to keeping TS available and well maintained, Lprent.

  15. raa 15
    8 September 2014 at 10:08 am

    “It is far closer to the bulletin boards and usenet that I played on when I was a somewhat younger person in my 20s and 30s.”

    It still has that feel .. long may it last – but
    depending only on one person may limit
    its longevity.

Links to post

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.