web analytics

Back to work time

Written By: - Date published: 11:55 pm, January 9th, 2013 - 24 comments
Categories: admin, infrastructure, internet, The Standard - Tags:

Thought I’d share this portrait of my holidays….

xmas_and_new_year

This is why we’re a bit slack about posts of the Xmas-New year period. At various times over the last five end of years we’ve tried having lot of posts, few posts, and at the end of 2011 we mostly had OpenMike (and the least drop in visitor numbers – looks like people like making their own fun). The pattern always looks much the same.

Page views start falling through December with visit numbers dropping a lot less (people read pages fewer times), then abruptly drop just before the 25th. The 25th is invariably the worst day of the year. We get a spike a few days after Christmas day as people realize there is nothing much to do. But then they sink back into a summer turpitude until work is inflicted upon them again (ie about now) when our page views start rising rapidly. Then it becomes worthwhile writing posts again.

Nice for me. It means that there is one part of the year that I can safely move servers without doing the allnighters.

The primary server has been moved out into the offshore cloud far away from the expensive costs of having to pay for the costs of the Southern Cross cable monopoly. The ~5% of our traffic from human readers offshore wasn’t a problem. However the searchbots (most recently the bloody bingbot) and spambots causing us to exceed our “free” datacap on overseas traffic. This was $3/GB when we started on the old server and is now $1/GB.

However it has cost us thousands of dollars in variable per GB charges since I moved the server back to NZ in 2011. Does nasty things to the site’s operating budget. For instance in November it cost $103 extra because the Bing searchbot tried to read the entire site including every comment link. They nearly did a million page views in November and 600k in December. The really irritating thing is that we get very few incoming searches from Bing. I’ve now constrained Bing to a more limited diet.

But now the server is back offshore, I can budget without those dratted gouging variable costs from the Southern Cross cable monopoly. Because of the cloudflare content delivery network I don’t need dedicated servers any more. We were heading up to the main server straining on 30-40% of the available processing power prior to using cloudflare. These days the same server seldom uses more than 10%. So in doing the shift we sould drop the size of server required and the base cost of running a server has dropped dramatically. And that is despite the ever increasing numbers of visitors, comments, and page views over the last two years.

So my current limit is set at 800GB per month for all traffic and I’ll review if I need to lift that later this month.

24 comments on “Back to work time ”

  1. xtasy 1

    Hey Lprent – I already feel very happy with the “cloud’ outsourcing, as it is a bit like “cloud nine” to me.

    Maybe the NZ SIS and others will find it a bit more difficult to track and trace us in future, that is a very optimistic kind of “hope” of course.

    Otherwise saving costs makes sense, and cloud server use is now the norm, so we move with them times.

    Thanks for all!

  2. lprent 2

    Ok – off to bed. Caching appears to be better tuned.

  3. Bob Simmons 3

    The difference between say, Fisher and Paykell moving overseas to save costs (which you lefties were pissing yourselves over) and you moving your website overseas to save cost is?

    Both make good business sense, yet one is worse then the other?

    I smell hypocrite.

    • Colonial Viper 3.1

      Shit Bob you must think you are hot stuff.

      In the first case, hundreds of skilled NZers lost jobs, the country lost significant manufacturing capability and ownership of the company and its IP was transferred to the Chinese, to enrich wealthy shareholders.

      In the second case, The Standard stopped paying monopoly rents on the Southern Cross Cable to overseas corporates and overseas shareholders.

      • MrV 3.1.1

        More convenient bullshit from the human colonic viper.
        Fact is you are directly leading to lost internet hosting activity etc in NZ. But you fail to see the hypocrisy here

        Fact:So you’d rather have your NZ visitors pay those “monopoly rents on the Southern Cross Cable to overseas corporates and overseas shareholders.” – and don’t forget to ad the bogeyman to your list. Boo.
        Fact:If you hosted in NZ most traffic is NZ based so wouldn’t be routed onto the cable.
        Fact:You can configure your site so external crawlers are limited traffic-wise.

    • lprent 3.2

      I thought that I made it pretty clear in the post that the main problem was paying for overseas traffic that we have little control or use for. Why should we pay for the overseas parasites? They aren’t our readership who are 95% on the local network. Why should we pay that money to a gouging near monopoly effectively owned by offshore interests? It isn’t like they’re investing in NZ because that would involve their destroying their own monopoly that sucks the life out of local businesses trying to develop locally.

      Whereas the rationale for F&P to move their production offshore was because they were largely selling into a global market where the freight for both purchases for the Bill of Materials (BOM) and the freight for market was a large part of the basic cost. The labour savings just aren’t that much of a cost in the BOM and wasn’t the main reason for the move. They kept the development, R&D, and much of the marketing here and communicated via the net with the production closer to suppliers and markets. In other words they kept the high paying intellectual property and jobs here. That is exactly what NZ wants and needs. We don’t want to be a low wage economy.

      I have no argument that there was a business logic to the F&P production move decision. I have no idea why you think I would – but I suspect it has to do with one of those rather stupid myths that some of the right like to cling to. I suspect what you smell is the rectal stench of your own bigotry.

      The actual argument here was that the National government is completely crap at fostering businesses and jobs here. But I guess taht was somewhat too subtle for your observational abilities. That the government who should be managing our structural infrastructure for businesses allows a profit gouging monopoly here like the Southern Cross cable is just plain daft. Effectively they should be trying to fix that market distortion so that businesses (like the export businesses I work for) don’t automatically move their servers offshore.

      As bandwidth requirements for running offshore sales and production operations increase, it will become more and more of a pain to maintain businesses here. If NZ businesses working largely offshore continue to have the life sucked out of them by the parasites then they will move both the server systems and eventually development offshore, taking those high paying jobs away.

      BTW: Unlimited bandwidth inside NZ was paid by the base charge on the server. However over the last two years when the server was back in NZ we wound up paying nearly as much for the overseas traffic as we did for all of the traffic inside the country. More inmportantly it was unscheduled and largely uncontrollable costs for something that we didn’t want nor need. Paradoxically moving to the other side of the Southern Cross both increases the costs to our actual readers as they will have increased charges on their ISP bills, while decreasing our site bills. It will actually increase costs to NZ consumers of our site.

      But in neither case are the bills in proportion to either the costs of running the Southern Cross nor to pay for new cables that will be required in the medium term (not that they’re doing that anyway). It is simple profit gouging.

      • Steve H 3.2.1

        L,

        The reason that they couldn’t get investors for a new cable was that it isn’t needed.

        It is no-where near capacity, and there are upgrades happening to it as we speak. And can you name me a company which charges less and less for its services every year, like SCCS does?

        The reason nobody invested was because it doesn’t make sense to an investor – build something which will lose you money long term providing a service which is dirt cheap anyway?

        • lprent 3.2.1.1

          That is the problem. It isn’t dirt-cheap to people providing content online. If it was then we wouldn’t be having this discussion.

          Dirt cheap is what the local network costs. The Southern Cross was and still is outrageously expensive by comparison.

          This site does between 400Gb and 700Gb per month if I didn’t put constraints on it. The majority of that traffic is inside NZ. But typically between 150GB and 250GB of that traffic comes from offshore. Most of that is from searchbots, most of the remainder is from spambots, and a small fraction is from humans.

          But it means that if I had 250GB of traffic from overseas – drop the 25GB of “free” limit from the hosting provider, and say 25GB from the actual overseas readers. Then at $1/GB we’re coughing up $200 for traffic that isn’t readers. The server cost was only $233/mo. And of course to make it worse we’re paying 15% tax on that to a lazy government.

          The pricing structures of the Southern Cross distort the market.Now some of that we want – searchbots for instance. While they could operate their searchbots on the local network, they don’t? Why? Because they don’t get charged for the southern cross – they force the cost on to us.

          Now look at this the other way. We just moved our servers offshore so we don’t get penalized for running a server in NZ by the Southern Cross. This means that literally hundreds of GB’s more traffic has to go over the Southern Cross from our servers into NZ. Ok so someone has to pay for that. It goes on the bills of the readers. Our move offshore has probably added slightly to the already very high ISP costs of local users.

          Meanwhile our local servers industry dies because there are market distortions

          BTW: It is pretty noticeable that the only time that the costs actually reduce on the Southern Cross is when people start talking about putting in more cables

        • mike e vipe e 3.2.1.2

          S#*t Head what utter trash monopolies charge what their research shows the market will pay`!
          One of the main reasons why the morgan push for another cable failed was that the charges the US government wanted to charge to let it come on shore in the US made it enviable they had vested interests in keeping the existing southern cross cable as a monopoly one being they want us to start paying for content on the internet ie Rupert Murdoch and mates!

          • Steve H 3.2.1.2.1

            Thats silly, Mike – why would the US government not want to make money off 2 providers?

            Do some math, 2 is bigger than 1.

            @ lprent, why not ban bing then? As you say, nobody uses it to search your site and then you could have save some cash.

            It sounds like you don’t want to accept that on one hand you complain about ISP’s charging too much, but on the other hand you take advantage of user pays, causing cost to ISP’s

            • lprent 3.2.1.2.1.1

              I’d love to. I even went so far as to find out all of their current searchbot IP’s ranges.

              However when I compared it to the actual ranges of bingbot IP’s requesting data from our site, I found that the list was quite incomplete (and I verified that they were in fact real bingbots). A quick hunt around google showed that everyone has the same problem.

              This means that I can’t kill at the firewall by refusing a connection. I have to ban using the agent at apache. But by the time it gets that far into the server it becomes a bit of a waste of time. I’m still paying for the request data and the CPU time to throw them off.

              Eventually I just went into the bing webmaster tools and got them to limit themselves.

              Not to mention that they are merely the worst of the current crop of search bots from slurp to baidu to google. And the innumerable spambots where there are no civilising tools. It would be a never-ending task. Strategically it is smarter to bypass the problem. I’d change cable suppliers, but there isn’t any other choice apart from satellite latency. So I’ve done the next best thing.

              This isn’t exactly a paid job and I don’t have time to fight bots on behalf of the Southern Cross cable. It is easier to remove our site from the costs. It is unfortunate that Southern Cross will wind up penalizing our readers instead, but that is less of a problem to me than wasting my time.

    • Mary 3.3

      Your comment is as silly and non-sensical as something Cameron Slater would say.

      • Steve H 3.3.1

        HOw is his comment nonsense?

        You guys are all for “keeping jobs and everything in new zealand to be paid for by everyone but the end user”, then your dear blog owner goes and makes it user-pays, to save some cash

        He’s just as bad as a corporate, but I don’t see you guys getting all angry about him doing so.

        Keep The Standard in New Zealand!

        [lprent: So you’re going to contribute. Good to hear – so how much? And when can I expect to see it? ]

        • Colonial Viper 3.3.1.1

          To repeat my earlier comments for the Right Wingnuts

          Back to work time

          In the first case (Fisher & Paykel), hundreds of skilled NZers lost jobs, the country lost significant manufacturing capability and ownership of the company and its IP was transferred to the Chinese, to enrich wealthy shareholders.

          In the second case, The Standard stopped paying monopoly rents on the Southern Cross Cable to overseas corporates and overseas shareholders.

          • Steve H 3.3.1.1.1

            So you support competition then? Odd, I’m sure in some of your other posts you write are for supporting government owned monopoly’s on things like power.

            Government owned Air NZ was great, wasn’t it?

            • McFlock 3.3.1.1.1.1

              Are you really that incapable of understanding that a person’s opinions on different things can vary according to the characteristics of each different thing, yet still be part of a single coherent and consistent world-view?

            • ropata 3.3.1.1.1.2

              err, Air NZ went bust under the “guidance” of Brierly and their mates in the NZX. Don’t you recall the Labour govt bailout of $1 billion? Air NZ is now 80% govt owned and doing much better with responsible leadership.

              • Colonial Viper

                So you support competition then? Odd, I’m sure in some of your other posts you write are for supporting government owned monopoly’s on things like power.

                Critical infrastructure and services must be government owned. Monopolies like that need to be kept out of the hands of the profiteering private sector.

    • tc 3.4

      F&P was sold Bob try and keep up. The standard has reduced it’s costs so that it can continue to deliver on it’s charter and play a role in NZ society….spot the difference, use crayons if it helps.

  4. QoT 4

    Bing = the devil.

    I seriously hate Bing, primarily because if I encounter it, it means I’m being forced to use a computer which doesn’t have Firefox/Chrome.

    • lprent 4.1

      I have never seen bing itself. Just their bots rampaging and the webmaster tools. Coming to think of it I haven’t used Windows 7, apart from booting up on it long enough to download and burn an ubuntu disk at work. There is half of a terabyte disk with windows there that I never use. And I forgot the damn password.

      There is a vista partition on my laptop, that I will be putting windows 8 in the next few weeks. I want to do some testing of some cross compiling using MinGW/gcc to make sure I know what I have to check.

  5. SHG (not Colonial Viper) 5

    searchbots (most recently the bloody bingbot) and spambots causing us to exceed our “free” datacap on overseas traffic

    robots.txt, learn it, love it.

    • lprent 5.1

      Of course that is set automatically, and is in fact a virtual file. The problem is that each individual bingbot bot seems to follow it individually (if at all). And of course most of the bots simply ignore it.

Links to post

Recent Comments

Recent Posts

  • Twenty highlights of 2020
    As we welcome in the new year, our focus is on continuing to keep New Zealanders safe and moving forward with our economic recovery. There’s a lot to get on with, but before we say a final goodbye to 2020, here’s a quick look back at some of the milestones ...
    3 weeks ago

  • Jobs for Nature funding will create training and employment opportunities
    A major investment to tackle wilding pines in Mt Richmond will create jobs and help protect the area’s unique ecosystems, Biosecurity Minister Damien O’Connor says. The Mt Richmond Forest Park has unique ecosystems developed on mineral-rich geology, including taonga plant species found nowhere else in the country. “These special plant ...
    BeehiveBy beehive.govt.nz
    18 hours ago
  • Pre-departure testing extended to all passengers to New Zealand
    To further protect New Zealand from COVID-19, the Government is extending pre-departure testing to all passengers to New Zealand except from Australia, Antarctica and most Pacific Islands, COVID-19 Response Minister Chris Hipkins said today. “The change will come into force for all flights arriving in New Zealand after 11:59pm (NZT) on Monday ...
    BeehiveBy beehive.govt.nz
    24 hours ago
  • Bay Cadets learn skills to protect environment
    Bay Conservation Cadets launched with first intake Supported with $3.5 million grant Part of $1.245b Jobs for Nature programme to accelerate recover from Covid Cadets will learn skills to protect and enhance environment Environment Minister David Parker today welcomed the first intake of cadets at the launch of the Bay ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Cook Islanders to resume travel to New Zealand
    The Prime Minister of New Zealand Jacinda Ardern and the Prime Minister of the Cook Islands Mark Brown have announced passengers from the Cook Islands can resume quarantine-free travel into New Zealand from 21 January, enabling access to essential services such as health. “Following confirmation of the Cook Islands’ COVID ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Supporting communities and landowners to grow employment opportunities
    Jobs for Nature funding is being made available to conservation groups and landowners to employ staff and contractors in a move aimed at boosting local biodiversity-focused projects, Conservation Minister Kiritapu Allan has announced. It is estimated some 400-plus jobs will be created with employment opportunities in ecology, restoration, trapping, ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Border exception for some returning international tertiary students
    The Government has approved an exception class for 1000 international tertiary students, degree level and above, who began their study in New Zealand but were caught offshore when border restrictions began. The exception will allow students to return to New Zealand in stages from April 2021. “Our top priority continues ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Tiwai deal gives time for managed transition
    Today’s deal between Meridian and Rio Tinto for the Tiwai smelter to remain open another four years provides time for a managed transition for Southland. “The deal provides welcome certainty to the Southland community by protecting jobs and incomes as the region plans for the future. The Government is committed ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • New member for APEC Business Advisory Council
    Prime Minister Jacinda Ardern has appointed Anna Curzon to the APEC Business Advisory Council (ABAC). The leader of each APEC economy appoints three private sector representatives to ABAC. ABAC provides advice to leaders annually on business priorities. “ABAC helps ensure that APEC’s work programme is informed by business community perspectives ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Govt’s careful economic management recognised
    The Government’s prudent fiscal management and strong policy programme in the face of the COVID-19 global pandemic have been acknowledged by the credit rating agency Fitch. Fitch has today affirmed New Zealand’s local currency rating at AA+ with a stable outlook and foreign currency rating at AA with a positive ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Additional actions to keep COVID-19 out of NZ
    The Government is putting in place a suite of additional actions to protect New Zealand from COVID-19, including new emerging variants, COVID-19 Response Minister Chris Hipkins said today. “Given the high rates of infection in many countries and evidence of the global spread of more transmissible variants, it’s clear that ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • 19 projects will clean up and protect waterways
    $36 million of Government funding alongside councils and others for 19 projects Investment will clean up and protect waterways and create local jobs Boots on the ground expected in Q2 of 2021 Funding part of the Jobs for Nature policy package A package of 19 projects will help clean up ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • New Zealand Government acknowledges 175th anniversary of Battle of Ruapekapeka
    The commemoration of the 175th anniversary of the Battle of Ruapekapeka represents an opportunity for all New Zealanders to reflect on the role these conflicts have had in creating our modern nation, says Associate Minister for Arts, Culture and Heritage Kiri Allan. “The Battle at Te Ruapekapeka Pā, which took ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Better care for babies with tongue-tie
    Babies born with tongue-tie will be assessed and treated consistently under new guidelines released by the Ministry of Health, Associate Minister of Health Dr Ayesha Verrall announced today. Around 5% to 10% of babies are born with a tongue-tie, or ankyloglossia, in New Zealand each year. At least half can ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Prisoner disorder event at Waikeria Prison over
    The prisoner disorder event at Waikeria Prison is over, with all remaining prisoners now safely and securely detained, Corrections Minister Kelvin Davis says. The majority of those involved in the event are members of the Mongols and Comancheros. Five of the men are deportees from Australia, with three subject to ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Pre-departure COVID-19 test for travellers from the UK and the US from 15 January
    Travellers from the United Kingdom or the United States bound for New Zealand will be required to get a negative test result for COVID-19 before departing, and work is underway to extend the requirement to other long haul flights to New Zealand, COVID-19 Response Minister Chris Hipkins confirmed today. “The new PCR test requirement, foreshadowed last ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • PM congratulates New Year Honour recipients
    Prime Minister Jacinda Ardern has added her warm congratulations to the New Zealanders recognised for their contributions to their communities and the country in the New Year 2021 Honours List. “The past year has been one that few of us could have imagined. In spite of all the things that ...
    BeehiveBy beehive.govt.nz
    3 weeks ago
  • David Parker congratulates New Year 2021 Honours recipients
    Attorney-General and Minister for the Environment David Parker has congratulated two retired judges who have had their contributions to the country and their communities recognised in the New Year 2021 Honours list. The Hon Tony Randerson QC has been appointed a Companion of the New Zealand Order of Merit for ...
    BeehiveBy beehive.govt.nz
    3 weeks ago