web analytics
The Standard
Advertising

Back to work time

Written By: - Date published: 11:55 pm, January 9th, 2013 - 23 comments
Categories: admin, infrastructure, internet, The Standard - Tags:

Thought I’d share this portrait of my holidays….

xmas_and_new_year

This is why we’re a bit slack about posts of the Xmas-New year period. At various times over the last five end of years we’ve tried having lot of posts, few posts, and at the end of 2011 we mostly had OpenMike (and the least drop in visitor numbers – looks like people like making their own fun). The pattern always looks much the same.

Page views start falling through December with visit numbers dropping a lot less (people read pages fewer times), then abruptly drop just before the 25th. The 25th is invariably the worst day of the year. We get a spike a few days after Christmas day as people realize there is nothing much to do. But then they sink back into a summer turpitude until work is inflicted upon them again (ie about now) when our page views start rising rapidly. Then it becomes worthwhile writing posts again.

Nice for me. It means that there is one part of the year that I can safely move servers without doing the allnighters.

The primary server has been moved out into the offshore cloud far away from the expensive costs of having to pay for the costs of the Southern Cross cable monopoly. The ~5% of our traffic from human readers offshore wasn’t a problem. However the searchbots (most recently the bloody bingbot) and spambots causing us to exceed our “free” datacap on overseas traffic. This was $3/GB when we started on the old server and is now $1/GB.

However it has cost us thousands of dollars in variable per GB charges since I moved the server back to NZ in 2011. Does nasty things to the site’s operating budget. For instance in November it cost $103 extra because the Bing searchbot tried to read the entire site including every comment link. They nearly did a million page views in November and 600k in December. The really irritating thing is that we get very few incoming searches from Bing. I’ve now constrained Bing to a more limited diet.

But now the server is back offshore, I can budget without those dratted gouging variable costs from the Southern Cross cable monopoly. Because of the cloudflare content delivery network I don’t need dedicated servers any more. We were heading up to the main server straining on 30-40% of the available processing power prior to using cloudflare. These days the same server seldom uses more than 10%. So in doing the shift we sould drop the size of server required and the base cost of running a server has dropped dramatically. And that is despite the ever increasing numbers of visitors, comments, and page views over the last two years.

So my current limit is set at 800GB per month for all traffic and I’ll review if I need to lift that later this month.

23 comments on “Back to work time”

  1. xtasy 1

    Hey Lprent – I already feel very happy with the “cloud’ outsourcing, as it is a bit like “cloud nine” to me.

    Maybe the NZ SIS and others will find it a bit more difficult to track and trace us in future, that is a very optimistic kind of “hope” of course.

    Otherwise saving costs makes sense, and cloud server use is now the norm, so we move with them times.

    Thanks for all!

  2. lprent 2

    Ok – off to bed. Caching appears to be better tuned.

  3. Bob Simmons 3

    The difference between say, Fisher and Paykell moving overseas to save costs (which you lefties were pissing yourselves over) and you moving your website overseas to save cost is?

    Both make good business sense, yet one is worse then the other?

    I smell hypocrite.

    • Colonial Viper 3.1

      Shit Bob you must think you are hot stuff.

      In the first case, hundreds of skilled NZers lost jobs, the country lost significant manufacturing capability and ownership of the company and its IP was transferred to the Chinese, to enrich wealthy shareholders.

      In the second case, The Standard stopped paying monopoly rents on the Southern Cross Cable to overseas corporates and overseas shareholders.

      • MrV 3.1.1

        More convenient bullshit from the human colonic viper.
        Fact is you are directly leading to lost internet hosting activity etc in NZ. But you fail to see the hypocrisy here

        Fact:So you’d rather have your NZ visitors pay those “monopoly rents on the Southern Cross Cable to overseas corporates and overseas shareholders.” – and don’t forget to ad the bogeyman to your list. Boo.
        Fact:If you hosted in NZ most traffic is NZ based so wouldn’t be routed onto the cable.
        Fact:You can configure your site so external crawlers are limited traffic-wise.

    • lprent 3.2

      I thought that I made it pretty clear in the post that the main problem was paying for overseas traffic that we have little control or use for. Why should we pay for the overseas parasites? They aren’t our readership who are 95% on the local network. Why should we pay that money to a gouging near monopoly effectively owned by offshore interests? It isn’t like they’re investing in NZ because that would involve their destroying their own monopoly that sucks the life out of local businesses trying to develop locally.

      Whereas the rationale for F&P to move their production offshore was because they were largely selling into a global market where the freight for both purchases for the Bill of Materials (BOM) and the freight for market was a large part of the basic cost. The labour savings just aren’t that much of a cost in the BOM and wasn’t the main reason for the move. They kept the development, R&D, and much of the marketing here and communicated via the net with the production closer to suppliers and markets. In other words they kept the high paying intellectual property and jobs here. That is exactly what NZ wants and needs. We don’t want to be a low wage economy.

      I have no argument that there was a business logic to the F&P production move decision. I have no idea why you think I would – but I suspect it has to do with one of those rather stupid myths that some of the right like to cling to. I suspect what you smell is the rectal stench of your own bigotry.

      The actual argument here was that the National government is completely crap at fostering businesses and jobs here. But I guess taht was somewhat too subtle for your observational abilities. That the government who should be managing our structural infrastructure for businesses allows a profit gouging monopoly here like the Southern Cross cable is just plain daft. Effectively they should be trying to fix that market distortion so that businesses (like the export businesses I work for) don’t automatically move their servers offshore.

      As bandwidth requirements for running offshore sales and production operations increase, it will become more and more of a pain to maintain businesses here. If NZ businesses working largely offshore continue to have the life sucked out of them by the parasites then they will move both the server systems and eventually development offshore, taking those high paying jobs away.

      BTW: Unlimited bandwidth inside NZ was paid by the base charge on the server. However over the last two years when the server was back in NZ we wound up paying nearly as much for the overseas traffic as we did for all of the traffic inside the country. More inmportantly it was unscheduled and largely uncontrollable costs for something that we didn’t want nor need. Paradoxically moving to the other side of the Southern Cross both increases the costs to our actual readers as they will have increased charges on their ISP bills, while decreasing our site bills. It will actually increase costs to NZ consumers of our site.

      But in neither case are the bills in proportion to either the costs of running the Southern Cross nor to pay for new cables that will be required in the medium term (not that they’re doing that anyway). It is simple profit gouging.

      • Steve H 3.2.1

        L,

        The reason that they couldn’t get investors for a new cable was that it isn’t needed.

        It is no-where near capacity, and there are upgrades happening to it as we speak. And can you name me a company which charges less and less for its services every year, like SCCS does?

        The reason nobody invested was because it doesn’t make sense to an investor – build something which will lose you money long term providing a service which is dirt cheap anyway?

        • lprent 3.2.1.1

          That is the problem. It isn’t dirt-cheap to people providing content online. If it was then we wouldn’t be having this discussion.

          Dirt cheap is what the local network costs. The Southern Cross was and still is outrageously expensive by comparison.

          This site does between 400Gb and 700Gb per month if I didn’t put constraints on it. The majority of that traffic is inside NZ. But typically between 150GB and 250GB of that traffic comes from offshore. Most of that is from searchbots, most of the remainder is from spambots, and a small fraction is from humans.

          But it means that if I had 250GB of traffic from overseas – drop the 25GB of “free” limit from the hosting provider, and say 25GB from the actual overseas readers. Then at $1/GB we’re coughing up $200 for traffic that isn’t readers. The server cost was only $233/mo. And of course to make it worse we’re paying 15% tax on that to a lazy government.

          The pricing structures of the Southern Cross distort the market.Now some of that we want – searchbots for instance. While they could operate their searchbots on the local network, they don’t? Why? Because they don’t get charged for the southern cross – they force the cost on to us.

          Now look at this the other way. We just moved our servers offshore so we don’t get penalized for running a server in NZ by the Southern Cross. This means that literally hundreds of GB’s more traffic has to go over the Southern Cross from our servers into NZ. Ok so someone has to pay for that. It goes on the bills of the readers. Our move offshore has probably added slightly to the already very high ISP costs of local users.

          Meanwhile our local servers industry dies because there are market distortions

          BTW: It is pretty noticeable that the only time that the costs actually reduce on the Southern Cross is when people start talking about putting in more cables

        • mike e vipe e 3.2.1.2

          S#*t Head what utter trash monopolies charge what their research shows the market will pay`!
          One of the main reasons why the morgan push for another cable failed was that the charges the US government wanted to charge to let it come on shore in the US made it enviable they had vested interests in keeping the existing southern cross cable as a monopoly one being they want us to start paying for content on the internet ie Rupert Murdoch and mates!

          • Steve H 3.2.1.2.1

            Thats silly, Mike – why would the US government not want to make money off 2 providers?

            Do some math, 2 is bigger than 1.

            @ lprent, why not ban bing then? As you say, nobody uses it to search your site and then you could have save some cash.

            It sounds like you don’t want to accept that on one hand you complain about ISP’s charging too much, but on the other hand you take advantage of user pays, causing cost to ISP’s

            • lprent 3.2.1.2.1.1

              I’d love to. I even went so far as to find out all of their current searchbot IP’s ranges.

              However when I compared it to the actual ranges of bingbot IP’s requesting data from our site, I found that the list was quite incomplete (and I verified that they were in fact real bingbots). A quick hunt around google showed that everyone has the same problem.

              This means that I can’t kill at the firewall by refusing a connection. I have to ban using the agent at apache. But by the time it gets that far into the server it becomes a bit of a waste of time. I’m still paying for the request data and the CPU time to throw them off.

              Eventually I just went into the bing webmaster tools and got them to limit themselves.

              Not to mention that they are merely the worst of the current crop of search bots from slurp to baidu to google. And the innumerable spambots where there are no civilising tools. It would be a never-ending task. Strategically it is smarter to bypass the problem. I’d change cable suppliers, but there isn’t any other choice apart from satellite latency. So I’ve done the next best thing.

              This isn’t exactly a paid job and I don’t have time to fight bots on behalf of the Southern Cross cable. It is easier to remove our site from the costs. It is unfortunate that Southern Cross will wind up penalizing our readers instead, but that is less of a problem to me than wasting my time.

    • Mary 3.3

      Your comment is as silly and non-sensical as something Cameron Slater would say.

      • Steve H 3.3.1

        HOw is his comment nonsense?

        You guys are all for “keeping jobs and everything in new zealand to be paid for by everyone but the end user”, then your dear blog owner goes and makes it user-pays, to save some cash

        He’s just as bad as a corporate, but I don’t see you guys getting all angry about him doing so.

        Keep The Standard in New Zealand!

        [lprent: So you’re going to contribute. Good to hear – so how much? And when can I expect to see it? ]

        • Colonial Viper 3.3.1.1

          To repeat my earlier comments for the Right Wingnuts

          http://thestandard.org.nz/back-to-work-time/#comment-571732

          In the first case (Fisher & Paykel), hundreds of skilled NZers lost jobs, the country lost significant manufacturing capability and ownership of the company and its IP was transferred to the Chinese, to enrich wealthy shareholders.

          In the second case, The Standard stopped paying monopoly rents on the Southern Cross Cable to overseas corporates and overseas shareholders.

          • Steve H 3.3.1.1.1

            So you support competition then? Odd, I’m sure in some of your other posts you write are for supporting government owned monopoly’s on things like power.

            Government owned Air NZ was great, wasn’t it?

            • McFlock 3.3.1.1.1.1

              Are you really that incapable of understanding that a person’s opinions on different things can vary according to the characteristics of each different thing, yet still be part of a single coherent and consistent world-view?

            • ropata 3.3.1.1.1.2

              err, Air NZ went bust under the “guidance” of Brierly and their mates in the NZX. Don’t you recall the Labour govt bailout of $1 billion? Air NZ is now 80% govt owned and doing much better with responsible leadership.

              • Colonial Viper

                So you support competition then? Odd, I’m sure in some of your other posts you write are for supporting government owned monopoly’s on things like power.

                Critical infrastructure and services must be government owned. Monopolies like that need to be kept out of the hands of the profiteering private sector.

    • tc 3.4

      F&P was sold Bob try and keep up. The standard has reduced it’s costs so that it can continue to deliver on it’s charter and play a role in NZ society….spot the difference, use crayons if it helps.

  4. QoT 4

    Bing = the devil.

    I seriously hate Bing, primarily because if I encounter it, it means I’m being forced to use a computer which doesn’t have Firefox/Chrome.

    • lprent 4.1

      I have never seen bing itself. Just their bots rampaging and the webmaster tools. Coming to think of it I haven’t used Windows 7, apart from booting up on it long enough to download and burn an ubuntu disk at work. There is half of a terabyte disk with windows there that I never use. And I forgot the damn password.

      There is a vista partition on my laptop, that I will be putting windows 8 in the next few weeks. I want to do some testing of some cross compiling using MinGW/gcc to make sure I know what I have to check.

  5. SHG (not Colonial Viper) 5

    searchbots (most recently the bloody bingbot) and spambots causing us to exceed our “free” datacap on overseas traffic

    robots.txt, learn it, love it.

    • lprent 5.1

      Of course that is set automatically, and is in fact a virtual file. The problem is that each individual bingbot bot seems to follow it individually (if at all). And of course most of the bots simply ignore it.

Links to post

Important links

Recent Comments

Recent Posts

1 2 3 7

  • Housing Accord not working – prices continue skyward
      The Government's Auckland Housing Accord isn't working as house prices continue to go through the roof, Labour's Housing spokesperson Phil Twyford says. “The average Auckland house has gone up by $110,000 since the Accord came into effect 15 months… ...
    2 hours ago
  • Justice for Teina Pora long overdue
    The Privy Council’s decision to quash Teina Pora’s convictions for the rape and murder of Susan Burdett could be the final chapter in a case that should have been closed years ago, Labour’s Justice Spokesperson Jacinda Ardern says. “Teina Pora… ...
    6 hours ago
  • Ministers must answer questions on IRD blowout
    The current and previous Revenue Ministers must front up and explain how the child support system had a budget blowout from $30 million to $210 million in just four years, says Labour’s Revenue spokesperson Clayton Cosgrove. “Peter Dunne was Revenue… ...
    1 day ago
  • Curb stratospheric public CEO salaries
    A review of the way MPs’ pay is set should also look at ways to curb excessive rises in the salaries of public service chief executives, Labour Leader Andrew Little says. “Some of these CEOs have had stratospheric pay increases… ...
    1 day ago
  • 50 cents? Makes no sense.
    The minimum wage rose by 50 cents this month from 14.25 to 14.75. While it’s a small step towards ensuring minimum workers get a fair share, it’s important to remember that real wages only rose 1.5% while productivity rose by… ...
    GreensBy Denise Roche MP
    2 days ago
  • The Serco corrections circus
    It should seem obvious to employers, private or public, that it’s important to do what you can to retain your best, most experienced staff. They make life easier for you because they’re effective, attentive and often respected by those around… ...
    GreensBy David Clendon MP
    2 days ago
  • Time for NZ to prohibit the killing of great apes
    That ban was widely hailed, and spurred efforts in other countries to get similar bans. However, apes are still being exploited, abused and killed, both in captivity and in the wild. Examples of cruelty, neglect and abuse abound. Apes are… ...
    GreensBy Mojo Mathers MP
    5 days ago
  • Auckland building consents: Tragic
    The only word to describe the latest building consent figures for Auckland is ‘tragic’, Labour’s Housing spokesperson Phil Twyford says. “Whatever the Government is doing to address the Auckland housing crisis, it is clearly not working. ...
    5 days ago
  • A whiff of a new biosecurity scandal?
    A pest which could create havoc for New Zealand’s horticulture and agriculture sector must be as much a focus for the Government as hunting out fruit flies, Labour’s Biosecurity spokesperson Damien O’Connor says. “While the Ministry for Primary Industries is… ...
    5 days ago
  • Government shrugs off health sector crisis
    Despite new evidence showing that cuts to health spending are costing lives the Government continues to deny the sector is struggling, Labour’s Health spokesperson Annette King says. “Health services in New Zealand are in crisis. ...
    6 days ago
  • Parata lowered the bar for failing charter school
    When Hekia Parata became aware that the Whangaruru charter school was experiencing major problems her first action was to drop standards by reducing the number of qualified teachers they had to employ, Labour’s Education spokesperson Chris Hipkins has revealed. “Hekia… ...
    6 days ago
  • National not being straight about the economy
    John Key and Bill English need to be straight with New Zealanders about the damage their failure to diversify the economy is doing, after new figures show export growth plunged due to a collapse in dairy exports, says Grant Robertson.… ...
    6 days ago
  • Mind the Gap
    This week the International Monetary Fund released a report on the wider economic value in closing the gender pay gap. When even the bastions of free-market economics start to raise concerns about gender pay gaps, we have to realise how… ...
    GreensBy Jan Logie MP
    6 days ago
  • Labour will hold National to parental leave promise
    Labour will hold National to its promise to increase the support given to new parents of premature, multiple birth and babies born with disabilities, Labour’s paid parental leave campaigner Sue Moroney says. "I am naturally disappointed that after battling for… ...
    7 days ago
  • It was all just pillar talk
    Steven Joyce’s confession that he can no longer guarantee a pillar-free design for the New Zealand International Convention Centre shows the Government has abandoned its dream of creating an ‘iconic’ ‘world-class’ structure, says Labour Economic Development spokesperson David Clark. “Steven… ...
    1 week ago
  • Australians move on offshore speculators
    John Key might want to have a quiet word with Australian Prime Minister Tony Abbott about Canberra's just-announced crack down on offshore speculators when he visits New Zealand this week, Labour's Housing spokesperson Phil Twyford says."Tony Abbott's centre right government… ...
    1 week ago
  • Government at odds on overseas driver crashes
    National backbencher Jacqui Dean has spoken out about overseas driver crashes, putting herself at odds with Prime Minister John Key who is on record as saying it’s not a big issue, Labour’s Transport spokesperson Phil Twyford says. “I’m not surprised… ...
    1 week ago
  • Human Rights and the Palestine Crisis
    Last week I heard two Palestinians speak at Wellington events about the ongoing crisis in their country. Samar Sabawi spoke to a full house about the history of Palestine and gave us a lucid and disturbing account of the situation… ...
    GreensBy Catherine Delahunty MP
    1 week ago
  • Time to take real care of our kids
    An Amnesty International report has once again criticised New Zealand’s track record on looking after our kids, Labour’s Children’s spokesperson Jacinda Ardern says. The annual report, which looks at global human rights abuses highlights not only the fact that high… ...
    1 week ago
  • John Key wrong about Labour’s war vote
    John Key’s desperate claims that the former Labour Government didn’t put combat troop deployment to a Parliamentary vote are simply wrong, Labour’s Defence spokesperson Phil Goff says. “It was disgraceful that the Prime Minister ran rough shod over democracy and… ...
    1 week ago
  • Manus Island and the New Zealand Government
    This week the Greens have participated in awareness activity about Manus Island, the refugee camp on an island in Papua New Guinea where Australia dumps asylum seekers. John Key says that he has every confidence in the Australian Government’s claim… ...
    GreensBy Catherine Delahunty MP
    2 weeks ago
  • Election Inquiry – Getting accessible voting on the agenda
    James Shaw has been doing a series of blogs on the Election Inquiry into last year’s general election.  I thought this was a great opportunity to raise an issue very dear to me – accessible voting. Last year’s general election… ...
    GreensBy Mojo Mathers MP
    2 weeks ago
  • RMA changes no solution to Christchurch housing
    Housing will continue to be a big issue in 2015. The latest Consumer Price Index, released last month, shows both good news and bad news on the housing front. After years of being the most expensive place to build a… ...
    GreensBy Eugenie Sage MP
    2 weeks ago
  • Saving kokako in South Auckland’s Hunua Ranges
    It is amazing that you can hear the song of the endangered North Island kokako in South Auckland’s Hunua Ranges, less than 50 kms from the central city. A heavy schedule of policy workshops at the Green Party’s Policy… ...
    GreensBy Eugenie Sage MP
    2 weeks ago
  • Let’s not turn a blind eye to human rights
    The Cricket World Cup has just opened in New Zealand, and it’s an opportunity for us to shine on the world stage. International sport can be a chance for us to build relationships with other countries, and examine what it… ...
    GreensBy Catherine Delahunty MP
    2 weeks ago
  • Its Just Not Cricket
    This week it was my privilege to work with Sri Lankan Tamil communities in this country and host Australian journalist and human rights advocate Trevor Grant. I knew a bit about Trevor from his biography but I didn’t know just… ...
    GreensBy Catherine Delahunty MP
    3 weeks ago
  • Time for NZ to #BeCrueltyFree
    The Government is about to progress the final stages of the Animal Welfare Amendment bill. This will be our last opportunity to get changes made to improve the bill to ensure a better outcome for animals. I have put forwards… ...
    GreensBy Mojo Mathers MP
    3 weeks ago
  • We want access!
    Access to buildings is a big issue for many New Zealanders. It looks like that, due to the hard work and persistence of people in the disability community, the Government may finally be starting to take access to buildings seriously.… ...
    GreensBy Mojo Mathers MP
    3 weeks ago
  • Greens call on Super Fund to divest from fossil fuels
    The Green Party today called on the New Zealand Superannuation Fund (the Fund) to divest from fossil fuels, starting immediately with coal. The call was accompanied with a new report, Making money from a climate catastrophe: The case for divesting… ...
    GreensBy Russel Norman MP
    3 weeks ago

Removed at the request of The Daily Blog.
Public service advertisements by The Standard

Current CO2 level in the atmosphere