Dealing with the DIA website-harvest parasite

Written By: - Date published: 2:27 am, February 3rd, 2011 - 13 comments
Categories: admin, blogs, humour, inoculation, interweb, scoundrels, Spying, The Standard - Tags: ,

Periodically I run a scan to identify network parasites that are sucking up our bandwidth and processing resources in excess. Of course I leave the benign parasites that provide search facilities alone. Like the ubiquitous googlebot and its other benign brethren. But I stomp on the nasties and inoculate the system against them.

It could be spambots trying to insert comments and requiring us to expend effort in cleaning up the garbage. There denying from .htaccess is your friend and vast swathes of the IP’s from Russia, China and some other regions are excluded from reading anything at all.

It could be virus slaved botnets mindlessly trying to exploit our contact us and contribute pages to send e-mails. There I find that a relatively simple captcha is sufficient to deter them.

Feedbots picking up RSS feeds used to be a problem. But since I put some limits on how much they can pick up and how old it can be, that problem has disappeared.

It could be a out of control spider trying to crawl all over our web pages in a single cataclysmic event. I find that putting restrictions on how often an IP number can connect within a given period of time tends to squelch them. Because of the nature of the other beasts on the net I have that to just harm the really out of control processes with grey-listing. The system starts refusing to accept new connections after a while while very slowwwly feeding the bot a dribble of data and holding them up.

But that leaves the processes that don’t really go out of control but are rather badly mannered. Specifically they don’t appear to obey the rules of my robots.txt particularly well. Usually I just put in rules to ban them permanently

Tonight the biggest parasite appears to have been a New Zealand government department – the Department of Internal Affairs. Processes mounted on IP numbers 192.122.171.75 and 192.122.171.11 ran very high number of queries on the site over a couple of hours. It advertised itself in the user agent as being “heritrix/1.14.1 + http://www.dia.govt.nz/website-harvest”.

Well ok, I looked at the web page and it doesn’t exist. I googled for website harvest at the DIA – nothing. I looked on the DIA website to no avail. I looked up heritrix which asserts that it follows the robots.txt. Thats not what I saw especially when it was picking up from two IPs.

At that point I was tired of being nice to whoever hadn’t put up the web page explaining why they were tapping our website. If anyone does know then put a comment here. I also didn’t know where to send my irritated e-mail to.

I’m hesitant to exclude a branch of the government from archiving our site for prosperity and historians (which is what heritrix is designed for). But I sure as hell want to be able to know what they’re doing. I’d also like them to moderate their behavior and obey the rules in the robots.txt somewhat better.

Rather than doing what I would have usually done and simply denying them access, I decided to draw their attention to their lack of a explanatory webpage.

I used a RewriteRule in .htaccess to permanently redirect every query from them to the missing webpage. I’m now happily off to bed contented that somewhere in the DIA there is a file system or database slowly filling up with copies of their missing page.

13 comments on “Dealing with the DIA website-harvest parasite ”

  1. Jenny 1

    Good for you Lynne, With the $millions dished out to our secretive spooks to prod and pry and fish for our private details and invest our private files.

    It is about time someone punctured their arrogant smug bubble again. (name check Waihopai here).

    capcha – “sequence” (as in code)

    • lprent 1.1

      I don’t think it is the spooks. Too overt for a starter. I figured it out to be a part of something like the wayback engine.

  2. Tim H 2

    Don’t know much about it, but you might want to look at natlib.govt.nz/website-harvest. What with them being subsumed into DIA, I reckon that might have something to do with it. Cock up rather than evil spooks, possibly.

    There’s contact addresses and all there.

    • lprent 2.1

      That will be it. I did this very early this morning. I was more interested in getting their pickup off my screen so I could see the effect of other parasites. Dropping what my site delivered to them from tens of k per page to 596 bytes did it – and amused me as well

    • lprent 2.2

      Google for natlib website-harvest gets it. They need to look at their SEO. website-harvest alone should have been enough for me to locate it.

      http://natlib.govt.nz/website-harvest

      I can’t find a mention of them running a sweep at present. In fact the whole website looks out of date. I guess that the DIA is doing what is expected and not doing the required work for natlib.

      • Idiot/Savant 2.2.1

        They run a sweep fairly regularly – they have a program to archive the top blogs for posterity. Though IIRC they asked permission (which obviously I granted).

        • lprent 2.2.1.1

          It isn’t running through the site that bugs me. I think it is a worthy project.

          It is that they were pulling it too fast whilst people were still active on the site. In particular authors were writing posts after work (it was after Marty commented that he’d had a few outages that I went and looked at what was using more than usual resources). Also that when I went to look for whoever it was, the URL was invalid – that tends to really annoy sysops because it wastes our time.

          They didn’t ask me, but that is because we’re in a .nz domain space rather than .com. There is some legislation covering that.

  3. prism 3

    Is this a follow-on from the political meddling with NZ Archives etc? Is it part of results from trying to reduce quality of service and the separateness of our old records to make short-term savings? We saw how uninterested even hostile previous Nat politicians have been to cultural needs of us as an advanced, educated society, such as in pushing RadioNZ out of their premises, specially built for their purposes.

    • lprent 3.1

      Is this a follow-on from the political meddling with NZ Archives etc?

      I suspect so. What annoys me is that the page pickup rate is too high and it was happening prior to midnight. We get a lot of traffic in the evenings and it runs all the way up to the wee hours of the morning.

      A bot picking up pages virtually every second may do great things for their efficiency. But it is a pain in the arse for sites with a lot of traffic and a very large number of pages. I suspect that it isn’t an issue for something like the NZ Herald site where they have quite a lot of hardware redundancy. But for us where we run on quite limited servers it is annoying.

      The norm for bots like baidubot or googlebot is to pick up a page every 20-30 seconds on any single site during a full scan and do that continuously. That pushes it neatly into the background processing. Of course there are usually a couple of dozen bots running sweeps at any particular point in time. So with the DIA picking up virtually every second it effectively doubles our background processing. I tend to block out servers that pick up every second.

      In practice we use a sitemap (listed in the virtual robots.txt) and push notifications for the major bots to tell them when a page has been altered and if they should pick it up. But that just covers googlebot, bing, yahoo, and ask. Bearing in mind that the bots are well over 50% of the in the day traffic (and 95% of the overnight), I’m starting to consider excluding all bots apart the ones we choose to inform.

  4. I had exactly this happen with a political candidate site, except it was the National Library of Australia, and yes, for a wayback / archive operation.

    We were paying a commercial host and within an hour their spider had used our entire month’s bandwidth allocation and got the site shut down – in the middle of a campaign.

    Figuring they must have the site by now (it was only a few dozen pages!) and they’d leave us alone and intending to complain later, I just paid for additional bandwidth. It came back, did it again. Within 24 hours we were off air again.

    I wrote them a very angry letter (even by my standards) and instead of saying “sorry, we’ll look at what it’s doing” they got lawyers involved. Assholes.

    Bill them for the bandwidth, time and inconvenience and send them off to the debt collectors if they don’t pay up.

    • lprent 4.1

      At present they are sucking up nothing much apart from instructions to read their own site. But the server has an allocation of about 700GB for bandwidth and we’re well under that. Just inconvenient and a bit stupid.

  5. lprent 5

    Finally caught up with my e-mail (I only read it a couple of times per day). There was a good e-mail of which the relevant bit was:-

    We have harvested the site before but unfortunately last night there was a glitch, which set off two harvests instead of one. This would explain the extra traffic on your site last night.

    Compounding that there was a mistake in our website address, which is why you couldn’t find the page that explains our web harvesting process. I’ve corrected that now. The correct URL is http://www.natlib.govt.nz/website-harvest which will appear in future log files.

    I can live with a single process scanning. I’ve released the redirects.

Links to post

Recent Comments

Recent Posts

  • Huge interest in Government’s infrastructure plans
    Hundreds of people in little over a week have turned out in Northland to hear Regional Development Minister Shane Jones speak about plans for boosting the regional economy through infrastructure. About 200 people from the infrastructure and associated sectors attended an event headlined by Mr Jones in Whangarei today. Last ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Health Minister thanks outgoing Health New Zealand Chair
    Health Minister Dr Shane Reti has today thanked outgoing Health New Zealand – Te Whatu Ora Chair Dame Karen Poutasi for her service on the Board.   “Dame Karen tendered her resignation as Chair and as a member of the Board today,” says Dr Reti.  “I have asked her to ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Roads of National Significance planning underway
    The NZ Transport Agency (NZTA) has signalled their proposed delivery approach for the Government’s 15 Roads of National Significance (RoNS), with the release of the State Highway Investment Proposal (SHIP) today, Transport Minister Simeon Brown says.  “Boosting economic growth and productivity is a key part of the Government’s plan to ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Navigating an unstable global environment
    New Zealand is renewing its connections with a world facing urgent challenges by pursuing an active, energetic foreign policy, Foreign Minister Winston Peters says.   “Our country faces the most unstable global environment in decades,” Mr Peters says at the conclusion of two weeks of engagements in Egypt, Europe and the United States.    “We cannot afford to sit back in splendid ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • NZ welcomes Australian Governor-General
    Prime Minister Christopher Luxon has announced the Australian Governor-General, His Excellency General The Honourable David Hurley and his wife Her Excellency Mrs Linda Hurley, will make a State visit to New Zealand from Tuesday 16 April to Thursday 18 April. The visit reciprocates the State visit of former Governor-General Dame Patsy Reddy ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Pseudoephedrine back on shelves for Winter
    Associate Health Minister David Seymour has announced that Medsafe has approved 11 cold and flu medicines containing pseudoephedrine. Pharmaceutical suppliers have indicated they may be able to supply the first products in June. “This is much earlier than the original expectation of medicines being available by 2025. The Government recognised ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • NZ and the US: an ever closer partnership
    New Zealand and the United States have recommitted to their strategic partnership in Washington DC today, pledging to work ever more closely together in support of shared values and interests, Foreign Minister Winston Peters says.    “The strategic environment that New Zealand and the United States face is considerably more ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Joint US and NZ declaration
    April 11, 2024 Joint Declaration by United States Secretary of State the Honorable Antony J. Blinken and New Zealand Minister of Foreign Affairs the Right Honourable Winston Peters We met today in Washington, D.C. to recommit to the historic partnership between our two countries and the principles that underpin it—rule ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • NZ and US to undertake further practical Pacific cooperation
    Foreign Minister Winston Peters has announced further New Zealand cooperation with the United States in the Pacific Islands region through $16.4 million in funding for initiatives in digital connectivity and oceans and fisheries research.   “New Zealand can achieve more in the Pacific if we work together more urgently and ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Government redress for Te Korowai o Wainuiārua
    The Government is continuing the bipartisan effort to restore its relationship with iwi as the Te Korowai o Wainuiārua Claims Settlement Bill passed its first reading in Parliament today, says Treaty Negotiations Minister Paul Goldsmith. “Historical grievances of Te Korowai o Wainuiārua relate to 19th century warfare, land purchased or taken ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Focus on outstanding minerals permit applications
    New Zealand Petroleum and Minerals is working to resolve almost 150 outstanding minerals permit applications by the end of the financial year, enabling valuable mining activity and signalling to the sector that New Zealand is open for business, Resources Minister Shane Jones says.  “While there are no set timeframes for ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Applications open for NZ-Ireland Research Call
    The New Zealand and Irish governments have today announced that applications for the 2024 New Zealand-Ireland Joint Research Call on Agriculture and Climate Change are now open. This is the third research call in the three-year Joint Research Initiative pilot launched in 2022 by the Ministry for Primary Industries and Ireland’s ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Tenancy rules changes to improve rental market
    The coalition Government has today announced changes to the Residential Tenancies Act to encourage landlords back to the rental property market, says Housing Minister Chris Bishop. “The previous Government waged a war on landlords. Many landlords told us this caused them to exit the rental market altogether. It caused worse ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Boosting NZ’s trade and agricultural relationship with China
    Trade and Agriculture Minister Todd McClay will visit China next week, to strengthen relationships, support Kiwi exporters and promote New Zealand businesses on the world stage. “China is one of New Zealand’s most significant trade and economic relationships and remains an important destination for New Zealand’s products, accounting for nearly 22 per cent of our good and ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Freshwater farm plan systems to be improved
    The coalition Government intends to improve freshwater farm plans so that they are more cost-effective and practical for farmers, Associate Environment Minister Andrew Hoggard and Agriculture Minister Todd McClay have announced. “A fit-for-purpose freshwater farm plan system will enable farmers and growers to find the right solutions for their farm ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • New Fast Track Projects advisory group named
    The coalition Government has today announced the expert advisory group who will provide independent recommendations to Ministers on projects to be included in the Fast Track Approvals Bill, say RMA Reform Minister Chris Bishop and Regional Development Minister Shane Jones. “Our Fast Track Approval process will make it easier and ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Pacific and Gaza focus of UN talks
    Foreign Minister Winston Peters says his official talks with the United Nations Secretary-General Antonio Guterres in New York today focused on a shared commitment to partnering with the Pacific Islands region and a common concern about the humanitarian catastrophe in Gaza.    “Small states in the Pacific rely on collective ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Government honours Taranaki Maunga deal
    The Government is honouring commitments made to Taranaki iwi with the Te Pire Whakatupua mō Te Kāhui Tupua/Taranaki Maunga Collective Redress Bill passing its first reading Parliament today, Treaty Negotiations Minister Paul Goldsmith says. “This Bill addresses the commitment the Crown made to the eight iwi of Taranaki to negotiate ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Enhanced partnership to reduce agricultural emissions
    The Government and four further companies are together committing an additional $18 million towards AgriZeroNZ to boost New Zealand’s efforts to reduce agricultural emissions. Agriculture Minister Todd McClay says the strength of the New Zealand economy relies on us getting effective and affordable emission reduction solutions for New Zealand. “The ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • 110km/h limit proposed for Kāpiti Expressway
    Transport Minister Simeon Brown has welcomed news the NZ Transport Agency (NZTA) will begin consultation this month on raising speed limits for the Kāpiti Expressway to 110km/h. “Boosting economic growth and productivity is a key part of the Government’s plan to rebuild the economy and this proposal supports that outcome ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • New Zealand Biosecurity Awards – Winners announced
    Two New Zealanders who’ve used their unique skills to help fight the exotic caulerpa seaweed are this year’s Biosecurity Awards Supreme Winners, says Biosecurity Minister Andrew Hoggard. “Strong biosecurity is vital and underpins the whole New Zealand economy and our native flora and fauna. These awards celebrate all those in ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Attendance action plan to lift student attendance rates
    The Government is taking action to address the truancy crisis and raise attendance by delivering the attendance action plan, Associate Education Minister David Seymour announced today.   New Zealand attendance rates are low by national and international standards. Regular attendance, defined as being in school over 90 per cent of the ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • World must act to halt Gaza catastrophe – Peters
    Foreign Minister Winston Peters has told the United Nations General Assembly (UNGA) in New York today that an immediate ceasefire is needed in Gaza to halt the ongoing humanitarian catastrophe.    “Palestinian civilians continue to bear the brunt of Israel’s military actions,” Mr Peters said in his speech to a ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Speech to United Nations General Assembly: 66th plenary meeting, 78th session
    Mr President,   The situation in Gaza is an utter catastrophe.   New Zealand condemns Hamas for its heinous terrorist attacks on 7 October and since, including its barbaric violations of women and children. All of us here must demand that Hamas release all remaining hostages immediately.   At the ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Government woolshed roadshow kicks off
    Today the Government Agriculture Ministers started their national woolshed roadshow, kicking off in the Wairarapa. Agriculture Minister Todd McClay said it has been a tough time for farmers over the past few years. The sector has faced high domestic inflation rates, high interest rates, adverse weather events, and increasing farm ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • PM heads to Singapore, Thailand, and Philippines
    Prime Minister Christopher Luxon will travel to Singapore, Thailand and the Philippines this week (April 14-20), along with a senior business delegation, signalling the Government’s commitment to deepen New Zealand’s international engagement, especially our relationships in South East Asia. “South East Asia is a region that is more crucial than ever to ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Prime Minister launches Government Targets
    Prime Minister Christopher Luxon has announced further steps to get New Zealand back on track, launching nine ambitious Government Targets to help improve the lives of New Zealanders. “Our Government has a plan that is focused on three key promises we made to New Zealanders – to rebuild the economy, ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Natural hydrogen resource should be free of Treaty claims entanglement
    Natural hydrogen could be a game-changing new source of energy for New Zealand but it is essential it is treated as a critical development that benefits all New Zealanders, Resources Minister Shane Jones says. Mr Jones is seeking to give regulatory certainty for those keen to develop natural, or geological, ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Government responds to unsustainable net migration
    ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • New Zealand on stage at global Space Symposium
    Space Minister Judith Collins will speak at the Space Symposium in the United States next week, promoting New Zealand’s rapidly growing place in the sector as we work to rebuild the economy. “As one of the largest global space events, attended by more than 10,000 business and government representatives from ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • $4.9m project completed with marae reopening
    A significant marae has reopened in the heart of Rotorua marking the end of renovations for the Ruatāhuna Marae Renovation Cluster, a project that provided much-needed jobs and regional economic stimulus, Regional Development Minister Shane Jones says. Mr Jones was at the official reopening of Mātaatua ki Rotorua Marae today. ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Pure Tūroa Limited to operate Tūroa ski field
    Ko Tahuarangi te waka – Tahuarangi is the ancestral vessel Ko Rangitukutuku te aho – Rangitukutuku is the fishing line Ko Pikimairawea te matau – Pikimairawea is the hook Ko Hāhā te Whenua te ika kei rō-wai – Hāhā te whenua is the fish (of Māui) whilst under the ocean ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Methane targets to be independently reviewed
    Rebuilding New Zealand’s economy will rely on the valuable agricultural sector working sustainably towards our climate change goals.  Today, the Climate Change and Agriculture Ministers announced that an independent panel of experts will review agricultural biogenic methane science and targets for consistency with no additional warming. Agriculture Minister Todd McClay ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • NZ and Nordics: likeminded partners
    Foreign Minister Winston Peters has highlighted the strong ties that bind New Zealand and the Nordic countries of Northern Europe during a trip to Sweden today.    “There are few countries in the world more likeminded with New Zealand than our friends in Northern Europe,” Mr Peters says.    “We ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • First New Zealand C-130J Hercules takes flight
    The first New Zealand C-130J Hercules to come off the production line in the United States has successfully completed its first test flights, Defence Minister Judith Collins announced today. “These successful flights are a significant milestone for the New Zealand Defence Force, bringing this once-in-a-generation renewal of a critical airlift ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government to rephase NCEA Change Programme
      The coalition Government is making significant changes to the NCEA Change Programme, delaying the implementation by two years, Minister of Education Erica Stanford announced today. “Ensuring New Zealand’s curriculum is world leading is a vital part of the Government’s plan to deliver better public services and ensure all students ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Ngāpuhi investment fund Chair appointed
    Ben Dalton has been appointed the new board Chair of Tupu Tonu, the Ngāpuhi Investment Fund, says Treaty Negotiations Minister Paul Goldsmith and Associate Finance Minister Shane Jones. “Ben brings a wealth of experience in governance and economic development to the position. He will have a strong focus on ensuring ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Education should be prioritised ahead of protesting
    Students should be in school and learning instead of protesting during school hours, Associate Education Minister David Seymour says. “If students feel strongly about sending a message, they could have marched on Tuesday when there was a nationwide teacher only day, or during the upcoming school holidays. It has become ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Delivering on Local Water Done Well
    Cabinet has agreed on key steps to implement Local Water Done Well, the Coalition Government’s plan for financially sustainable locally delivered water infrastructure and services, Local Government Minister Simeon Brown says.  "Councils and voters resoundingly rejected Labour’s expensive and bureaucratic Three Waters regime, and earlier this year the Coalition Government ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Peters to visit New York, Washington D.C.
    Foreign Minister Winston Peters will engage with high-level United States Government and United Nations officials in the United States next week (6-12 April).    The visit, with programmes in New York and Washington D.C., will focus on major global and regional security challenges and includes meetings with US Secretary of ...
    BeehiveBy beehive.govt.nz
    1 week ago

Page generated in The Standard by Wordpress at 2024-04-13T23:12:25+00:00