web analytics

Dealing with the DIA website-harvest parasite

Written By: - Date published: 2:27 am, February 3rd, 2011 - 13 comments
Categories: admin, blogs, humour, inoculation, interweb, scoundrels, Spying, The Standard - Tags: ,

Periodically I run a scan to identify network parasites that are sucking up our bandwidth and processing resources in excess. Of course I leave the benign parasites that provide search facilities alone. Like the ubiquitous googlebot and its other benign brethren. But I stomp on the nasties and inoculate the system against them.

It could be spambots trying to insert comments and requiring us to expend effort in cleaning up the garbage. There denying from .htaccess is your friend and vast swathes of the IP’s from Russia, China and some other regions are excluded from reading anything at all.

It could be virus slaved botnets mindlessly trying to exploit our contact us and contribute pages to send e-mails. There I find that a relatively simple captcha is sufficient to deter them.

Feedbots picking up RSS feeds used to be a problem. But since I put some limits on how much they can pick up and how old it can be, that problem has disappeared.

It could be a out of control spider trying to crawl all over our web pages in a single cataclysmic event. I find that putting restrictions on how often an IP number can connect within a given period of time tends to squelch them. Because of the nature of the other beasts on the net I have that to just harm the really out of control processes with grey-listing. The system starts refusing to accept new connections after a while while very slowwwly feeding the bot a dribble of data and holding them up.

But that leaves the processes that don’t really go out of control but are rather badly mannered. Specifically they don’t appear to obey the rules of my robots.txt particularly well. Usually I just put in rules to ban them permanently

Tonight the biggest parasite appears to have been a New Zealand government department – the Department of Internal Affairs. Processes mounted on IP numbers 192.122.171.75 and 192.122.171.11 ran very high number of queries on the site over a couple of hours. It advertised itself in the user agent as being “heritrix/1.14.1 + http://www.dia.govt.nz/website-harvest”.

Well ok, I looked at the web page and it doesn’t exist. I googled for website harvest at the DIA – nothing. I looked on the DIA website to no avail. I looked up heritrix which asserts that it follows the robots.txt. Thats not what I saw especially when it was picking up from two IPs.

At that point I was tired of being nice to whoever hadn’t put up the web page explaining why they were tapping our website. If anyone does know then put a comment here. I also didn’t know where to send my irritated e-mail to.

I’m hesitant to exclude a branch of the government from archiving our site for prosperity and historians (which is what heritrix is designed for). But I sure as hell want to be able to know what they’re doing. I’d also like them to moderate their behavior and obey the rules in the robots.txt somewhat better.

Rather than doing what I would have usually done and simply denying them access, I decided to draw their attention to their lack of a explanatory webpage.

I used a RewriteRule in .htaccess to permanently redirect every query from them to the missing webpage. I’m now happily off to bed contented that somewhere in the DIA there is a file system or database slowly filling up with copies of their missing page.

13 comments on “Dealing with the DIA website-harvest parasite ”

  1. Jenny 1

    Good for you Lynne, With the $millions dished out to our secretive spooks to prod and pry and fish for our private details and invest our private files.

    It is about time someone punctured their arrogant smug bubble again. (name check Waihopai here).

    capcha – “sequence” (as in code)

    • lprent 1.1

      I don’t think it is the spooks. Too overt for a starter. I figured it out to be a part of something like the wayback engine.

  2. Tim H 2

    Don’t know much about it, but you might want to look at natlib.govt.nz/website-harvest. What with them being subsumed into DIA, I reckon that might have something to do with it. Cock up rather than evil spooks, possibly.

    There’s contact addresses and all there.

    • lprent 2.1

      That will be it. I did this very early this morning. I was more interested in getting their pickup off my screen so I could see the effect of other parasites. Dropping what my site delivered to them from tens of k per page to 596 bytes did it – and amused me as well

    • lprent 2.2

      Google for natlib website-harvest gets it. They need to look at their SEO. website-harvest alone should have been enough for me to locate it.

      http://natlib.govt.nz/website-harvest

      I can’t find a mention of them running a sweep at present. In fact the whole website looks out of date. I guess that the DIA is doing what is expected and not doing the required work for natlib.

      • Idiot/Savant 2.2.1

        They run a sweep fairly regularly – they have a program to archive the top blogs for posterity. Though IIRC they asked permission (which obviously I granted).

        • lprent 2.2.1.1

          It isn’t running through the site that bugs me. I think it is a worthy project.

          It is that they were pulling it too fast whilst people were still active on the site. In particular authors were writing posts after work (it was after Marty commented that he’d had a few outages that I went and looked at what was using more than usual resources). Also that when I went to look for whoever it was, the URL was invalid – that tends to really annoy sysops because it wastes our time.

          They didn’t ask me, but that is because we’re in a .nz domain space rather than .com. There is some legislation covering that.

  3. prism 3

    Is this a follow-on from the political meddling with NZ Archives etc? Is it part of results from trying to reduce quality of service and the separateness of our old records to make short-term savings? We saw how uninterested even hostile previous Nat politicians have been to cultural needs of us as an advanced, educated society, such as in pushing RadioNZ out of their premises, specially built for their purposes.

    • lprent 3.1

      Is this a follow-on from the political meddling with NZ Archives etc?

      I suspect so. What annoys me is that the page pickup rate is too high and it was happening prior to midnight. We get a lot of traffic in the evenings and it runs all the way up to the wee hours of the morning.

      A bot picking up pages virtually every second may do great things for their efficiency. But it is a pain in the arse for sites with a lot of traffic and a very large number of pages. I suspect that it isn’t an issue for something like the NZ Herald site where they have quite a lot of hardware redundancy. But for us where we run on quite limited servers it is annoying.

      The norm for bots like baidubot or googlebot is to pick up a page every 20-30 seconds on any single site during a full scan and do that continuously. That pushes it neatly into the background processing. Of course there are usually a couple of dozen bots running sweeps at any particular point in time. So with the DIA picking up virtually every second it effectively doubles our background processing. I tend to block out servers that pick up every second.

      In practice we use a sitemap (listed in the virtual robots.txt) and push notifications for the major bots to tell them when a page has been altered and if they should pick it up. But that just covers googlebot, bing, yahoo, and ask. Bearing in mind that the bots are well over 50% of the in the day traffic (and 95% of the overnight), I’m starting to consider excluding all bots apart the ones we choose to inform.

  4. I had exactly this happen with a political candidate site, except it was the National Library of Australia, and yes, for a wayback / archive operation.

    We were paying a commercial host and within an hour their spider had used our entire month’s bandwidth allocation and got the site shut down – in the middle of a campaign.

    Figuring they must have the site by now (it was only a few dozen pages!) and they’d leave us alone and intending to complain later, I just paid for additional bandwidth. It came back, did it again. Within 24 hours we were off air again.

    I wrote them a very angry letter (even by my standards) and instead of saying “sorry, we’ll look at what it’s doing” they got lawyers involved. Assholes.

    Bill them for the bandwidth, time and inconvenience and send them off to the debt collectors if they don’t pay up.

    • lprent 4.1

      At present they are sucking up nothing much apart from instructions to read their own site. But the server has an allocation of about 700GB for bandwidth and we’re well under that. Just inconvenient and a bit stupid.

  5. lprent 5

    Finally caught up with my e-mail (I only read it a couple of times per day). There was a good e-mail of which the relevant bit was:-

    We have harvested the site before but unfortunately last night there was a glitch, which set off two harvests instead of one. This would explain the extra traffic on your site last night.

    Compounding that there was a mistake in our website address, which is why you couldn’t find the page that explains our web harvesting process. I’ve corrected that now. The correct URL is http://www.natlib.govt.nz/website-harvest which will appear in future log files.

    I can live with a single process scanning. I’ve released the redirects.

Links to post

Recent Comments

Recent Posts

  • Clean car package to drive down emissions
    New rebates for electric and plug-in hybrid vehicles start July 1 with up to $8,625 for new vehicles and $3,450 for used. Electric vehicle chargers now available every 75km along most state highways to give Kiwis confidence. Low Emission Transport Fund will have nearly four times the funding by 2023 ...
    BeehiveBy beehive.govt.nz
    13 hours ago
  • Progress towards simpler process for changing sex on birth certificates
    The Government is taking the next step to support transgender, non-binary and intersex New Zealanders, by progressing the Births, Deaths, Marriages and Relationships Registration Bill, Minister of Internal Affairs, Jan Tinetti announced today. “This Government understands that self-identification is a significant issue for transgender, non-binary and intersex New Zealanders, and ...
    BeehiveBy beehive.govt.nz
    15 hours ago
  • Crown speeds up engagement with takutai moana applicants
    The Crown is taking a new approach to takutai moana applications to give all applicants an opportunity to engage with the Crown and better support the Māori-Crown relationship, Treaty of Waitangi Negotiations Minister Andrew Little says. Following discussions with applicant groups, the Crown has reviewed the existing takutai moana application ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Alcohol and Other Drug Treatment (AODT) Court opens
    The Minister of Justice, Kris Faafoi, and the Minister for Courts, Aupito William Sio, have welcomed the opening of a new Alcohol and Other Drug Treatment (AODT) Court in Hamilton. The AODT Court (Te Whare Whakapiki Wairua) addresses situations where substance abuse and offending are intertwined. “New Zealanders have told ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • EU and UK FTAs top of list for first ministerial trip since COVID-19
    Trade and Export Growth Minister Damien O’Connor today announced details of his planned visit to the United Kingdom and European Union next week, where he will hold trade and agriculture discussions to further New Zealand’s economic recovery from COVID-19. The visit will add political weight to ongoing negotiations with both the EU ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Arihia Bennett to chair Royal Commission Ministerial Advisory Group
    Te Rūnanga o Ngāi Tahu chief executive Arihia Bennett MNZM has been appointed chair of the newly appointed Ministerial Advisory Group on the Government’s Response to the Royal Commission of Inquiry into the terrorist attack on Christchurch mosques. “Twenty-eight people from diverse backgrounds across Aotearoa have been selected for the ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Speech to the New Zealand Medical Association General Practitioners' Conference, Rotorua
    Ki ngā pou maha o te whare hauora o Aotearoa, kei te mihiTo the pillars of our health system I acknowledge/thank you Ki te ope hapai hauora o roto o tēnei rūma, kei te mihi To our health force here in the room today, I acknowledge/thank you He taura tangata, ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Karangahape Road upgrades are streets ahead
    The upgrades to Karangahape Road makes the iconic street more pedestrian and cycle-friendly, attractive and environmentally sustainable, Transport Minister Michael Wood and Auckland Mayor Phil Goff said at the formal celebration of the completion of the Karangahape Road Enhancements project. The project included widening footpaths supporting a better outdoor dining ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Speech to APEC business event
    E ngā tumu herenga waka, ākina ā ngaru, ākina ā tai ka whakatere ngā waka ki te whakapapa pounamu, otirā, ki Tamaki o ngā waka Tena koutou katoa… To the great leaders assembled, who guided your waka through turbulent times, challenging waters and you continue to navigate your respective waka ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Pause on Quarantine Free Travel with Victoria extended
    Following an assessment of the COVID-19 outbreak in greater Melbourne, New Zealand’s Quarantine Free Travel pause with Victoria will continue for a further seven days, COVID-19 Response Minister Chris Hipkins says. There are now 93 cases associated with the outbreak in greater Melbourne, spread over four clusters. Contact tracing efforts ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Supplier Diversity Aotearoa Summit: Navigate 2021
    *** Check with delivery *** A mihi to all who have contributed to making today a success – starting with you! As you have explored and navigated government procurement today you will hopefully have reflected on the journey of our people so far – and how you can make a ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Pukemiro School to close
    Pukemiro Primary School near Huntly will close following years of declining roll numbers, Education Minister Chris Hipkins announced today. “I’ve consulted with the School Commissioner, and this decision acknowledges the fact that the few remaining students from last term are now settled at other nearby schools. “I want to thank ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Govt acts to protect NZers from harmful content
    New Zealanders will be better protected from harmful or illegal content as a result of work to design a modern, flexible and coherent regulatory framework, Minister of Internal Affairs Jan Tinetti announced today. New Zealand currently has a content regulatory system that is comprised of six different arrangements covering some ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Consultation on exemption of new builds from proposed tax rules
    The Government has today confirmed new builds will be exempt from planned changes to the tax treatment of residential investment property.  Public consultation is now open on details of the proposals, which stop interest deductions being claimed for residential investment properties other than new builds.   “The Government’s goal is to ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Speech for Predator Free 2050 Conference
    Introduction E ngā mana E ngā reo E ngā iwi Tēnā koutou katoa   Ka huri ki ngā mana whenua o te rohe nei. Tēnā koutou He mihi hoki ki a tatou kua tau mai nei i raro i te kaupapa o te rā Ko Ayesha Verrall toku ingoa No ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • New stock exchange to help grow small businesses
    A new share trading market, designed as a gateway to the NZX for small-to-medium-sized enterprises (SMEs), has been granted a licence by the Government. Commerce and Consumer Affairs Minister, David Clark said Catalist Markets Ltd will provide a simpler and more affordable ‘stepping stone’ for SMEs to raise capital. “This ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Visa extensions provide certainty to employers and 10,000 visa holders
    Changes to onshore visas will provide employers and visa holders with more certainty, Immigration Minister Kris Faafoi has announced. Around 10,000 Working Holiday visas and Supplementary Seasonal Employment (SSE) work visas due to expire between 21 June 2021 and 31 December 2021 will be extended for another six months to ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Border class exceptions approved for more farm workers and vets
    The Government has approved border class exceptions for an additional 200 dairy workers and 50 veterinarians to enter New Zealand, Agriculture Minister Damien O’Connor announced today.  “It is clear from conversations with the dairy and veterinarian sectors that they are facing workforce pressures. These border exceptions will go a long ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • More freezers and South Island hub to support vaccine roll-out
    A South Island hub and 17 new ultra-low temperature freezers will help further prepare New Zealand for the ramp up of the vaccination programme in the second half of this year, COVID-19 Response Minister Chris Hipkins says. The new freezers arrived in New Zealand on 27 May. They’re currently being ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Speech at the release of Climate Change Commission's final advice
    Good morning – and thank you Prime Minister. Over the last three and half years we have been putting in place the foundations for a low-carbon Aotearoa that will be a catalyst for job creation, innovation, and prosperity for decades to come. In that future, many of our everyday tasks ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Achievable blueprint for addressing climate change released
    Report says Government making good progress on emissions reduction, but more action required Meeting climate targets achievable and affordable with existing technology Economic cost of delaying action higher than taking action now Benefits from climate action include health improvements and lower energy bills All Ministers to help meet climate targets ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Speech to release of Climate Commission final report
    A few years ago in a speech in Auckland, I compared climate change to the nuclear free movement of roughly four decades ago. And I did so for a few reasons. Firstly, because the movement of the 1980s represented a life or death situation for the Pacific, and so does ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Appointment of Judge of the High Court
    Auckland Barrister Michael Robinson has been appointed a Judge of the High Court, Attorney‑General David Parker announced today. Justice Robinson graduated with a BA and an LLB (Hons) from the University of Auckland in 1996, and commenced practice as a solicitor with Brookfields in Auckland.  In 1998 he travelled to London ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Government takes action to improve protections for subcontractors
    The Construction Contracts (Retention Money) Amendment Bill – which provides greater financial protection for subcontractors, has passed its first reading today. The Bill amends the retention provisions in the Construction Contracts Act 2002 (CCA) to provide increased confidence and transparency for subcontractors that retention money they are owed is safe. ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • 1 million more Pfizer doses to arrive in July
    Pfizer has scheduled delivery of an estimated 1 million doses of vaccine to New Zealand during July, COVID1-9 Response Minister Chris Hipkins said today. “These consignments will double the total number of Pfizer doses we have received this year to more than 1,900,000 – enough to fully vaccinate almost 1 ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Long-term home of the Independent Children’s Monitor identified
    The Independent Children’s Monitor (Te Mana Whakamaru Tamariki Motuhake), which is currently located within the Ministry of Social Development (MSD), will become its own departmental agency within Government. “Following the recommendations of several reviews, Cabinet agreed in 2019 to build a significantly expanded independent monitor for children in care,” Carmel ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Racing Integrity Board members announced
    The new Racing Integrity Board will be up and running from July 1 to ensure high standards of animal welfare, integrity and professionalism in the racing industry. Racing Minister Grant Robertson today announced the appointments to the new Board: Sir Bruce Robertson KNZM – Chair Kristy McDonald ONZM QC Penelope ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Govt crackdown on organised crime continues
    A major operation against multiple organised crime groups with international links will make a significant dent in drug harm and violent offending linked to organised crime networks, Police Minister Poto Williams says. “I want to take an opportunity to congratulate the Police for their role in Operation Trojan Shield. This ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Farm planning framework supports farmers into the future
    A new framework, agreed between Government and industry, will make it easier for farmers and growers to integrate future greenhouse gas emissions and freshwater regulatory requirements into their farm planning, Agriculture Minister Damien O’Connor said. “The Good Farm Planning Principles Guide out today, provides guidance for how farmers can organise ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Enhanced Task Force Green Approved for Canterbury
    The Government has activated Enhanced Taskforce Green (ETFG) in response to the Canterbury floods. The Minister of Social Development and Employment, Hon Carmel Sepuloni says $500,000 will be made available to help with the clean-up. The flooding in Canterbury has been a significant and adverse event damaging farmland, homes, roads ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Connecting rangatahi to the soil
    A Jobs for Nature project to raise 480,000 native plants in nurseries across South Auckland will provide work for communities disproportionately affected by the economic impacts of COVID-19, Acting Conservation Minister Ayesha Verrall says. The Mana in Kaimahi project is being run by Te Whāngai Trust Board and will establish ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Roll out of high-resolution elevation mapping begins
    The first tranche of mapping data from the Provincial Growth Fund (PGF)-LiDAR project is now available to the public from Toitū Te Whenua Land Information New Zealand. LiDAR data, which creates 3D baseline elevation information, will deliver multiple uses over the coming decades to councils and regional industries. “This mapping ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Champions of Pacific education rewarded in Queen’s Birthday Honours
    Minister for Pacific Peoples, Aupito William Sio said the Queen’s Birthday 2021 Honours list show that across Aotearoa New Zealand there were many champions of Pacific education. “Education is so vital to the success of Pacific people that it’s truly fitting that a number of educators have been honoured this ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • PM congratulates Queen’s Birthday Honours recipients
    Prime Minister Jacinda Ardern has added her congratulations to the New Zealanders recognised for their contributions to their communities and the country in the Queen’s Birthday 2021 Honours List. “This group represents decades of services across many areas, and those honoured highlight how many New Zealanders are going above and ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Change of status for Rangiriri kura
    A change of status for Te Kura o Rangiriri sees it become a designated character school within the Māori-medium network, Associate Minister of Education Kelvin Davis announced today. “This kura has been providing Māori immersion learning since 2003 in the historic town of Rangiriri, so I’m delighted that it is ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • APEC trade ministers’ unite on COVID-19 vaccine steps and rejuvenating the WTO
    APEC trade ministers today committed to speeding up the cross-border flow of vaccines and related goods to fight the COVID-19 pandemic. This followed the completion of the APEC Ministers Responsible for Trade Meeting chaired by Minister for Trade and Export Growth Damien O’Connor early this morning. “As we face the ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Further consultation for Melville schools
    Formal consultation is set to begin on specific options for the future of schooling in South West Hamilton, Education Minister Chris Hipkins says. “Recent engagement has shown that the schools and community want a change to Year 7-13 schooling for the area.  “I am now asking the Boards of Melville ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Primary schoolkids dig trees for Arbor Day
    A new Arbor Day initiative announced by Forestry Minister Stuart Nash will see thousands of primary school children get the chance to plant native trees in their communities. The initiative is open to more than 2,400 primary schools. It is a partnership between Te Uru Rākau/NZ Forest Service and the ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • NZ Upgrade Programme kept on track
    The Government is increasing its investment in the New Zealand Upgrade Programme (NZUP) to support New Zealand’s economic recovery. Over two thirds of the projects will proceed as announced despite increased costs due to COVID, with modifications being made to others. “NZUP is already supporting over 1,000 jobs with 13 ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Northern Pathway gets green light
    The Government is committed to the Northern Pathway with its preferred option being a separate structure for walking and cycling alongside the Auckland Harbour Bridge, Transport Minister Michael Wood announced today. Geotechnical investigations and testing has determined that building a structure connected to the Auckland Harbour Bridge is not possible ...
    BeehiveBy beehive.govt.nz
    1 week ago