web analytics

Dealing with the DIA website-harvest parasite

Written By: - Date published: 2:27 am, February 3rd, 2011 - 13 comments
Categories: admin, blogs, humour, inoculation, interweb, scoundrels, Spying, The Standard - Tags: ,

Periodically I run a scan to identify network parasites that are sucking up our bandwidth and processing resources in excess. Of course I leave the benign parasites that provide search facilities alone. Like the ubiquitous googlebot and its other benign brethren. But I stomp on the nasties and inoculate the system against them.

It could be spambots trying to insert comments and requiring us to expend effort in cleaning up the garbage. There denying from .htaccess is your friend and vast swathes of the IP’s from Russia, China and some other regions are excluded from reading anything at all.

It could be virus slaved botnets mindlessly trying to exploit our contact us and contribute pages to send e-mails. There I find that a relatively simple captcha is sufficient to deter them.

Feedbots picking up RSS feeds used to be a problem. But since I put some limits on how much they can pick up and how old it can be, that problem has disappeared.

It could be a out of control spider trying to crawl all over our web pages in a single cataclysmic event. I find that putting restrictions on how often an IP number can connect within a given period of time tends to squelch them. Because of the nature of the other beasts on the net I have that to just harm the really out of control processes with grey-listing. The system starts refusing to accept new connections after a while while very slowwwly feeding the bot a dribble of data and holding them up.

But that leaves the processes that don’t really go out of control but are rather badly mannered. Specifically they don’t appear to obey the rules of my robots.txt particularly well. Usually I just put in rules to ban them permanently

Tonight the biggest parasite appears to have been a New Zealand government department – the Department of Internal Affairs. Processes mounted on IP numbers 192.122.171.75 and 192.122.171.11 ran very high number of queries on the site over a couple of hours. It advertised itself in the user agent as being “heritrix/1.14.1 + http://www.dia.govt.nz/website-harvest”.

Well ok, I looked at the web page and it doesn’t exist. I googled for website harvest at the DIA – nothing. I looked on the DIA website to no avail. I looked up heritrix which asserts that it follows the robots.txt. Thats not what I saw especially when it was picking up from two IPs.

At that point I was tired of being nice to whoever hadn’t put up the web page explaining why they were tapping our website. If anyone does know then put a comment here. I also didn’t know where to send my irritated e-mail to.

I’m hesitant to exclude a branch of the government from archiving our site for prosperity and historians (which is what heritrix is designed for). But I sure as hell want to be able to know what they’re doing. I’d also like them to moderate their behavior and obey the rules in the robots.txt somewhat better.

Rather than doing what I would have usually done and simply denying them access, I decided to draw their attention to their lack of a explanatory webpage.

I used a RewriteRule in .htaccess to permanently redirect every query from them to the missing webpage. I’m now happily off to bed contented that somewhere in the DIA there is a file system or database slowly filling up with copies of their missing page.

13 comments on “Dealing with the DIA website-harvest parasite ”

  1. Jenny 1

    Good for you Lynne, With the $millions dished out to our secretive spooks to prod and pry and fish for our private details and invest our private files.

    It is about time someone punctured their arrogant smug bubble again. (name check Waihopai here).

    capcha – “sequence” (as in code)

    • lprent 1.1

      I don’t think it is the spooks. Too overt for a starter. I figured it out to be a part of something like the wayback engine.

  2. Tim H 2

    Don’t know much about it, but you might want to look at natlib.govt.nz/website-harvest. What with them being subsumed into DIA, I reckon that might have something to do with it. Cock up rather than evil spooks, possibly.

    There’s contact addresses and all there.

    • lprent 2.1

      That will be it. I did this very early this morning. I was more interested in getting their pickup off my screen so I could see the effect of other parasites. Dropping what my site delivered to them from tens of k per page to 596 bytes did it – and amused me as well

    • lprent 2.2

      Google for natlib website-harvest gets it. They need to look at their SEO. website-harvest alone should have been enough for me to locate it.

      http://natlib.govt.nz/website-harvest

      I can’t find a mention of them running a sweep at present. In fact the whole website looks out of date. I guess that the DIA is doing what is expected and not doing the required work for natlib.

      • Idiot/Savant 2.2.1

        They run a sweep fairly regularly – they have a program to archive the top blogs for posterity. Though IIRC they asked permission (which obviously I granted).

        • lprent 2.2.1.1

          It isn’t running through the site that bugs me. I think it is a worthy project.

          It is that they were pulling it too fast whilst people were still active on the site. In particular authors were writing posts after work (it was after Marty commented that he’d had a few outages that I went and looked at what was using more than usual resources). Also that when I went to look for whoever it was, the URL was invalid – that tends to really annoy sysops because it wastes our time.

          They didn’t ask me, but that is because we’re in a .nz domain space rather than .com. There is some legislation covering that.

  3. prism 3

    Is this a follow-on from the political meddling with NZ Archives etc? Is it part of results from trying to reduce quality of service and the separateness of our old records to make short-term savings? We saw how uninterested even hostile previous Nat politicians have been to cultural needs of us as an advanced, educated society, such as in pushing RadioNZ out of their premises, specially built for their purposes.

    • lprent 3.1

      Is this a follow-on from the political meddling with NZ Archives etc?

      I suspect so. What annoys me is that the page pickup rate is too high and it was happening prior to midnight. We get a lot of traffic in the evenings and it runs all the way up to the wee hours of the morning.

      A bot picking up pages virtually every second may do great things for their efficiency. But it is a pain in the arse for sites with a lot of traffic and a very large number of pages. I suspect that it isn’t an issue for something like the NZ Herald site where they have quite a lot of hardware redundancy. But for us where we run on quite limited servers it is annoying.

      The norm for bots like baidubot or googlebot is to pick up a page every 20-30 seconds on any single site during a full scan and do that continuously. That pushes it neatly into the background processing. Of course there are usually a couple of dozen bots running sweeps at any particular point in time. So with the DIA picking up virtually every second it effectively doubles our background processing. I tend to block out servers that pick up every second.

      In practice we use a sitemap (listed in the virtual robots.txt) and push notifications for the major bots to tell them when a page has been altered and if they should pick it up. But that just covers googlebot, bing, yahoo, and ask. Bearing in mind that the bots are well over 50% of the in the day traffic (and 95% of the overnight), I’m starting to consider excluding all bots apart the ones we choose to inform.

  4. I had exactly this happen with a political candidate site, except it was the National Library of Australia, and yes, for a wayback / archive operation.

    We were paying a commercial host and within an hour their spider had used our entire month’s bandwidth allocation and got the site shut down – in the middle of a campaign.

    Figuring they must have the site by now (it was only a few dozen pages!) and they’d leave us alone and intending to complain later, I just paid for additional bandwidth. It came back, did it again. Within 24 hours we were off air again.

    I wrote them a very angry letter (even by my standards) and instead of saying “sorry, we’ll look at what it’s doing” they got lawyers involved. Assholes.

    Bill them for the bandwidth, time and inconvenience and send them off to the debt collectors if they don’t pay up.

    • lprent 4.1

      At present they are sucking up nothing much apart from instructions to read their own site. But the server has an allocation of about 700GB for bandwidth and we’re well under that. Just inconvenient and a bit stupid.

  5. lprent 5

    Finally caught up with my e-mail (I only read it a couple of times per day). There was a good e-mail of which the relevant bit was:-

    We have harvested the site before but unfortunately last night there was a glitch, which set off two harvests instead of one. This would explain the extra traffic on your site last night.

    Compounding that there was a mistake in our website address, which is why you couldn’t find the page that explains our web harvesting process. I’ve corrected that now. The correct URL is http://www.natlib.govt.nz/website-harvest which will appear in future log files.

    I can live with a single process scanning. I’ve released the redirects.

Links to post

Recent Comments

Recent Posts

  • Government investments boost and diversify local economies in lower South Island
    $2.9 million convertible loan for Scapegrace Distillery to meet growing national and international demand $4.5m underwrite to support Silverlight Studios’ project to establish a film studio in Wanaka Gore’s James Cumming Community Centre and Library to be official opened tomorrow with support of $3m from the COVID-19 Response and Recovery ...
    BeehiveBy beehive.govt.nz
    2 hours ago
  • Government future-proofs EV charging
    Transport Minister Michael Wood has today launched the first national EV (electric vehicle) charging strategy, Charging Our Future, which includes plans to provide EV charging stations in almost every town in New Zealand. “Our vision is for Aotearoa New Zealand to have world-class EV charging infrastructure that is accessible, affordable, ...
    BeehiveBy beehive.govt.nz
    18 hours ago
  • World-leading family harm prevention campaign supports young NZers
    Associate Minister for Social Development and Employment Priyanca Radhakrishnan has today launched the Love Better campaign in a world-leading approach to family harm prevention. Love Better will initially support young people through their experience of break-ups, developing positive and life-long attitudes to dealing with hurt. “Over 1,200 young kiwis told ...
    BeehiveBy beehive.govt.nz
    21 hours ago
  • First Chief Clinical Advisor welcomed into Coroners Court
    Hon Rino Tirikatene, Minister for Courts, welcomes the Ministry of Justice’s appointment of Dr Garry Clearwater as New Zealand’s first Chief Clinical Advisor working with the Coroners Court. “This appointment is significant for the Coroners Court and New Zealand’s wider coronial system.” Minister Tirikatene said. Through Budget 2022, the Government ...
    BeehiveBy beehive.govt.nz
    21 hours ago
  • Next steps for affected properties post Cyclone and floods
    The Government via the Cyclone Taskforce is working with local government and insurance companies to build a picture of high-risk areas following Cyclone Gabrielle and January floods. “The Taskforce, led by Sir Brian Roche, has been working with insurance companies to undertake an assessment of high-risk areas so we can ...
    BeehiveBy beehive.govt.nz
    22 hours ago
  • New appointment to Māori Land Court bench
    E te huia kaimanawa, ko Ngāpuhi e whakahari ana i tau aupikinga ki te tihi o te maunga. Ko te Ao Māori hoki e whakanui ana i a koe te whakaihu waka o te reo Māori i roto i te Ao Ture. (To the prized treasure, it is Ngāpuhi who ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Government focus on jobs sees record number of New Zealanders move from Benefits into work
    113,400 exits into work in the year to June 2022 Young people are moving off Benefit faster than after the Global Financial Crisis Two reports released today by the Ministry of Social Development show the Government’s investment in the COVID-19 response helped drive record numbers of people off Benefits and ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Vertical farming partnership has upward momentum
    The Government’s priority to keep New Zealand at the cutting edge of food production and lift our sustainability credentials continues by backing the next steps of a hi-tech vertical farming venture that uses up to 95 per cent less water, is climate resilient, and pesticide-free. Agriculture Minister Damien O’Connor visited ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Conference of Pacific Education Ministers – Keynote Address
    E nga mana, e nga iwi, e nga reo, e nga hau e wha, tena koutou, tena koutou, tena koutou kātoa. Warm Pacific greetings to all. It is an honour to host the inaugural Conference of Pacific Education Ministers here in Tāmaki Makaurau. Aotearoa is delighted to be hosting you ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • New $13m renal unit supports Taranaki patients
    The new renal unit at Taranaki Base Hospital has been officially opened by the Minister of Health Dr Ayesha Verrall this afternoon. Te Huhi Raupō received around $13 million in government funding as part of Project Maunga Stage 2, the redevelopment of the Taranaki Base Hospital campus. “It’s an honour ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Second Poseidon aircraft on home soil
    Defence Minister Andrew Little has marked the arrival of the country’s second P-8A Poseidon aircraft alongside personnel at the Royal New Zealand Air Force’s Base at Ohakea today. “With two of the four P-8A Poseidons now on home soil this marks another significant milestone in the Government’s historic investment in ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Further humanitarian aid for Türkiye and Syria
    Aotearoa New Zealand will provide further humanitarian support to those seriously affected by last month’s deadly earthquakes in Türkiye and Syria, says Foreign Minister Nanaia Mahuta. “The 6 February earthquakes have had devastating consequences, with almost 18 million people affected. More than 53,000 people have died and tens of thousands more ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Community voice to help shape immigration policy
    Migrant communities across New Zealand are represented in the new Migrant Community Reference Group that will help shape immigration policy going forward, Immigration Minister Michael Wood announced today.  “Since becoming Minister, a reoccurring message I have heard from migrants is the feeling their voice has often been missing around policy ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • State Highway 3 project to deliver safer journeys, better travel connections for Taranaki
    Construction has begun on major works that will deliver significant safety improvements on State Highway 3 from Waitara to Bell Block, Associate Minister of Transport Kiri Allan announced today. “This is an important route for communities, freight and visitors to Taranaki but too many people have lost their lives or ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Ginny Andersen appointed as Minister of Police
    Prime Minister Chris Hipkins has today appointed Ginny Andersen as Minister of Police. “Ginny Andersen has a strong and relevant background in this important portfolio,” Chris Hipkins said. “Ginny Andersen worked for the Police as a non-sworn staff member for around 10 years and has more recently been chair of ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Government confirms vital roading reconnections
    Six further bailey bridge sites confirmed Four additional bridge sites under consideration 91 per cent of damaged state highways reopened Recovery Dashboards for impacted regions released The Government has responded quickly to restore lifeline routes after Cyclone Gabrielle and can today confirm that an additional six bailey bridges will ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Foreign Minister Mahuta to meet with China’s new Foreign Minister
    Foreign Minister Nanaia Mahuta departs for China tomorrow, where she will meet with her counterpart, State Councillor and Foreign Minister Qin Gang, in Beijing. This will be the first visit by a New Zealand Minister to China since 2019, and follows the easing of COVID-19 travel restrictions between New Zealand and China. ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Education Ministers from across the Pacific gather in Aotearoa
    Education Ministers from across the Pacific will gather in Tāmaki Makaurau this week to share their collective knowledge and strategic vision, for the benefit of ākonga across the region. New Zealand Education Minister Jan Tinetti will host the inaugural Conference of Pacific Education Ministers (CPEM) for three days from today, ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • State Highway 5 reopens between Napier and Taupō following Cyclone Gabrielle
    A vital transport link for communities and local businesses has been restored following Cyclone Gabrielle with the reopening of State Highway 5 (SH5) between Napier and Taupō, Associate Minister of Transport Kiri Allan says. SH5 reopened to all traffic between 7am and 7pm from today, with closure points at SH2 (Kaimata ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Special Lotto draw raises $11.7 million for Cyclone Gabrielle recovery
    Internal Affairs Minister Barbara Edmonds has thanked generous New Zealanders who took part in the special Lotto draw for communities affected by Cyclone Gabrielle. Held on Saturday night, the draw raised $11.7 million with half of all ticket sales going towards recovery efforts. “In a time of need, New Zealanders ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Government delivers a $3 million funding boost for Building Financial Capability services
    The Government has announced funding of $3 million for providers to help people, and whānau access community-based Building Financial Capability services. “Demand for Financial Capability Services is growing as people face cost of living pressures. Those pressures are increasing further in areas affected by flooding and Cyclone Gabrielle,” Minister for ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Education New Zealand | Manapou ki te Ao – new Chair and member
    Minister of Education, Hon Jan Tinetti, has announced appointments to the Board of Education New Zealand | Manapou ki te Ao. Tracey Bridges is joining the Board as the new Chair and Dr Therese Arseneau will be a new member. Current members Dr Linda Sissons CNZM and Daniel Wilson have ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Scholarships honouring Ngarimu VC and the 28th (Māori) Battalion announced
    Fifteen ākonga Māori from across Aotearoa have been awarded the prestigious Ngarimu VC and 28th (Māori) Battalion Memorial Scholarships and Awards for 2023, Associate Education Minister and Ngarimu Board Chair, Kelvin Davis announced today.  The recipients include doctoral, masters’ and undergraduate students. Three vocational training students and five wharekura students, ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Appointment of Judge of the Court of Appeal and Judge of the High Court
    High Court Judge Jillian Maree Mallon has been appointed a Judge of the Court of Appeal, and District Court Judge Andrew John Becroft QSO has been appointed a Judge of the High Court, Attorney‑General David Parker announced today. Justice Mallon graduated from Otago University in 1988 with an LLB (Hons), and with ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • NZ still well placed to meet global challenges
    The economy has continued to show its resilience despite today’s GDP figures showing a modest decline in the December quarter, leaving the Government well positioned to help New Zealanders face cost of living pressures in a challenging global environment. “The economy had grown strongly in the two quarters before this ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Western Ring Route Complete
    Aucklanders now have more ways to get around as Transport Minister Michael Wood opened the direct State Highway 1 (SH1) to State Highway 18 (SH18) underpass today, marking the completion of the 48-kilometre Western Ring Route (WRR). “The Government is upgrading New Zealand’s transport system to make it safer, more ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Briefings to Incoming Ministers
    This section contains briefings received by incoming ministers following changes to Cabinet in January. Some information may have been withheld in accordance with the Official Information Act 1982. Where information has been withheld that is indicated within the document. ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Teaming up for a stronger, more resilient Fiji
    Aotearoa New Zealand Foreign Affairs Minister Nanaia Mahuta reaffirmed her commitment to working together with the new Government of Fiji on issues of shared importance, including on the prioritisation of climate change and sustainability, at a meeting today, in Nadi. Fiji and Aotearoa New Zealand’s close relationship is underpinned by the Duavata ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Investment in blue highway a lifeline for regional economies and cyclone recovery
    The Government is delivering a coastal shipping lifeline for businesses, residents and the primary sector in the cyclone-stricken regions of Hawkes Bay and Tairāwhiti, Regional Development Minister Kiri Allan announced today. The Rangitata vessel has been chartered for an emergency coastal shipping route between Gisborne and Napier, with potential for ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Next steps developing clean energy for NZ
    The Government will progress to the next stage of the NZ Battery Project, looking at the viability of pumped hydro as well as an alternative, multi-technology approach as part of the Government’s long term-plan to build a resilient, affordable, secure and decarbonised energy system in New Zealand, Energy and Resources ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Statement from the Prime Minister on Stuart Nash
    This morning I was made aware of a media interview in which Minister Stuart Nash criticised a decision of the Court and said he had contacted the Police Commissioner to suggest the Police appeal the decision. The phone call took place in 2021 when he was not the Police Minister. ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • CPTPP Trade Ministers coming to Auckland
    The Government’s sharp focus on trade continues with Aotearoa New Zealand set to host Trade Ministers and delegations from 10 Asia Pacific economies at a meeting of Comprehensive and Progressive Agreement for Trans-Pacific Partnership (CPTPP) Commission members in July, Minister for Trade and Export Growth Damien O’Connor announced today. “New Zealand ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Govt approves $25 million extension for cyclone-affected businesses
    $25 million boost to support more businesses with clean-up in cyclone affected regions, taking total business support to more than $50 million Demand for grants has been strong, with estimates showing applications will exceed the initial $25 million business support package Grants of up to a maximum of $40,000 per ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • More than 160,000 new Kiwis to call NZ home
    80 per cent of 2021 Resident Visas applications have been processed – three months ahead of schedule Residence granted to 160,000 people 84,000 of 85,000 applications have been approved Over 160,000 people have become New Zealand residents now that 80 per cent of 2021 Resident Visa (2021RV) applications have been ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Scholarships propel Kiwi students to NASA
    The Government continues to invest in New Zealand’s burgeoning space industry, today announcing five scholarships for Kiwi Students to undertake internships at NASA’s Jet Propulsion Laboratory (JPL) in California. Economic Development Minister Stuart Nash congratulated Michaela Dobson (University of Auckland), Leah Albrow (University of Canterbury) and Jack Naish, Celine Jane ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Zealand to attend regional security meeting in Australia
    The Lead Coordination Minister for the Government’s Response to the Royal Commission’s Report into the Terrorist Attack on the Christchurch Mosques travels to Melbourne, Australia today to represent New Zealand at the fourth Sub-Regional Meeting on Counter-Terrorism and Transnational Security. “The Government is committed to reducing the threat of terrorism ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Health and safety action plan for ports
    The health and safety practices at our nation’s ports will be improved as part of a new industry-wide action plan, Workplace Relations and Safety, and Transport Minister Michael Wood has announced. “Following the tragic death of two port workers in Auckland and Lyttelton last year, I asked the Port Health ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Bikes and scooters to be exempt from FBT
    Bikes, electric bikes and scooters will be added to the types of transport exempted from fringe benefit tax under changes proposed today. Revenue Minister David Parker said the change would allow bicycles, electric bicycles, scooters, electric scooters, and micro-mobility share services to be exempt from fringe benefit tax where they ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Foreign Affairs Minister to reaffirm our close relationship with Fiji
    Foreign Affairs Minister Nanaia Mahuta will hold bilateral meetings with Fiji this week. The visit will be her first to the country since the election of the new coalition Government led by Prime Minister and Minister of Foreign Affairs Sitiveni Rabuka. The visit will be an opportunity to meet kanohi ki ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New legislation to streamline Cyclone recovery
    The Government is introducing the Severe Weather Emergency Legislation Bill to ensure the recovery and rebuild from Cyclone Gabrielle is streamlined and efficient with unnecessary red tape removed. The legislation is similar to legislation passed following the Christchurch and Kaikōura earthquakes that modifies existing legislation in order to remove constraints ...
    BeehiveBy beehive.govt.nz
    1 week ago

Page generated in The Standard by Wordpress at 2023-03-22T23:16:38+00:00