web analytics

Dealing with the DIA website-harvest parasite

Written By: - Date published: 2:27 am, February 3rd, 2011 - 12 comments
Categories: admin, blogs, humour, inoculation, interweb, scoundrels, Spying, The Standard - Tags: ,

Periodically I run a scan to identify network parasites that are sucking up our bandwidth and processing resources in excess. Of course I leave the benign parasites that provide search facilities alone. Like the ubiquitous googlebot and its other benign brethren. But I stomp on the nasties and inoculate the system against them.

It could be spambots trying to insert comments and requiring us to expend effort in cleaning up the garbage. There denying from .htaccess is your friend and vast swathes of the IP’s from Russia, China and some other regions are excluded from reading anything at all.

It could be virus slaved botnets mindlessly trying to exploit our contact us and contribute pages to send e-mails. There I find that a relatively simple captcha is sufficient to deter them.

Feedbots picking up RSS feeds used to be a problem. But since I put some limits on how much they can pick up and how old it can be, that problem has disappeared.

It could be a out of control spider trying to crawl all over our web pages in a single cataclysmic event. I find that putting restrictions on how often an IP number can connect within a given period of time tends to squelch them. Because of the nature of the other beasts on the net I have that to just harm the really out of control processes with grey-listing. The system starts refusing to accept new connections after a while while very slowwwly feeding the bot a dribble of data and holding them up.

But that leaves the processes that don’t really go out of control but are rather badly mannered. Specifically they don’t appear to obey the rules of my robots.txt particularly well. Usually I just put in rules to ban them permanently

Tonight the biggest parasite appears to have been a New Zealand government department – the Department of Internal Affairs. Processes mounted on IP numbers 192.122.171.75 and 192.122.171.11 ran very high number of queries on the site over a couple of hours. It advertised itself in the user agent as being “heritrix/1.14.1 + http://www.dia.govt.nz/website-harvest”.

Well ok, I looked at the web page and it doesn’t exist. I googled for website harvest at the DIA – nothing. I looked on the DIA website to no avail. I looked up heritrix which asserts that it follows the robots.txt. Thats not what I saw especially when it was picking up from two IPs.

At that point I was tired of being nice to whoever hadn’t put up the web page explaining why they were tapping our website. If anyone does know then put a comment here. I also didn’t know where to send my irritated e-mail to.

I’m hesitant to exclude a branch of the government from archiving our site for prosperity and historians (which is what heritrix is designed for). But I sure as hell want to be able to know what they’re doing. I’d also like them to moderate their behavior and obey the rules in the robots.txt somewhat better.

Rather than doing what I would have usually done and simply denying them access, I decided to draw their attention to their lack of a explanatory webpage.

I used a RewriteRule in .htaccess to permanently redirect every query from them to the missing webpage. I’m now happily off to bed contented that somewhere in the DIA there is a file system or database slowly filling up with copies of their missing page.

12 comments on “Dealing with the DIA website-harvest parasite”

  1. Jenny 1

    Good for you Lynne, With the $millions dished out to our secretive spooks to prod and pry and fish for our private details and invest our private files.

    It is about time someone punctured their arrogant smug bubble again. (name check Waihopai here).

    capcha – “sequence” (as in code)

    • lprent 1.1

      I don’t think it is the spooks. Too overt for a starter. I figured it out to be a part of something like the wayback engine.

  2. Tim H 2

    Don’t know much about it, but you might want to look at natlib.govt.nz/website-harvest. What with them being subsumed into DIA, I reckon that might have something to do with it. Cock up rather than evil spooks, possibly.

    There’s contact addresses and all there.

    • lprent 2.1

      That will be it. I did this very early this morning. I was more interested in getting their pickup off my screen so I could see the effect of other parasites. Dropping what my site delivered to them from tens of k per page to 596 bytes did it – and amused me as well

    • lprent 2.2

      Google for natlib website-harvest gets it. They need to look at their SEO. website-harvest alone should have been enough for me to locate it.

      http://natlib.govt.nz/website-harvest

      I can’t find a mention of them running a sweep at present. In fact the whole website looks out of date. I guess that the DIA is doing what is expected and not doing the required work for natlib.

      • Idiot/Savant 2.2.1

        They run a sweep fairly regularly – they have a program to archive the top blogs for posterity. Though IIRC they asked permission (which obviously I granted).

        • lprent 2.2.1.1

          It isn’t running through the site that bugs me. I think it is a worthy project.

          It is that they were pulling it too fast whilst people were still active on the site. In particular authors were writing posts after work (it was after Marty commented that he’d had a few outages that I went and looked at what was using more than usual resources). Also that when I went to look for whoever it was, the URL was invalid – that tends to really annoy sysops because it wastes our time.

          They didn’t ask me, but that is because we’re in a .nz domain space rather than .com. There is some legislation covering that.

  3. prism 3

    Is this a follow-on from the political meddling with NZ Archives etc? Is it part of results from trying to reduce quality of service and the separateness of our old records to make short-term savings? We saw how uninterested even hostile previous Nat politicians have been to cultural needs of us as an advanced, educated society, such as in pushing RadioNZ out of their premises, specially built for their purposes.

    • lprent 3.1

      Is this a follow-on from the political meddling with NZ Archives etc?

      I suspect so. What annoys me is that the page pickup rate is too high and it was happening prior to midnight. We get a lot of traffic in the evenings and it runs all the way up to the wee hours of the morning.

      A bot picking up pages virtually every second may do great things for their efficiency. But it is a pain in the arse for sites with a lot of traffic and a very large number of pages. I suspect that it isn’t an issue for something like the NZ Herald site where they have quite a lot of hardware redundancy. But for us where we run on quite limited servers it is annoying.

      The norm for bots like baidubot or googlebot is to pick up a page every 20-30 seconds on any single site during a full scan and do that continuously. That pushes it neatly into the background processing. Of course there are usually a couple of dozen bots running sweeps at any particular point in time. So with the DIA picking up virtually every second it effectively doubles our background processing. I tend to block out servers that pick up every second.

      In practice we use a sitemap (listed in the virtual robots.txt) and push notifications for the major bots to tell them when a page has been altered and if they should pick it up. But that just covers googlebot, bing, yahoo, and ask. Bearing in mind that the bots are well over 50% of the in the day traffic (and 95% of the overnight), I’m starting to consider excluding all bots apart the ones we choose to inform.

  4. I had exactly this happen with a political candidate site, except it was the National Library of Australia, and yes, for a wayback / archive operation.

    We were paying a commercial host and within an hour their spider had used our entire month’s bandwidth allocation and got the site shut down – in the middle of a campaign.

    Figuring they must have the site by now (it was only a few dozen pages!) and they’d leave us alone and intending to complain later, I just paid for additional bandwidth. It came back, did it again. Within 24 hours we were off air again.

    I wrote them a very angry letter (even by my standards) and instead of saying “sorry, we’ll look at what it’s doing” they got lawyers involved. Assholes.

    Bill them for the bandwidth, time and inconvenience and send them off to the debt collectors if they don’t pay up.

    • lprent 4.1

      At present they are sucking up nothing much apart from instructions to read their own site. But the server has an allocation of about 700GB for bandwidth and we’re well under that. Just inconvenient and a bit stupid.

  5. lprent 5

    Finally caught up with my e-mail (I only read it a couple of times per day). There was a good e-mail of which the relevant bit was:-

    We have harvested the site before but unfortunately last night there was a glitch, which set off two harvests instead of one. This would explain the extra traffic on your site last night.

    Compounding that there was a mistake in our website address, which is why you couldn’t find the page that explains our web harvesting process. I’ve corrected that now. The correct URL is http://www.natlib.govt.nz/website-harvest which will appear in future log files.

    I can live with a single process scanning. I’ve released the redirects.

Links to post

Recent Comments

Recent Posts

  • Justice Minister represents New Zealand at Berlin nuclear disarmament summit
    Justice Minister Andrew Little will travel to Berlin tomorrow to represent New Zealand at a high-level summit on nuclear disarmament. This year, the Nuclear Non-Proliferation Treaty (NPT) celebrates 50 years since it entered into force. “New Zealand’s proud record and leadership on nuclear disarmament is unwavering, so it’s important we are present ...
    BeehiveBy beehive.govt.nz
    15 hours ago
  • Prime Minister to visit Fiji and Australia
    Prime Minister Jacinda Ardern will visit two of New Zealand’s most important Pacific partners, Fiji and Australia, next week. The visit to Fiji will be the first by a New Zealand Prime Minister in four years and comes during the 50th anniversary of Fijian independence and diplomatic relations between our ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Next steps in Criminal Cases Review Commission announced
    Justice Minister Andrew Little and New Zealand First MP Darroch Ball, have today announced the appointment of the Chief Commissioner of the Criminal Cases Review Commission (CCRC), the location, and the membership of the Establishment Advisory Group. Colin Carruthers QC has been appointed Chief Commissioner of the CCRC for an ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Horticultural Ahuwhenua Trophy finalists announced
    Māori Development Minister Hon Nanaia Mahuta and Agriculture Minister Hon Damien O’Connor co-announced the first horticultural finalists for the Ahuwhenua Trophy celebrating excellence in the Māori agricultural sector.  The three finalists are Ngai Tukairangi Trust from Mt Maunganui, Otama Marere Trust from Tauranga, and Hineora Orchard Te Kaha 15B Ahuwhenua ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • New support for students with dyslexia
    A new kete of resources to strengthen support for students with dyslexia will provide extra tools for the new Learning Support Coordinators (LSCs) as they start in schools, Associate Education Minister Tracey Martin announced today. The Minister launched the kete in Wellington this morning, at the first of three induction ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Rental reforms progress to select committee stage
    The Government continues to make progress on a fairer and more secure rental market for renters and landlords with the First Reading of the Residential Tenancies Amendment Bill and its referral to the Social Services and Community Select Committee.  “Now is the opportunity for landlords, tenants and others who want ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Papua New Guinea Prime Minister to visit New Zealand
    Papua New Guinea’s Prime Minister Hon James Marape will visit New Zealand from 21-25 February, Prime Minister Jacinda Ardern announced today. “New Zealand and Papua New Guinea have a warm and friendly relationship. I look forward to welcoming Prime Minister Marape here and strengthening the relationship between our two countries,” ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Free school lunches served up to thousands
    Thousands of children have begun receiving a free lunch on every day of the school week, Prime Minister Jacinda Ardern said. The Government’s free and healthy school lunch programme is under way for 7,000 students at 31 schools in Hawke’s Bay / Tairāwhiti and Bay of Plenty / Waiariki, extending ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Social Wellbeing Agency replaces Social Investment Agency with new approach
    The Minister for Social Development Carmel Sepuloni today announced a new approach that continues to broaden the Government’s social sector focus from a narrow, investment approach to one centred on people and wellbeing. Minister Sepuloni said redefining the previous approach to social investment by combining science, data and lived experience ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Government to strengthen protections for whistleblowers
    The Government is strengthening the Protected Disclosures Act to provide better protection for whistle blowers, Minister of State Services Chris Hipkins said today. “The Protected Disclosures Act is meant to encourage people to speak up about serious wrongdoing in the workplace and protect them from losing their jobs or being ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • PM speech at Parliamentary Chinese New Year celebration 2020
    Nǐn hǎo (Hello in Mandarin). Xīn Nián Kuài Lè (Happy New Year in Mandarin) Néi Hóu (Hello in Cantonese). Sun Nin Fai Lok (Happy New Year in Cantonese) Tēnā koutou, tēnā koutou, tēnā koutou katoa. Thank you for your invitation to attend this celebration today. I would like to acknowledge ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • IPANZ Annual Address
    Kia ora. Ngā mihi nui ki a koutou katoa. Nau mai haere mai ki te Whare Pāremata. E ngā mana whenua ki tēnei rohe Taranaki Whānui, Te Upoko o Te Ika, Ngāti Toa Rangatira, Ngāti Raukawa – kei te mihi, kei te mihi, kei te mihi. E ngā mana, e ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • 2020 IPANZ Annual Address
    Kia ora. Ngā mihi nui ki a koutou katoa. Nau mai haere mai ki te Whare Pāremata. E ngā mana whenua ki tēnei rohe Taranaki Whānui, Te Upoko o Te Ika, Ngāti Toa Rangatira, Ngāti Raukawa – kei te mihi, kei te mihi, kei te mihi. E ngā mana, e ngā ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Tougher penalties for gun crime a step closer
    Tougher penalties for gun crime are a step closer with the passage of firearms reform legislation through another stage in Parliament. The Arms Legislation Bill has tonight passed its Second Reading. “The changes have one objective - to prevent firearms falling into the wrong hands,” said Police Minister Stuart Nash. ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Arms Legislation Bill: Second Reading
    Introduction Mr Speaker We all know why we are here today. It has been a long journey. The journey did not actually begin on 15 March 2019. It began on 30 June 1997. Almost 23 years ago, Justice Sir Thomas Thorp told us what was wrong with our firearms legislation. ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • New era for vocational education
    The Government’s work to put trades and vocational education back on the agenda took another major step forward today with the passing of the Education (Vocational Education and Training Reform) Amendment Bill, Education Minister Chris Hipkins said. “This is a watershed day for trades and vocational education. These law changes formalise ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Bill to Amend the Greater Christchurch Regeneration Act
    Speeding up the return of Christchurch regeneration activities to local leadership is behind the Greater Christchurch Regeneration Amendment Bill introduced to Parliament today by Minister Megan Woods. “As we approach nine years since the February 2011 earthquake in Canterbury, and with the transition to local leadership well underway, the time ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Milford Track to partly reopen after storm damage
    Hundreds of New Zealanders and international visitors will be able to get back out into nature with the Milford Track partially reopening next week, after extensive assessments and repairs, the Minister of Conservation Eugenie Sage announced today. The popular Great Walk has been closed since 3 February after an extreme ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Government drives low-emissions transport momentum
    Up to 110 new EV chargers nationwide in cities and regions 50 electric vehicles for ride-sharing The Government is helping deliver more infrastructure and options for low emissions transport through new projects, Energy and Resources Minister Dr Megan Woods says. Tauranga, Nelson, Levin, New Plymouth and Oamaru are just some ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Kiwis better off under Coalition Government
    New Zealanders are increasingly better off under this Government as wages rise and families have more disposable income, Finance Minister Grant Robertson says. Stats NZ reported today that average household disposable incomes after housing costs rose 4.9% in 2019. This was the highest rise in four years and came as ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Another step towards restoring rights for screen production workers
    All New Zealanders need to have their voices heard at work to ensure we have an inclusive and productive economy. Today we introduce a Bill to do this for workers in the New Zealand screen industry, Workplace Relations and Safety Minister Iain Lees-Galloway says. “The Screen Industry Workers Bill will ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Enhanced Taskforce Green for Southland and South Otago
    The Government has announced further help for the Southland and Otago regions to speed up recovery efforts from the floods.  “I’ve approved Enhanced Taskforce Green (ETFG), making $500,000 available to help with the clean-up in Fiordland, Southland, and the Clutha district in Otago,” Minister for Social Development Carmel Sepuloni said.  ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Employers and Industry take the lead to connect students to vocational education
    Following the announcement that more than 340 schools will be funded to run events promoting vocational education, the Government has announced it will fund a further 257 events to be run by employers and industry. “These industry-run events will allow more than 30,000 students to connect with more than 2,000 ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Rental reforms a step closer with the introduction of the Residential Tenancies Amendment Bill
    Today the Government is making progress on a fairer and more secure rental market for renters and landlords with the introduction of the Residential Tenancies Amendment Bill in Parliament.  “This Bill includes a series of reforms to improve the wellbeing of the 609,700 households that live in rented homes, and ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Biosecurity Minister announces world first eradication of pea weevil
    A Government programme to wipe out pea weevil has achieved a world first, with Biosecurity Minister Damien O’Connor today announcing the successful eradication of the noxious pest from Wairarapa. This means the nearly four-year ban on pea plants and pea straw was lifted today. Commercial and home gardeners can again grow ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Temporary Accommodation Service activated for Southland flooding
    Southland residents hit by flooding caused by heavy rainfall can now access help finding temporary accommodation with the Government activating the Temporary Accommodation Service, Minister of Civil Defence Peeni Henare announced today. “The Temporary Accommodation Service (TAS) has been activated by the Ministry of Business, Innovation and Employment to help ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Bridges: Over-hyped and under-delivered
    “Is that it?” That’s Finance Minister Grant Robertson’s response to Simon Bridges’ much-hyped economic speech today. “Simon Bridges just gave the most over-hyped and under-delivered speech that I can remember during my time in politics,” Grant Robertson said. “It’s not surprising. Simon Bridges literally said on the radio this morning ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Police to trial eye in the sky in Christchurch
    A trial deployment of the Police Eagle helicopter in Christchurch will test whether the aircraft would make a significant difference to crime prevention and community safety. “The Bell 429 helicopter will be based in Christchurch for five weeks, from 17 February to 20 March,” said Police Minister Stuart Nash. “The ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Momentum of trade talks continues with visits to promote Pacific and Middle East links
    The Government has kept up the pace of its work to promote New Zealand’s trade interests and diversify our export markets, with visits to Fiji and the United Arab Emirates (UAE) by Trade and Export Growth Minister David Parker. Building momentum to bring the PACER Plus trade and development agreement ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Coalition Govt’s investment in Customs nets record drugs haul: 3 tonnes stopped at borders in 2019
    The Coalition Government’s investment in a strong border and disrupting transnational organised crime produced record results for stopping drugs in 2019, says Minister of Customs Jenny Salesa. The illegal drugs were seized at the New Zealand border by Customs, and overseas by Customs’ international border partners before the drugs could ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Separated scenic cycleway starts
    Transport Minister Phil Twyford today kicked off construction of a separated cycleway alongside Tamaki Drive. A two-way separated cycleway will be built along the northern side of Tamaki Drive, between the Quay Street Cycleway extension and Ngapipi Road. There will be a separate walking path alongside. Phil Twyford said giving ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Earthquake-Prone Building loan scheme: eligibility criteria announced
    Owner-occupiers of unit and apartments living in earthquake-prone buildings will have certainty about the financial support they’ll be eligible for with the release of criteria for an upcoming assistance scheme, Minister for Building and Construction Jenny Salesa says. The Residential Earthquake-Prone Building Financial Assistance Scheme will help unit owners facing ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Travel restrictions to remain in place as coronavirus precaution
    Temporary restrictions on travel from China will remain in place as a precautionary measure to protect against the spread of the COVID-19 coronavirus. The restrictions which prevent foreign nationals travelling from, or transiting through, mainland China from entering New Zealand have been extended for a further 8 days. This position ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Over $1 million to help Tairāwhiti youth into employment
    Employment Minister Willie Jackson announced today that Tairāwhiti rangatahi will benefit from an investment made by the Government’s He Poutama Rangatahi (HPR) scheme. The funding will go to the Tautua Village, Kauneke programme and the Matapuna Supported Employment Programme which will fund 120 rangatahi over two years. “Both programmes work ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • School attendance has to improve
    All parents and caregivers need to ensure that their children go to school unless they are sick, Associate Education Minister Tracey Martin said today. “The school attendance results for 2019 show, across the board, a drop in the number of students going to school regularly,” the Minister says. “Apart from ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Crown and Moriori sign a Deed of Settlement
    A Deed of Settlement agreeing redress for historical Treaty claims has been signed by the Crown and Moriori at Kōpinga Marae on Rēkohu (Chatham Islands) today, Minister for Treaty of Waitangi Negotiations Andrew Little has announced. Moriori have a tradition of peace that extends back over 600 years. This settlement ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Waikato Expressway driving towards completion
    Transport Minister Phil Twyford today with Māori King Tuheitia Pōtatau Te Wherowhero VII officially opened the country’s newest road, the $384 million Huntly section of the Waikato Expressway. The 15km four-lane highway with side and central safety barriers takes State Highway 1 east of Huntly town, across lowlands and streams ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • 3400 New Zealanders treated in first year of new hepatitis C treatment
    The rapid uptake of life-saving new hepatitis C medicine Maviret since it was funded by PHARMAC a year ago means the elimination of the deadly disease from this country is a realistic goal, Health Minister David Clark says. Hepatitis C is a blood-borne virus which attacks the liver, proving fatal ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Kaupapa Māori approach for homelessness
      Kaupapa Māori will underpin the Government’s new plan to deal with homelessness announced by the Prime Minister in Auckland this morning. “Māori are massively overrepresented among people experiencing homelessness, so, to achieve different outcomes for Māori, we have to do things very differently,” says the Minister of Māori Development ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government steps up action to prevent homelessness
    1000 new transitional housing places delivered by end of year to reduce demand for emergency motel accommodation. Introduce 25% of income payment, after 7 days, for those in emergency motel accommodation to bring in line with other forms of accommodation support. Over $70m extra to programmes that prevents those at ...
    BeehiveBy beehive.govt.nz
    1 week ago