Dealing with the DIA website-harvest parasite

Written By: - Date published: 2:27 am, February 3rd, 2011 - 13 comments
Categories: admin, blogs, humour, inoculation, interweb, scoundrels, Spying, The Standard - Tags: ,

Periodically I run a scan to identify network parasites that are sucking up our bandwidth and processing resources in excess. Of course I leave the benign parasites that provide search facilities alone. Like the ubiquitous googlebot and its other benign brethren. But I stomp on the nasties and inoculate the system against them.

It could be spambots trying to insert comments and requiring us to expend effort in cleaning up the garbage. There denying from .htaccess is your friend and vast swathes of the IP’s from Russia, China and some other regions are excluded from reading anything at all.

It could be virus slaved botnets mindlessly trying to exploit our contact us and contribute pages to send e-mails. There I find that a relatively simple captcha is sufficient to deter them.

Feedbots picking up RSS feeds used to be a problem. But since I put some limits on how much they can pick up and how old it can be, that problem has disappeared.

It could be a out of control spider trying to crawl all over our web pages in a single cataclysmic event. I find that putting restrictions on how often an IP number can connect within a given period of time tends to squelch them. Because of the nature of the other beasts on the net I have that to just harm the really out of control processes with grey-listing. The system starts refusing to accept new connections after a while while very slowwwly feeding the bot a dribble of data and holding them up.

But that leaves the processes that don’t really go out of control but are rather badly mannered. Specifically they don’t appear to obey the rules of my robots.txt particularly well. Usually I just put in rules to ban them permanently

Tonight the biggest parasite appears to have been a New Zealand government department – the Department of Internal Affairs. Processes mounted on IP numbers 192.122.171.75 and 192.122.171.11 ran very high number of queries on the site over a couple of hours. It advertised itself in the user agent as being “heritrix/1.14.1 + http://www.dia.govt.nz/website-harvest”.

Well ok, I looked at the web page and it doesn’t exist. I googled for website harvest at the DIA – nothing. I looked on the DIA website to no avail. I looked up heritrix which asserts that it follows the robots.txt. Thats not what I saw especially when it was picking up from two IPs.

At that point I was tired of being nice to whoever hadn’t put up the web page explaining why they were tapping our website. If anyone does know then put a comment here. I also didn’t know where to send my irritated e-mail to.

I’m hesitant to exclude a branch of the government from archiving our site for prosperity and historians (which is what heritrix is designed for). But I sure as hell want to be able to know what they’re doing. I’d also like them to moderate their behavior and obey the rules in the robots.txt somewhat better.

Rather than doing what I would have usually done and simply denying them access, I decided to draw their attention to their lack of a explanatory webpage.

I used a RewriteRule in .htaccess to permanently redirect every query from them to the missing webpage. I’m now happily off to bed contented that somewhere in the DIA there is a file system or database slowly filling up with copies of their missing page.

13 comments on “Dealing with the DIA website-harvest parasite ”

  1. Jenny 1

    Good for you Lynne, With the $millions dished out to our secretive spooks to prod and pry and fish for our private details and invest our private files.

    It is about time someone punctured their arrogant smug bubble again. (name check Waihopai here).

    capcha – “sequence” (as in code)

    • lprent 1.1

      I don’t think it is the spooks. Too overt for a starter. I figured it out to be a part of something like the wayback engine.

  2. Tim H 2

    Don’t know much about it, but you might want to look at natlib.govt.nz/website-harvest. What with them being subsumed into DIA, I reckon that might have something to do with it. Cock up rather than evil spooks, possibly.

    There’s contact addresses and all there.

    • lprent 2.1

      That will be it. I did this very early this morning. I was more interested in getting their pickup off my screen so I could see the effect of other parasites. Dropping what my site delivered to them from tens of k per page to 596 bytes did it – and amused me as well

    • lprent 2.2

      Google for natlib website-harvest gets it. They need to look at their SEO. website-harvest alone should have been enough for me to locate it.

      http://natlib.govt.nz/website-harvest

      I can’t find a mention of them running a sweep at present. In fact the whole website looks out of date. I guess that the DIA is doing what is expected and not doing the required work for natlib.

      • Idiot/Savant 2.2.1

        They run a sweep fairly regularly – they have a program to archive the top blogs for posterity. Though IIRC they asked permission (which obviously I granted).

        • lprent 2.2.1.1

          It isn’t running through the site that bugs me. I think it is a worthy project.

          It is that they were pulling it too fast whilst people were still active on the site. In particular authors were writing posts after work (it was after Marty commented that he’d had a few outages that I went and looked at what was using more than usual resources). Also that when I went to look for whoever it was, the URL was invalid – that tends to really annoy sysops because it wastes our time.

          They didn’t ask me, but that is because we’re in a .nz domain space rather than .com. There is some legislation covering that.

  3. prism 3

    Is this a follow-on from the political meddling with NZ Archives etc? Is it part of results from trying to reduce quality of service and the separateness of our old records to make short-term savings? We saw how uninterested even hostile previous Nat politicians have been to cultural needs of us as an advanced, educated society, such as in pushing RadioNZ out of their premises, specially built for their purposes.

    • lprent 3.1

      Is this a follow-on from the political meddling with NZ Archives etc?

      I suspect so. What annoys me is that the page pickup rate is too high and it was happening prior to midnight. We get a lot of traffic in the evenings and it runs all the way up to the wee hours of the morning.

      A bot picking up pages virtually every second may do great things for their efficiency. But it is a pain in the arse for sites with a lot of traffic and a very large number of pages. I suspect that it isn’t an issue for something like the NZ Herald site where they have quite a lot of hardware redundancy. But for us where we run on quite limited servers it is annoying.

      The norm for bots like baidubot or googlebot is to pick up a page every 20-30 seconds on any single site during a full scan and do that continuously. That pushes it neatly into the background processing. Of course there are usually a couple of dozen bots running sweeps at any particular point in time. So with the DIA picking up virtually every second it effectively doubles our background processing. I tend to block out servers that pick up every second.

      In practice we use a sitemap (listed in the virtual robots.txt) and push notifications for the major bots to tell them when a page has been altered and if they should pick it up. But that just covers googlebot, bing, yahoo, and ask. Bearing in mind that the bots are well over 50% of the in the day traffic (and 95% of the overnight), I’m starting to consider excluding all bots apart the ones we choose to inform.

  4. I had exactly this happen with a political candidate site, except it was the National Library of Australia, and yes, for a wayback / archive operation.

    We were paying a commercial host and within an hour their spider had used our entire month’s bandwidth allocation and got the site shut down – in the middle of a campaign.

    Figuring they must have the site by now (it was only a few dozen pages!) and they’d leave us alone and intending to complain later, I just paid for additional bandwidth. It came back, did it again. Within 24 hours we were off air again.

    I wrote them a very angry letter (even by my standards) and instead of saying “sorry, we’ll look at what it’s doing” they got lawyers involved. Assholes.

    Bill them for the bandwidth, time and inconvenience and send them off to the debt collectors if they don’t pay up.

    • lprent 4.1

      At present they are sucking up nothing much apart from instructions to read their own site. But the server has an allocation of about 700GB for bandwidth and we’re well under that. Just inconvenient and a bit stupid.

  5. lprent 5

    Finally caught up with my e-mail (I only read it a couple of times per day). There was a good e-mail of which the relevant bit was:-

    We have harvested the site before but unfortunately last night there was a glitch, which set off two harvests instead of one. This would explain the extra traffic on your site last night.

    Compounding that there was a mistake in our website address, which is why you couldn’t find the page that explains our web harvesting process. I’ve corrected that now. The correct URL is http://www.natlib.govt.nz/website-harvest which will appear in future log files.

    I can live with a single process scanning. I’ve released the redirects.

Links to post

Recent Comments

Recent Posts

  • Anzac Commemorative Address, Buttes New British Cemetery Belgium
    Ambassador Millar, Burgemeester, Vandepitte, Excellencies, military representatives, distinguished guests, ladies and gentlemen – good morning and welcome to this sacred Anzac Day dawn service.  It is an honour to be here on behalf of the Government and people of New Zealand at Buttes New British Cemetery, Polygon Wood – a deeply ...
    BeehiveBy beehive.govt.nz
    6 hours ago
  • Anzac Commemorative Address – NZ National Service, Chunuk Bair
    Distinguished guests -   It is an honour to return once again to this site which, as the resting place for so many of our war-dead, has become a sacred place for generations of New Zealanders.   Our presence here and at the other special spaces of Gallipoli is made ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Anzac Commemorative Address – Dawn Service, Gallipoli, Türkiye
    Mai ia tawhiti pamamao, te moana nui a Kiwa, kua tae whakaiti mai matou, ki to koutou papa whenua. No koutou te tapuwae, no matou te tapuwae, kua honoa pumautia.   Ko nga toa kua hinga nei, o te Waipounamu, o te Ika a Maui, he okioki tahi me o ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • PM announces changes to portfolios
    Paul Goldsmith will take on responsibility for the Media and Communications portfolio, while Louise Upston will pick up the Disability Issues portfolio, Prime Minister Christopher Luxon announced today. “Our Government is relentlessly focused on getting New Zealand back on track. As issues change in prominence, I plan to adjust Ministerial ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • New catch limits for unique fishery areas
    Recreational catch limits will be reduced in areas of Fiordland and the Chatham Islands to help keep those fisheries healthy and sustainable, Oceans and Fisheries Minister Shane Jones says. The lower recreational daily catch limits for a range of finfish and shellfish species caught in the Fiordland Marine Area and ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Minister welcomes hydrogen milestone
    Energy Minister Simeon Brown has welcomed an important milestone in New Zealand’s hydrogen future, with the opening of the country’s first network of hydrogen refuelling stations in Wiri. “I want to congratulate the team at Hiringa Energy and its partners K one W one (K1W1), Mitsui & Co New Zealand ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Urgent changes to system through first RMA Amendment Bill
    The coalition Government is delivering on its commitment to improve resource management laws and give greater certainty to consent applicants, with a Bill to amend the Resource Management Act (RMA) expected to be introduced to Parliament next month. RMA Reform Minister Chris Bishop has today outlined the first RMA Amendment ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Overseas decommissioning models considered
    Overseas models for regulating the oil and gas sector, including their decommissioning regimes, are being carefully scrutinised as a potential template for New Zealand’s own sector, Resources Minister Shane Jones says. The Coalition Government is focused on rebuilding investor confidence in New Zealand’s energy sector as it looks to strengthen ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Release of North Island Severe Weather Event Inquiry
    Emergency Management and Recovery Minister Mark Mitchell has today released the Report of the Government Inquiry into the response to the North Island Severe Weather Events. “The report shows that New Zealand’s emergency management system is not fit-for-purpose and there are some significant gaps we need to address,” Mr Mitchell ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Justice Minister to attend Human Rights Council
    Justice Minister Paul Goldsmith is today travelling to Europe where he’ll update the United Nations Human Rights Council on the Government’s work to restore law and order.  “Attending the Universal Periodic Review in Geneva provides us with an opportunity to present New Zealand’s human rights progress, priorities, and challenges, while ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Patterson reopens world’s largest wool scouring facility
    Associate Agriculture Minister, Mark Patterson, formally reopened the world’s largest wool processing facility today in Awatoto, Napier, following a $50 million rebuild and refurbishment project. “The reopening of this facility will significantly lift the economic opportunities available to New Zealand’s wool sector, which already accounts for 20 per cent of ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Speech to the Southland Otago Regional Engineering Collective Summit, 18 April 2024
    Hon Andrew Bayly, Minister for Small Business and Manufacturing  At the Southland Otago Regional Engineering Collective (SOREC) Summit, 18 April, Dunedin    Ngā mihi nui, Ko Andrew Bayly aho, Ko Whanganui aho    Good Afternoon and thank you for inviting me to open your summit today.    I am delighted ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Government to introduce revised Three Strikes law
    The Government is delivering on its commitment to bring back the Three Strikes legislation, Associate Justice Minister Nicole McKee announced today. “Our Government is committed to restoring law and order and enforcing appropriate consequences on criminals. We are making it clear that repeat serious violent or sexual offending is not ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • New diplomatic appointments
    Foreign Minister Winston Peters has today announced four new diplomatic appointments for New Zealand’s overseas missions.   “Our diplomats have a vital role in maintaining and protecting New Zealand’s interests around the world,” Mr Peters says.    “I am pleased to announce the appointment of these senior diplomats from the ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Humanitarian support for Ethiopia and Somalia
    New Zealand is contributing NZ$7 million to support communities affected by severe food insecurity and other urgent humanitarian needs in Ethiopia and Somalia, Foreign Minister Rt Hon Winston Peters announced today.   “Over 21 million people are in need of humanitarian assistance across Ethiopia, with a further 6.9 million people ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Arts Minister congratulates Mataaho Collective
    Minister for Arts, Culture and Heritage Paul Goldsmith is congratulating Mataaho Collective for winning the Golden Lion for best participant in the main exhibition at the Venice Biennale. "Congratulations to the Mataaho Collective for winning one of the world's most prestigious art prizes at the Venice Biennale.  “It is good ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Supporting better financial outcomes for Kiwis
    The Government is reforming financial services to improve access to home loans and other lending, and strengthen customer protections, Commerce and Consumer Affairs Minister Andrew Bayly and Housing Minister Chris Bishop announced today. “Our coalition Government is committed to rebuilding the economy and making life simpler by cutting red tape. We are ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Trade relationship with China remains strong
    “China remains a strong commercial opportunity for Kiwi exporters as Chinese businesses and consumers continue to value our high-quality safe produce,” Trade and Agriculture Minister Todd McClay says.   Mr McClay has returned to New Zealand following visits to Beijing, Harbin and Shanghai where he met ministers, governors and mayors and engaged in trade and agricultural events with the New ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • PM’s South East Asia mission does the business
    Prime Minister Christopher Luxon has completed a successful trip to Singapore, Thailand and the Philippines, deepening relationships and capitalising on opportunities. Mr Luxon was accompanied by a business delegation and says the choice of countries represents the priority the New Zealand Government places on South East Asia, and our relationships in ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • $41m to support clean energy in South East Asia
    New Zealand is demonstrating its commitment to reducing global greenhouse emissions, and supporting clean energy transition in South East Asia, through a contribution of NZ$41 million (US$25 million) in climate finance to the Asian Development Bank (ADB)-led Energy Transition Mechanism (ETM). Prime Minister Christopher Luxon and Climate Change Minister Simon Watts announced ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Minister releases Fast-track stakeholder list
    The Government is today releasing a list of organisations who received letters about the Fast-track applications process, says RMA Reform Minister Chris Bishop. “Recently Ministers and agencies have received a series of OIA requests for a list of organisations to whom I wrote with information on applying to have a ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Judicial appointments announced
    Attorney-General Judith Collins today announced the appointment of Wellington Barrister David Jonathan Boldt as a Judge of the High Court, and the Honourable Justice Matthew Palmer as a Judge of the Court of Appeal. Justice Boldt graduated with an LLB from Victoria University of Wellington in 1990, and also holds ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Education Minister heads to major teaching summit in Singapore
    Education Minister Erica Stanford will lead the New Zealand delegation at the 2024 International Summit on the Teaching Profession (ISTP) held in Singapore. The delegation includes representatives from the Post Primary Teachers’ Association (PPTA) Te Wehengarua and the New Zealand Educational Institute (NZEI) Te Riu Roa.  The summit is co-hosted ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Value of stopbank project proven during cyclone
    A stopbank upgrade project in Tairawhiti partly funded by the Government has increased flood resilience for around 7000ha of residential and horticultural land so far, Regional Development Minister Shane Jones says. Mr Jones today attended a dawn service in Gisborne to mark the end of the first stage of the ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Anzac commemorations, Türkiye relationship focus of visit
    Foreign Affairs Minister Winston Peters will represent the Government at Anzac Day commemorations on the Gallipoli Peninsula next week and engage with senior representatives of the Turkish government in Istanbul.    “The Gallipoli campaign is a defining event in our history. It will be a privilege to share the occasion ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Minister to Europe for OECD meeting, Anzac Day
    Science, Innovation and Technology and Defence Minister Judith Collins will next week attend the OECD Science and Technology Ministerial conference in Paris and Anzac Day commemorations in Belgium. “Science, innovation and technology have a major role to play in rebuilding our economy and achieving better health, environmental and social outcomes ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Comprehensive Partnership the goal for NZ and the Philippines
    Prime Minister Christopher Luxon held a bilateral meeting today with the President of the Philippines, Ferdinand Marcos Jr.  The Prime Minister was accompanied by MP Paulo Garcia, the first Filipino to be elected to a legislature outside the Philippines. During today’s meeting, Prime Minister Luxon and President Marcos Jr discussed opportunities to ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government commits $20m to Westport flood protection
    The Government has announced that $20 million in funding will be made available to Westport to fund much needed flood protection around the town. This measure will significantly improve the resilience of the community, says Local Government Minister Simeon Brown. “The Westport community has already been allocated almost $3 million ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Taupō takes pole position
    The Government is proud to support the first ever Repco Supercars Championship event in Taupō as up to 70,000 motorsport fans attend the Taupō International Motorsport Park this weekend, says Economic Development Minister Melissa Lee. “Anticipation for the ITM Taupō Super400 is huge, with tickets and accommodation selling out weeks ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Cost of living support for low-income homeowners
    Local Government Minister Simeon Brown has announced an increase to the Rates Rebate Scheme, putting money back into the pockets of low-income homeowners.  “The coalition Government is committed to bringing down the cost of living for New Zealanders. That includes targeted support for those Kiwis who are doing things tough, such ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government backing mussel spat project
    The Coalition Government is investing in a project to boost survival rates of New Zealand mussels and grow the industry, Oceans and Fisheries Minister Shane Jones has announced. “This project seeks to increase the resilience of our mussels and significantly boost the sector’s productivity,” Mr Jones says. “The project - ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government focused on getting people into work
    Benefit figures released today underscore the importance of the Government’s plan to rebuild the economy and have 50,000 fewer people on Jobseeker Support, Social Development and Employment Minister Louise Upston says. “Benefit numbers are still significantly higher than when National was last in government, when there was about 70,000 fewer ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Clean energy key driver to reducing emissions
    The Government’s commitment to doubling New Zealand’s renewable energy capacity is backed by new data showing that clean energy has helped the country reach its lowest annual gross emissions since 1999, Climate Change Minister Simon Watts says. New Zealand’s latest Greenhouse Gas Inventory (1990-2022) published today, shows gross emissions fell ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Earthquake-prone buildings review brought forward
    The Government is bringing the earthquake-prone building review forward, with work to start immediately, and extending the deadline for remediations by four years, Building and Construction Minister Chris Penk says. “Our Government is focused on rebuilding the economy. A key part of our plan is to cut red tape that ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Thailand and NZ to agree to Strategic Partnership
    Prime Minister Christopher Luxon and his Thai counterpart, Prime Minister Srettha Thavisin, have today agreed that New Zealand and the Kingdom of Thailand will upgrade the bilateral relationship to a Strategic Partnership by 2026. “New Zealand and Thailand have a lot to offer each other. We have a strong mutual desire to build ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government consults on extending coastal permits for ports
    RMA Reform Minister Chris Bishop and Transport Minister Simeon Brown have today announced the Coalition Government’s intention to extend port coastal permits for a further 20 years, providing port operators with certainty to continue their operations. “The introduction of the Resource Management Act in 1991 required ports to obtain coastal ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Inflation coming down, but more work to do
    Today’s announcement that inflation is down to 4 per cent is encouraging news for Kiwis, but there is more work to be done - underlining the importance of the Government’s plan to get the economy back on track, acting Finance Minister Chris Bishop says. “Inflation is now at 4 per ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • School attendance restored as a priority in health advice
    Refreshed health guidance released today will help parents and schools make informed decisions about whether their child needs to be in school, addressing one of the key issues affecting school attendance, says Associate Education Minister David Seymour. In recent years, consistently across all school terms, short-term illness or medical reasons ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Unnecessary bureaucracy cut in oceans sector
    Oceans and Fisheries Minister Shane Jones is streamlining high-level oceans management while maintaining a focus on supporting the sector’s role in the export-led recovery of the economy. “I am working to realise the untapped potential of our fishing and aquaculture sector. To achieve that we need to be smarter with ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Opinion: It’s time for an arts and creative sector strategy
    I was initially resistant to the idea often suggested to me that the Government should deliver an arts strategy. The whole point of the arts and creativity is that people should do whatever the hell they want, unbound by the dictates of politicians in Wellington. Peter Jackson, Kiri Te Kanawa, Eleanor ...
    BeehiveBy beehive.govt.nz
    1 week ago

Page generated in The Standard by Wordpress at 2024-04-26T08:38:14+00:00