The mathematics of surveillance

Written By: - Date published: 10:30 am, January 22nd, 2015 - 30 comments
Categories: Abuse of power, International, Spying - Tags: , , , , ,

We are told that there is…

No alternative to bulk data collection

No existing technology can fully replace collecting data in bulk to obtain electronic intelligence, but some methods could be developed to improve how information is gathered and used, the US National Research Council said in a report on Thursday.

(See The Guardian for much more nuanced coverage of this report: “the panel did not specifically endorse any bulk collection conducted by the NSA on Americans’ phone records and international communications and foreigners’ emails, phone calls and internet searches”.)

Of course there is no alternative to bulk data collection if you define “electronic intelligence” as bulk data. But how useful is this “intelligence”?

However, a blue-ribbon panel set up by Obama following Snowden’s revelations reported it could find no evidence that sweeping collection of the telephone metadata of Americans led to a single major counter-terrorism breakthrough.

So it turns out that this “intelligence” isn’t useful for catching terrorists in practice. The following piece (which appeared in New Scientist following the Paris attacks) explains why:

Mass surveillance not effective for finding terrorists
…
Brothers Said and Cherif Kouachi and Amedy Coulibaly, who murdered 17 people, were known to the French security services and considered a serious threat. France has blanket electronic surveillance. It didn’t avert what happened.
…
Police, intelligence and security systems are imperfect. They process vast amounts of imperfect intelligence data and do not have the resources to monitor all known suspects 24/7. The French authorities lost track of these extremists long enough for them to carry out their murderous acts. You cannot fix any of this by treating the entire population as suspects and then engaging in suspicionless, blanket collection and processing of personal data.

Mass data collectors can dig deeply into anyone’s digital persona but don’t have the resources to do so with everyone. Surveillance of the entire population, the vast majority of whom are innocent, leads to the diversion of limited intelligence resources in pursuit of huge numbers of false leads. Terrorists are comparatively rare, so finding one is a needle in a haystack problem. You don’t make it easier by throwing more needleless hay on the stack.

It is statistically impossible for total population surveillance to be an effective tool for catching terrorists.

Even if your magic terrorist-catching machine has a false positive rate of 1 in 1000 – and no security technology comes anywhere near this – every time you asked it for suspects in the UK it would flag 60,000 innocent people.
…
Mass surveillance makes the job of the security services more difficult and the rest of us less secure.

The “statistically impossible” link above takes you to an analysis based on Bayes Theorem. Given any reasonable assumption of base rate (number of terrorists in the population) and misidentification rate (false positives) “domestic monitoring of everyone’s email and phone calls is useless for finding terrorists”. But it would be useful in contexts where the base rate is higher, so it would “very effective for domestic political intelligence”.

The Guardian weighs in:

The Guardian view on mass surveillance: missing the target

The pretext of Paris is being used to reheat the argument for ubiquitous snooping, shorn of the checks and balances that might achieve consent

The Paris gunmen had been on watchlists for years. Building up extra intelligence on all 66 million residents of France would not have helped; keeping an unflinching eye on the few thousands who provoke serious fears might have done. If the question were resources, the spies would deserve a fair hearing. But they seem more interested in the power to add hay to the stack, a perverse way to hunt the needle. For all the claims made for untargeted sifting, the sole “plot” that the US authorities can hold up as having been disrupted by it is a taxi driver’s payment of a few thousand dollars to al-Shabbab. Terrorists, from 9/11 to the Woolwich jihadists and the neo-Nazi Anders Breivik have almost always come to the authorities’ attention before murdering. Society can’t afford too many scruples about the privacy of those who provoke such suspicions.

But snooping on everybody shreds the implicit consent on which all effective government activity – including legitimately secret activity – must rest.

Mass surveillance cannot accomplish its stated goals. It is likely that many within the security / government system understand this full well. But mass surveillance is being pushed on us anyway. This means of course, that it is being used for unstated goals.

Late updates: similar piece here. Short version here.

30 comments on “The mathematics of surveillance ”

  • Comments are now closed

  1. adam 1
    22 January 2015 at 10:49 am

    What unstated goal, FEAR?!?

  2. Colonial Rawshark 2
    22 January 2015 at 11:25 am

    Bill Binney and Jacob Appelbaum

    Binney is one of the NSA’s most senior ever whistleblowers. He developed many of the first concepts for surveillance of the internet and the world wide web (to keep up with the volume, velocity and variety of the data going across the web).

    He suggested systems and methods for maintaining the privacy of US citizens and preventing this problem of overwhelming, near-useless mass surveillance. For instance, all captured data would be encrypted and could only be decrypted for individuals one by one, on a court order. His proposals were turned down by the NSA.

    The whole idea is that when you are trying to find a terrorist needle in a haystack, the last thing you want to be doing is adding more irrelevant haystacks.

    Well worth watching.

  3. Lanthanide 3
    22 January 2015 at 12:26 pm

    “France has blanket electronic surveillance. It didn’t avert what happened.”

    This is a tautology. If it had averted what happened, it wouldn’t have happened, and we wouldn’t have known that it was averted. How many other terrorist plots have been averted? Singling this one out and saying “see, I told you so!” is a stupid argument.

    “Mass surveillance makes the job of the security services more difficult and the rest of us less secure.”

    Where has anyone said that mass surveillance is used as the first piece of evidence for whether someone is a terrorist or not? It makes far more sense that it would be used *after* other evidence has been discovered. Something flags up that person X might be suspicious, so you dig into the records that you’ve been recording of person X for the last 4 years and see what they’ve been up to.

    • tracey 3.1
      22 January 2015 at 12:34 pm

      and in the example cited above you are proposing “digging” into over 60,000 individuals records… you better hope the ones you find of concern are at the early stages of their planning…

      In any event, that both Sydney and Paris’ murderers were known to the various authorities and they didn’t stop it, or couldn’t, why would being able to “dig into” over 60,000 innocents flagged, have made the difference?

      • Lanthanide 3.1.1
        22 January 2015 at 12:39 pm

        “and in the example cited above you are proposing “digging” into over 60,000 individuals records… you better hope the ones you find of concern are at the early stages of their planning…”

        No, i’m proposing precisely the opposite of that: you don’t do any “digging” at all until you have a suggestion from some other source that someone might be up to something naughty.

        • tracey 3.1.1.1
          22 January 2015 at 12:47 pm

          I think you over estimate how much common sense plays a part in our spy services. I suspect that the info, while not great for terrorists may be great for market data mining.

    • Draco T Bastard 3.2
      22 January 2015 at 1:19 pm

      Where has anyone said that mass surveillance is used as the first piece of evidence for whether someone is a terrorist or not? It makes far more sense that it would be used *after* other evidence has been discovered.

      That would make it targeted surveillance and not mass surveillance, i.e, someone comes to the authorities attention they then put in place surveillance on that person and, from there, others that have connection with that person. Coming to the attention of the authorities comes first. The problem is the mass surveillance that happens without any need or warrant to do so which then becomes the haystack with a needle in it.

      Something flags up that person X might be suspicious, so you dig into the records that you’ve been recording of person X for the last 4 years and see what they’ve been up to.

      There’s no need of that mass surveillance and it actually makes finding the criminals harder as there’s now so much more data to sift through.

    • Ross 3.3
      22 January 2015 at 1:55 pm

      ” …and we wouldn’t have known that it was averted.”

      Are you serious? The French government would have been broadcasting that result like a spasm. Note the number of cases that the NSA has averted through mass surveillance. ZERO. And we don’t have to “single” out this one. There is the nut job in Sydney, the cop killer in New York and the marathon bombings that just leap to mind as perfect examples of what mass surveillance doesn’t do. ALL of those people had flags all over them. They were known to be deranged and murderous and still it was useless.
      Mass surveillance is never about preventing crime and is always about control.

    • Stephanie Rodgers 3.4
      22 January 2015 at 11:07 pm

      This is a tautology. If it had averted what happened, it wouldn’t have happened, and we wouldn’t have known that it was averted.

      And in the kitchen we use a sieve until it lets a weevil through into the cake mix. That’s how we know the sieve isn’t working, and the number of weevils it stopped beforehand is completely irrelevant.

    • Murray Rawshark 3.5
      23 January 2015 at 3:00 am

      I didn’t think the article was that hard to understand.

      We get a fairly good idea of how many terrorist plots have been averted because such a fuss is made whenever an Arab is found with a plastic sword, or the FBI set up a sting on some microencephalic idiot. Given the bullshit they spread, I’d say basically none.

      We get to see their mistakes, and these often turn out to be people they knew about anyway. Mass surveillance is about protecting the wealth because they know the present system is unsustainable and will be shaken by riots and worse if things continue as they have been. They think they can set up a panopticon and keep us all under control. Labour wants to help them.

    • Pete George 3.6
      23 January 2015 at 6:43 am

      It’s obvious that all terrorist attacks can’t be prevented. But governments claim that they prevent some. Including the French government.

      French intelligence officials foiled several terrorist attacks in the weeks before the attack on the Charlie Hebdo magazine offices, French President Francois Hollande …

      In the past 18 months, French intelligence officials have prevented as many as five terror attacks from taking place in France, Interior Minister Bernard Cazeneuve said

      http://mashable.com/2015/01/07/hollande-france-terror-plots-foiled/

      It’s impossible to know how many potential terrorists are deterred by security measures, or whether their actions are limited by security.

      We can’t prevent all attacks: EU anti-terror chief

      The EU’s counter-terrorism chief said Tuesday it is impossible to completely prevent new Islamist attacks like those in Paris, and warned that Europe’s prisons have become a “massive incubator” for radicalisation.

      Gilles de Kerchove told AFP that the Islamic State group and Al-Qaeda wanted to launch more attacks on the West…

      De Kerchove warned that the Al-Nusra Front, the Al-Qaeda branch in Syria, is also looking for “clean skins,” Europeans with valid passports and no record of radical activity, to mount attacks in Europe.

      http://www.eubusiness.com/news-eu/france-attacks.zg5

      They can’t have no security. They get severely criticised if their security fails to prevent attacks occasionally. So the pressure is on to prevent attacks, and a range of measures are necessary. Getting the balance between security and privacy right will always be difficult and imperfect.

      No surveillance is hardly an acceptable option. So the argument has to be how much and what type of surveillance is acceptable for a reasonable level of security.

      • Colonial Rawshark 3.6.1
        23 January 2015 at 10:48 pm

        So you are satisfied with the explanations of how the French intelligence services missed the fact that several people that they had on watch lists pulled this massacre off under their noses?

        Oh that’s right, there have been no actual explanations, not that you actually care for that detail, you’ll give them a pass mark.

        By the way this bullshit line of there needing to be a “balance” between security and privacy stems back to immediately after 9/11 and the Patriot Act. You’ve gotta think we are idiots to keep falling for it again.

        • JRyan 3.6.1.1
          27 January 2015 at 8:51 am

          So you say we do not need any surveillance? Security of a country requires a certain amount of information gathering. It is simply naïve to not monitor people that could possibility carry out some warped act of violence. And if one thinks there is no one on our shores that wants to agitate, carry out instructions from afar or simplistic domestic violence then one is disillusioned. Its like saying we do not require a police force. The police are a domestic army.

  4. Treetop 4
    22 January 2015 at 3:12 pm

    I read today either on stuff.co or the Herald that when it came to Smith’s passport (absconded to Brazil) being issued, facial recognition was used. Even using facial recognition has its failure. Legal name matched the face, so got a passport. Even alias names used were checked.

  5. One Anonymous Bloke 5
    22 January 2015 at 3:37 pm

    You mean to say the abstract art known as Mathematics has a Liberal bias too!

  6. Truth Will Out 6
    22 January 2015 at 4:47 pm

    The issue is not mass surveillance.

    The issue is potential misuse of the information for political gain/reasons.

    Through a quirk of fate, I was born in Yemen.

    Even though my parents are both British, and we left there when I was less than 3 months old, it appears on my birth certificate and as my birthplace on my UK & NZ passports.

    This, in itself, is not a problem.

    What IS a problem, however, is that I also happen to have been very outspoken criticising the current government over their handling of one particular issue, and subsequently others.

    In a properly functioning democracy, this would not be a problem.

    But we do not live in a properly functioning democracy.

    What we DO live in now, is a system where a politician like John Key can have me surveilled and even arrested, as well as held indefinitely without charge, on the “suspicion that I might be planning activities” which are not to his government’s liking.

    History has already proven more than once that Key and his goons are more than willing to criminalise dissent at every available opportunity.

    Simply to silence their critics, and intimidate everyone else into conformity and submission.

    And he could do it very, very easily by simply saying “he is from Yemen”.

    And more than half the population will need no further evidence than that to satisfy their minds that I deserve such treatment.

    All they will need is John Key’s word for it, or Chris Finlayson’s,

    People should think about that. They should think about how quickly and easily they dismiss the rights of other people based on accusations and innuendo, because if anything is weakening our democracy and rights, that is.

    It gives power to politicians they should not have, and it places them above the courts, which effectively means we lost World War 2 in reality.

    So, next time you attend an Anzac Dawn Parade, standing there with your hand on your heart, make sure you actually do care about the sacrifices our soldiers made, instead of just pretending to.

    • One Anonymous Bloke 6.1
      22 January 2015 at 4:56 pm

      Well said.

    • fisiani 6.2
      22 January 2015 at 11:03 pm

      Paranoia can be treated. We do live in a totalitarian country. Even a Leftist like you will be treated with respect.

      • Truth Will Out 6.2.1
        23 January 2015 at 10:05 am

        @Fisiani

        And so John Key deletes his txt messages because he isn’t paranoid and has nothing to hide?

        He employs teams of people to help him maintain a carefully constructed public image because he just feels like it?

        Your assumptions are laughable.

        I support neither the left nor the right, and I have provided ample evidence on this website to prove it.

        Why do you need to categorise people so much, what are you afraid of?

        You speak with thinly disguised contempt of “respect”, as if it is something I should feel privileged to have conferred upon me by the likes of you.

        Why would I want respect from such an ignorant piece of sh*t like you?

        How would that serve me, when I rate you beneath bacteria in the great scheme of things?

  7. saveNZ 7
    22 January 2015 at 9:37 pm

    Mass surveillance and all the censorship is making any ‘terrorist’ situation worse in two ways. Firstly by surveilling the masses you are taking away resources from more likely suspects and two you are making any potential terrorists (or nutcase) more careful and less likely to get caught. You want nutcase to speak out on Facebook etc and by using profiling could probably predict more accurately any potential terrorist. You could then surveil them covertly and find out if there is any actual threat. By threatening and over reacting and persecuting innoscent people it is making those people more radicalised, defeating the purpose of containing terrorism – instead it is increasing the risk.

    But let’s face it, mass surveillance is about power and control not terrorism.

    The Illegal war on Iraq on the basis of WMD which was false, and over time seems to have be forgotten that there were no grounds for invasion. Now NZ might be going there to be part of ‘the club’. F@$K these are peoples lives at stake.

    The 911 terrorists were mostly Saudi Arabians. They have terrible human rights record and have just beheaded a woman but But hey, America does nothing to them, but instead concentrating on countries that are not linked to terrorism… that is, until the ‘war on terror’.

    Saudi Arabia publicly beheads woman in holy Mecca as blogger lashings are postponed
    http://www.independent.co.uk/news/world/middle-east/saudi-arabia-publicly-beheads-woman-in-holy-mecca-as-blogger-set-to-receive-second-lashing-9982134.html

    It’s perfect capitalism, taxpayers funding the war on terror, and the war industry meeting the market and creating more demand for their services by creating more terrorists.

  8. Pete George 8
    23 January 2015 at 7:02 am

    Mass surveillance cannot accomplish its stated goals. It is likely that many within the security / government system understand this full well. But mass surveillance is being pushed on us anyway. This means of course, that it is being used for unstated goals.

    It’s been stated a number of times that we don’t do mass surveillance in New Zealand.

    Prime Minister John Key has released a series of documents ‘setting the record straight’ over claims the GCSB had spied on New Zealanders.

    Mr Key responded quickly to Edward Snowden and Glen Greenwald’s freshest claims – that “if you live in New Zealand, you are being watched” – this afternoon.

    “Claims have been made tonight that are simply wrong and that is because they are based on incomplete information.

    ”There is not, and never has been, a cable access surveillance programme operating in New Zealand.

    “There is not, and never has been, mass surveillance of New Zealanders undertaken by the GCSB.

    http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11325069

    And

    And GCSB spies respond to mass surveillance allegations

    The Government Communications Security Bureau (GCSB) has responded to election week allegations it carries out mass surveillance on New Zealanders, denying its programmes are for anything other than cyber security.

    It’s been likened to scanning of everything on your computer with virus protection but on a country scale. It’s also been said that large companies and organisations have been assisted in cyber protection.

    Early September this year, somewhere in the world unknown computer hackers set their sights on New Zealand. Boffins in charge of security at Telecom, now called Spark, saw a cyber-attack coming in, a big one.

    Its internet and email system went down on the Friday and stayed down for 72 hours.

    The experts are still trying to work out exactly what did happen when foreign hackers took control of 120 home computers.

    Cyber-attacks happen across the world every hour of every day. It’s these sort of attacks the GCSB says it is trying to prevent – shadowy hackers from all over the world, sending out complex viruses to damage big businesses or Government departments, or even getting inside and taking them over.

    My guess is that most people would be happy to have their home computers protected from being taken over.

    There is no direct proof that the GCSB is hovering up the metadata of ordinary New Zealanders, but the cable programme 7148 and the approach to Spark are possible indications that last year it was on the cards and it may be again.

    Mass surveillance/collection of all metadata of New Zealanders by the GCSB is illegal. There are very specific legal processes involved in allowing targeted surveillance.

    Not legal. No proof.

    Mass surveillance is not being pushed on us. What is the unstated goal of implying that it is?

    • One Anonymous Bloke 8.1
      23 January 2015 at 7:41 am

      The beige parrot has learned another lie.

      Bzzt.

    • Tracey 8.2
      23 January 2015 at 10:14 am

      you didnt actually read the documents he released and compare them to the statements he said they refuted, did you Pete?

      Truly, take the time to actually read the documents the PM declassified to “prove” his point… then go back and read what was said by Snowden

    • Murray Rawshark 8.3
      23 January 2015 at 4:43 pm

      “My guess is that most people would be happy to have their home computers protected from being taken over.”

      Yeah, you get an antivirus for that, not an all powerful agency full of lying squirrels with allegiance to the Republican caucus of the US and A.

    • Colonial Rawshark 8.4
      23 January 2015 at 6:36 pm

      Mass surveillance/collection of all metadata of New Zealanders by the GCSB is illegal. There are very specific legal processes involved in allowing targeted surveillance.

      That’s why they get their foreign partners to do it sillybilly.

      You have not been keeping up. Don’t comment on things that you have not been following.

      BTW do you remember when James Clapper National Security Director told a panel of the House that the NSA did not collect the communication information of millions or hundreds of millions of Americans?

      Turns out he was fucking lying through his teeth.

      Given this, and the many other lies the power elite have told us about their systems and activities, what makes you so trusting of the FVEY security apparatus on this side of the Pacific?

Links to post

The server will be getting hardware changes this evening starting at 10pm NZDT.
The site will be off line for some hours.