- Date published:
10:30 am, January 22nd, 2015 - 29 comments
Categories: Abuse of power, International, Spying - Tags: bait and switch, bayes theroem, mass surveillance, maths, surveillance, Surveillance state
We are told that there is…
No alternative to bulk data collection
No existing technology can fully replace collecting data in bulk to obtain electronic intelligence, but some methods could be developed to improve how information is gathered and used, the US National Research Council said in a report on Thursday.
(See The Guardian for much more nuanced coverage of this report: “the panel did not specifically endorse any bulk collection conducted by the NSA on Americans’ phone records and international communications and foreigners’ emails, phone calls and internet searches”.)
Of course there is no alternative to bulk data collection if you define “electronic intelligence” as bulk data. But how useful is this “intelligence”?
However, a blue-ribbon panel set up by Obama following Snowden’s revelations reported it could find no evidence that sweeping collection of the telephone metadata of Americans led to a single major counter-terrorism breakthrough.
So it turns out that this “intelligence” isn’t useful for catching terrorists in practice. The following piece (which appeared in New Scientist following the Paris attacks) explains why:
Mass surveillance not effective for finding terrorists
Brothers Said and Cherif Kouachi and Amedy Coulibaly, who murdered 17 people, were known to the French security services and considered a serious threat. France has blanket electronic surveillance. It didn’t avert what happened.
Police, intelligence and security systems are imperfect. They process vast amounts of imperfect intelligence data and do not have the resources to monitor all known suspects 24/7. The French authorities lost track of these extremists long enough for them to carry out their murderous acts. You cannot fix any of this by treating the entire population as suspects and then engaging in suspicionless, blanket collection and processing of personal data.
Mass data collectors can dig deeply into anyone’s digital persona but don’t have the resources to do so with everyone. Surveillance of the entire population, the vast majority of whom are innocent, leads to the diversion of limited intelligence resources in pursuit of huge numbers of false leads. Terrorists are comparatively rare, so finding one is a needle in a haystack problem. You don’t make it easier by throwing more needleless hay on the stack.
It is statistically impossible for total population surveillance to be an effective tool for catching terrorists.
Even if your magic terrorist-catching machine has a false positive rate of 1 in 1000 – and no security technology comes anywhere near this – every time you asked it for suspects in the UK it would flag 60,000 innocent people.
Mass surveillance makes the job of the security services more difficult and the rest of us less secure.
The “statistically impossible” link above takes you to an analysis based on Bayes Theorem. Given any reasonable assumption of base rate (number of terrorists in the population) and misidentification rate (false positives) “domestic monitoring of everyone’s email and phone calls is useless for finding terrorists”. But it would be useful in contexts where the base rate is higher, so it would “very effective for domestic political intelligence”.
The Guardian weighs in:
The Guardian view on mass surveillance: missing the target
The pretext of Paris is being used to reheat the argument for ubiquitous snooping, shorn of the checks and balances that might achieve consent
The Paris gunmen had been on watchlists for years. Building up extra intelligence on all 66 million residents of France would not have helped; keeping an unflinching eye on the few thousands who provoke serious fears might have done. If the question were resources, the spies would deserve a fair hearing. But they seem more interested in the power to add hay to the stack, a perverse way to hunt the needle. For all the claims made for untargeted sifting, the sole “plot” that the US authorities can hold up as having been disrupted by it is a taxi driver’s payment of a few thousand dollars to al-Shabbab. Terrorists, from 9/11 to the Woolwich jihadists and the neo-Nazi Anders Breivik have almost always come to the authorities’ attention before murdering. Society can’t afford too many scruples about the privacy of those who provoke such suspicions.
But snooping on everybody shreds the implicit consent on which all effective government activity – including legitimately secret activity – must rest.
Mass surveillance cannot accomplish its stated goals. It is likely that many within the security / government system understand this full well. But mass surveillance is being pushed on us anyway. This means of course, that it is being used for unstated goals.