Big Data

Why antiterror datamining is stupid

Me, I'm a big fan of math, and a big fan of civil liberties, so I really enjoyed it when Wired's Bruce Schneier statistically enumerated the stupidity of the U.S. Defense Department's antiterror datamining projects:

"Let's look at some numbers. We'll be optimistic — we'll assume the

system has a one in 100 false-positive rate (99 percent accurate), and

a one in 1,000 false-negative rate (99.9 percent accurate). Assume 1

trillion possible indicators to sift through: that's about 10 events —

e-mails, phone calls, purchases, web destinations, whatever — per

person in the United States per day. Also assume that 10 of them are

actually terrorists plotting.

"This unrealistically accurate system will generate 1 billion false alarms for every real terrorist plot it uncovers. Every day of every

year, the police will have to investigate 27 million potential plots in

order to find the one real terrorist plot per month. Raise that

false-positive accuracy to an absurd 99.9999 percent and you're still

chasing 2,750 false alarms per day — but that will inevitably raise

your false negatives, and you're going to miss some of those 10 real


Bottom line, the terror-sniffing algorithm will nevr leave BETA

mode, never get to the point of actually guiding law enforcement

personnel in any meaningful way (other than creating an easily abused

central data repository of innocent U.S. citizens), and will succeed

only in diverting valuable antiterror resources away from programs that

an actually use them. That, boys and girls, is stupid math.

About Jay Garmon

Jay Garmon has a vast and terrifying knowledge of all things obscure, obtuse, and irrelevant. One day, he hopes to write science fiction, but for now he'll settle for something stranger — amusing and abusing IT pros. Read his full profile. You can a...

Editor's Picks

Free Newsletters, In your Inbox