One of the great promises, and more obvious use cases for big
data, is in IT security.
Security-related data clearly fits the definition of “big,” with
nearly every device and application on the network generating reams of logging
and performance data. Churning through massive amounts of data in near
real-time and identifying anomalies as they occur is the holy grail of IT
security. There’s also the interesting aspect of environmental data. Perhaps
economic conditions, news reports, or even the weather in certain geographies
might affect the probability of a security incident, presumably allowing big
data and predictive analytics to predict a security breach before it even

The small reality

Much like flying cars
or dehydrated beer, a “drop in” big data security application that
patiently scans every iota of internal and environmental data, and then quietly
drops an appointment on your calendar for the denial of service attack that
will happen next week, is not yet ready for mass consumption. While
conceptually simple, the data gathering, storage, and analytic technology
required to pull off such a feat are still in the juvenile stages at best.
Furthermore, the cost for these technologies and the integration required for
true predictive security are significant. Unless your business is highly
sensitive to security concerns, at this point, the cost likely puts it out of

The good news is that predictive security has a compelling
and obvious benefit, one that’s captured the attention of CIOs and, in turn,
spurred investment by the large big data and IT security companies. While none
of the “usual suspects” in the vendor pool have a prepackaged and
easily installed big data security offering, there are several things you can
do to get ready for predictive security.


Instrumentation is consultant-speak for establishing logging
and data capture on relevant devices and services. While your firewalls and
package software may do a fine job with logging out of the box, third party
applications or custom code that’s accessible to the outside world may have
minimal logging, or “orphaned” logging that’s not integrated into
your overall security and monitoring infrastructure. Even if a low-cost, drop-in
solution for predictive security existed, if your IT shop doesn’t have every
application and device properly instrumented and centrally monitored, big data
simply won’t help.

weakest link: Technology no match for social engineering

Practice and procedure

Even in organizations that have a well-managed security
infrastructure, once a breach is identified there are often befuddled looks and
no clear lines of reporting or responsibility, costing precious hours during an
attack. While it’s easy for IT to do security drills, what happens in a real
incident where your back-end transactional or financial system is compromised?
Can IT unilaterally shut it down, essentially pressing “pause” on
your company’s ability to market, ship goods, or record, and manage cash? Who
needs to be notified, and who has ultimate decision making authority? The main
promise of predictive security is buying an organization time; however,
well-planned procedures that include all elements of your business, not just
IT, can do the same in the short term.


Thanks to the US Government and its NSA antics, IT security
is on the forefront of many executives’ minds. Not only may some remote hacker
be snooping around your network, but now government actors from around the
world may be siphoning data for their own mercurial purposes. It may be
tempting to fuel the fear and speculation to capture a fattened security
budget, but now is the time for IT and data experts to bring some calm and
rationality to discussions around security. This is a concern that’s been
elevated to the Board level at many organizations, and one that could use some
sound, technically-grounded advice. Conveying what’s currently possible, and what’s
coming down the road in terms of big data-driven predictive security and
forensics, is a great start regardless of whether that technology is available