Most Hadoop jobs can be found in Silicon Valley, where "The best minds of [a] generation are thinking about how to make people click ads," to quote Cloudera co-founder and chief scientist Jeff Hammerbacher. As he also said, "That sucks."
While Hadoop jobs revolve around Silicon Valley, Hadoop innovation — and big data innovation, generally — is anything but a Silicon Valley phenomenon. In fact, if you want to see entire industries changed through data, go to Middle America. That's where I've been this week, and it's eye-opening.
High-tech isn't the biggest beneficiary of big data
Not all industries are created equal when it comes to being able to put data to use. Some, like the construction industry, are constrained by the amount of data they can capture and even more hamstrung by their ability to get value from it, as McKinsey has written.
Others, like the finance industry, both generate a lot of data and can put it to use:
This has led to broad adoption of big data technologies, and not simply in the Valley, as a 2012 Gartner survey captures:
In 2012, Gartner found that 58% of organizations had purchased or were planning to invest in big data systems, a number that had jumped to 64% by 2013. In other words, the odds are pretty good that brands you know are already fiddling with big data, including some that might surprise you.
Revolutionizing agriculture... one seed at a time
Though the agricultural world has embraced big data in fits and starts, looking with sometime suspicion on what giants like Monsanto might do with their crop data, data-driven agriculture is catching on.
Because John Deere equipment comes laden with data-emitting sensors, farmers can see how they farmed a given plot of land last year (e.g., density of seeds per acre, down to the individual row) and set their equipment to optimize seeding in the current year based on historical yield, soil conditions, and more.
No wonder that David Everitt, president of John Deere's Agriculture and Turf Division, has said:
"[Digital] is the next wave of productivity for the future. Since 1918, we have delivered productivity to agriculture by building equipment that was bigger, better and faster. Now we're moving to a new level."
While it might seem like the lowest of low-tech industries, agriculture is turning out to be one of the more interesting and innovative places for big data.
Insurance gets an upgrade
McKinsey notes insurance as a likely place for big data to thrive, because insurers crunch big numbers to try to minimize risk. However, sometimes where it's thriving is not at all intuitive.
Consider MetLife, for example.
Usually, "big data" is assumed to mean "petabytes of data," but that's not typically the problem that most enterprises need to solve. Rather, variety of data is a far more substantial big data problem.
MetLife spent 10 years trying to stitch together more than 70 different data silos — holed up in relational databases, mainframes, and elsewhere — to create a unified view of all customer data. MetLife wanted a customer service representative to be able to solve customer issues in real time, rather than scavenging frantically between the 70+ systems trying to figure out who the customer was, what they'd bought, and what they needed.
So, MetLife built a Wall, a system that brings together all of its disparate data into one Facebook-like interface.
Initially using NoSQL database technology, and later adding in Hadoop for long-term analysis, MetLife was able to go from nothing to proof of concept in two weeks and then into full production within three months. This has generated "tremendous excitement" within MetLife, as it helps the 151-year old startup revolutionize its customer experience even as the Fortune 500 enterprise operates like a startup.
Big data beyond Silicon Valley
These are but two examples of companies I saw on my Middle America field trip. There are plenty of others, covering industries like mining, healthcare, manufacturing and more.
The older the industry, the more profound the change big data can make.
Ford, for example, uses data to optimize fuel efficiency of its cars. Ford also uses data to make its manufacturing far more efficient. Given the adverse impact cars have on the environment, Ford's big data initiatives could have a massive positive impact on the planet.
For developers, this may mean that the most revolutionary places to use Hadoop and other big data technologies may not be in the comfortable confines of Silicon Valley, but rather somewhere out in the Heartland.
Matt Asay is a veteran technology columnist who has written for CNET, ReadWrite, and other tech media. Asay has also held a variety of executive roles with leading mobile and big data software companies.