Big Data

How your big data career is killing other jobs

As we use data to drive efficiencies, people may be the casualty, leading to data-driven unemployment.

Image: iStockphoto/Pinkypills

There's a lot of money to be made in data science, as a new O'Reilly report indicates. How much money? Over the last three years, "About half of [those surveyed] reported a 20% change [in salary], and the salary of 12% of the sample doubled." With a median salary of $106,000 for US data professionals, those are significant jumps.

Yet, there's also a lot of unemployment. At least, for those people who data science helps to put out of a job. What Patrick McKenzie wrote of engineers is equally true of data scientists: "You're in the business of unemploying people." Data, done right, makes systems more efficient and, inevitably, "efficiency gains" generally translate into "somebody will lose their job to a machine."

But not all jobs. Gartner's Svetlana Sicular described the importance of people and machines collaborating on data, something that the industry is only just now settling into accepting.

Your data science mileage may vary

In the early, hype-prone days of big data, breathless reports abounded of machines replacing humans as data analytics uncovered "actionable insights" that humans would then execute. Reality, however, has been very different.

SEE: Why data science is just grade school math and writing

It turns out that machines aren't very good at interpreting data. As biased as we people are (and we are, both in the data we choose to collect and the questions we ask of it), and as flawed as our analysis can be, people remain essential to understanding data. The key is to figure out the right balance between human and machine, as Sicular highlighted:

I call the symbiosis of a man and a machine Centaur Intelligence, in which the head is always human: People do at what they excel—curiosity, creativity and compassion. And machines do their best too: Learn at scale, crunch data and answer questions lightning fast. Machine answers require a human interpretation to turn correlations into causation. People curate data and select the right questions. Both parties augment each other, and the centaur is riding the wave of intelligence, human and artificial.

This smacks of common sense, because it is. But, again, common sense has been lacking in the frothy excitement around big data's possibilities. That excitement is warranted but only, as Sicular posited, as a companion to human interpretation. You need both machine and human intelligence to get the most from data.

You also need human ingenuity to architect and build complex, distributed systems. As Jesse Anderson argued, this proves to be very difficult in practice. It will keep many an engineer employed for years to come.

Data-driven unemployment

Others, however, are not so fortunate. While I think it's an inescapable fact that machines will replace many human jobs, it's not necessarily a cheerful one. Sicular noted, "We should remember that each machine is good at a particular aspect (for which it has data and algorithms). Humans are a universal machine that is good at many things at once." But for those people that aren't particularly "good at many things at once," single-purpose machines may prove a ready, superior replacement.

SEE: Why 2016 might be the year of citizen data scientists

It has ever been thus with technology, but perhaps never more so than now.

As a recent report in The Economist showcased:

In the old days companies with large revenues and global footprints almost always had lots of assets and employees. Some superstar companies, such as Walmart and Exxon, still do. But digital companies with huge market valuations and market shares typically have few assets. In 1990 the top three carmakers in Detroit between them had nominal revenues of $250 billion, a market capitalisation of $36 billion and 1.2m employees. In 2014 the top three companies in Silicon Valley had revenues of $247 billion and a market capitalisation of over $1 trillion but just 137,000 employees.

Perhaps even more tellingly, as companies like Uber dismantle old industries, they're using data to turn people into functional cogs with minimal, if any, benefits (e.g., health insurance, disability, etc.).

None of which is a reason to try to turn back the clock on technological progress, Luddite-style. No, it's just a reminder that when we code, particularly in our increasingly data-driven world, we need to be sensitive to the lives we impact for both good and ill.

Also see

About Matt Asay

Matt Asay is a veteran technology columnist who has written for CNET, ReadWrite, and other tech media. Asay has also held a variety of executive roles with leading mobile and big data software companies.

Editor's Picks

Free Newsletters, In your Inbox