Big Data

How to succeed with big data on a tight budget

Data science doesn't have to require a big spend in order to play a useful role in your organization. Get tips on benefiting from big data when you have a small, fixed budget.

Image: iStock

Big data means big dollars for many companies, but it doesn't have to be that way.

Much of the work I do with large companies involves using big data to drive a competitive advantage; as such, data science is woven into the corporate strategy and deservedly requires a significant investment in time, money, and resources. But do all data science efforts need to drive a competitive advantage? No.

I'm not suggesting you repeat the mistakes of early big data adopters who threw money at big data with no real plan or goal. But I do believe data science can play a useful role without breaking the bank.

Conduct data science experiments

If you aren't spending a lot of money on data science, then make sure to set your expectations low. For instance, if you're looking at a strategic effort with a $2 billion upside where big data plays a large role, your investment could easily be $100 million or more per year.

SEE: Special report: Tech budgets 2017, A CXO's guide (free ebook) (TechRepublic)

On the other hand, if you want to experiment with big data, you could hire a data scientist and buy a few machines and tools for approximately $300,000 per year (I'm basing these figures on costs in Silicon Valley). To take an approach like this, you must think of data science as research and development; with an R&D effort, there are no strong or immediate expectations for a return on investment. But it's not a frivolous investment.

For a pharmaceutical company, R&D is the foundation for how its business works--but there's a lot of faith involved. This is the way you should approach your data science experimentation. It's okay to start with a low budget if you're willing to accept little or no initial benefit.

Reach out to your available resources

A smart way to financially ease into data science is to leverage your available functions. If you have a business intelligence and data warehousing team that manages your decision support systems, you should consider carving out a small segment for data science. I bet it won't take long to find one or two talented data professionals who would like to experiment with data science.

Another area to consider is your Lean Six Sigma or continuous improvement function. If you have a center of excellence where green and black belts live, it won't be difficult to turn them into data scientists; they have the analytic background and they're comfortable with data, but they may need some education on information systems and data management. A small investment in learning and development could go a long way.

If you can't spare resources from the data warehousing or Lean Six Sigma groups, consider finance professionals--they're becoming more and more proficient with business intelligence. And, it's not unusual to find a finance analyst that knows a thing or two about computer programming.

Create a small data science team

If you want to put a little more into big data analytics, it's best to create a small data science team--in fact, you can start with a team of one. Hire a data scientist with skills in data management, artificial intelligence, computer programming, advanced mathematics, and visualization. Next, turn him or her loose on a small server running open source software such as Apache Hadoop.

Two heads are better than one when it comes to data scientists, so if you can spare the money, it's well worth it. Two data scientists can benefit from pair programming and other collaborative techniques that will multiply your chances of great returns. Either hire two well-rounded data scientists, or hire two data scientists with complementary skills.

SEE: Hiring kit: Data architect (Tech Pro Research)

When your team goes beyond two data scientists, you're starting to get serious. While I still put a three-person team in the limited-budget category, your investment is reaching a respectable level so I suggest striking a balance between experimentation and targeted return on investment. If you go beyond three data scientists, you should consider breaking them off into their own function and using them more directly to forward your corporate strategy.


You can dip your toe into the big data lake without a big spend by treating it more like an R&D effort. Pull resources from one of your existing functions, or hire one or more dedicated data scientists, but keep the operation small.

Take the time to ask your Business Intelligence, Lean Six Sigma, or Finance departments to see if there's any interest in experimenting with big data. I'm sure you'll get at least a couple of takers.

Also see

Visit TechRepublic