It takes more than just great data scientists to make a great Big Data team. Given all the hype around data scientists, when building out the capability for your Big Data strategy, it’s a common mistake to overlook the other resources that make up your Big Data resources–especially leadership and management. In general terms, leadership is the ability to deal with change and management is the ability to deal with complexity.

Although your strategy provides an essential component of leadership, it’s not enough to lead a Big Data team. And although you have competent management within your strategic business units, they may not be appropriate for your Big Data team. You must have management and leadership built within your Big Data team. Big Data capability is not built on data scientists alone; you must also build competence in dealing with change and complexity.

Managing change

One of the biggest challenges you will face with your data scientists is the ability to deal with change. This is a leader’s primary responsibility and the competence must be tightly integrated within your Big Data team. It’s a fatal mistake to assume data scientists can lead themselves; you must have a team leader working closely with data scientists to integrate their problem-solving competence with the flexibility required to survive.

Regardless of how Big Data is used in your organization, the team must model a culture of innovation. Sometimes this is obvious–sometimes not. When Big Data is adopted for information product innovation, your Big Data culture may fold in nicely with your existing product development culture. However, if Big Data is being used to solve a specific marketing problem, the required culture of innovation may not be so obvious.

A culture of innovation is sometimes difficult for a data scientist. Although they love to solve problems, they like problems that are well defined. Innovation doesn’t work that way; the problems to solve can change radically even if the strategy is firm. Also, data scientists are afraid to fail because they hate being wrong. Unfortunately, failure is a central component of proper innovation. An innovative team should be failing on a regular basis; think about how often Edison failed before the incandescent light bulb emerged.

The leader’s job is to manage expectations, reward behaviors instead of results, and keep the team moving in the right direction. Team leaders must work with data scientists to make sure they’re comfortable with rapid changes and failure–this is not an easy task. Leaders must also ensure that the team doesn’t stray from the strategic vision. With data exploration, it’s easy to get distracted. Sometimes distractions are good; however, sometimes they serve no purpose. It’s the leader’s job to weed out the bad distractions and keep the team focused in the right direction.

Managing complexity
Big Data efforts can get very complex; this is where managers are required. On a Big Data effort that’s tied to the corporate strategy, it’s not unusual to have cross-functional concerns that span over dozens of subject areas. This creates a daunting network of communication channels that’s not for the faint at heart. There is integrated planning, resource allocation, risk management, quality management, cost control, and schedules to maintain. And with frequently changing requirements, change management can quickly become a nightmare. This is absolutely the wrong role for a data scientist–do not assign a data scientist to manage your Big Data efforts.

Data scientists should be focused on solving data problems–that’s it. If they get distracted with management details, two things will happen: your core problem-solving competence will be severely compromised and you’ll never be on schedule. Instead, place a competent manager on the team that understands how to work with data scientists.

It’s important that your Big Data manager knows three things: 1) how to manage a team effectively, 2) how to work with data scientists, and 3) how to deal with rapidly changing requirements. Core management skills are essential; however, it’s also important for them to understand the culture of data scientists and how to handle the their management challenges. For instance, data scientists don’t like to be pushed–they want to be left alone to figure out their problems. Also, data scientists don’t like to explain themselves to lay people, especially if they feel they’re being challenged. Unfortunately, even something as exploratory as innovation must stay on schedule, and stakeholders will have questions that must be answered. These and many other nuances of managing data scientists must be dealt with, so a good manager is required. Finally, as explained before, innovative efforts are typically a bit chaotic when it comes to requirements, so it pays to have a manager that can deal with changing requirements.

Conclusion

Adopting a Big Data strategy requires a good Big Data team that can deal with both change and complexity. Unfortunately, data scientists won’t solve either challenge so you must build your team with effective leaders and managers. With both Big Data leaders and managers, a good understanding of the data scientists’ culture is vital. A good leader will work with the data scientists to promote a culture of innovation; a good manager will remove data scientists from all the complexity and keep them on schedule. If you’re trying to build out the capability for your Big Data strategy, I’m sure you’ve already considered data scientists; however, take some time today to consider the other people on your Big Data team. There’s no sense in spending big dollars on data scientists that aren’t producing anything useful for your strategy.