When working on a data science problem, sometimes analysts get stuck in a writer's block-like state I call quant block. But what should you do if your entire team is struggling to solve the problem, and it's causing a development bottleneck? Your best move is to call a tabula rasa (which means blank slate).
Starting over isn't that radical
Neither mathematical problem-solving nor computer programming is a linear process, so it's absurd to think that data science development will be linear. There are bursts of genius, followed by bursts of frustration, and then you eventually crack the code. Most of the time a good data science team can pull through when their engine stalls (though if you're asking them to solve something like the Hodge conjecture, you may be waiting a while).
Starting over may sound like a desperate move, but it's not that radical. When training a neural network, the system will sometimes get stuck in what's called a local optimum. In general, the nodes of a neural network start in a neutral state, and then learn by gradually getting better. Think of it as slowly climbing a hill and constantly looking for the next step up; however, sometimes there is no step up, there are only steps down — but the highest region (the global optimum) is in another area of the problem space that the nodes can't see right now. In this case, you might bounce the system, and give it a new place to start so it has a chance to find that high peak.
It's not reasonable to completely disregard the experience that brought the team to its local optimum; this experience is extremely valuable, even though it's not readily apparent or applicable. With that, it's important to condition your team for the tabula rasa, so they're ready if it comes.
Running clean slate drills
Why run the drills
I recommend running tabula rasa (or clean slate) drills with your data science team. Clean slate drills are when you intentionally delete everything the team is working on, forcing them to start over. It may seem like a waste of time, but it's all in your perspective of waste.
The primary reason for running clean slate drills is to debunk the preconceived myths discussed earlier about starting over. Many times, the team will recover from a clean slate drill with a better solution in less time — even if they weren't stuck in the first place. Once they emerge from this a few times, they'll come to know the reality of a tabula rasa.
How to run the drills
Your data science team should be progressively conditioned for the tabula rasa. Start out with very controlled exercises. Bring the team together, explain what will happen, and state when it will happen. Also, don't blow away their entire system — just start with a small functional area. Back up everything before you flip the switch; you probably won't need the backup, but do it anyway because it's a nice security blanket.
After a few of these exercises, follow up with some surprise drills. Alert the team that a drill is going to occur, but don't tell them when it will start. These sting a little at first, but the team will get used to it. Back everything up again, so they don't panic too much.
Communicate your support
Be sure to communicate to the team that you fully support these drills, and that you won't hold them accountable for losing development time; this is all part of the process, and it's built into the development schedule. Once the team becomes accustomed to handling this situation, they'll be ready for a real tabula rasa.
Calling a tabula rasa may sound extreme, but sometimes it's the only way to get the team moving forward. By organizing a controlled tabula rasa, it will help ease the pain the next time your data science team is suffering from quant block, and it's necessary to wipe the slate clean.
- What analytic managers can learn from neural networks
- Deep learning: A high-risk/high-reward big data investment
- Big data: Neither snake oil or silver bullet
- Going Deep on Big Data (Special Feature from ZDNet and TechRepublic)
John Weathington is President and CEO of Excellent Management Systems, Inc., a management consultancy that helps executives turn chaotic information into profitable wisdom.