Big Data

Two programmers are better than one for your data science team

John Weathington makes the case for embracing pair programming when structuring the workflows of your data science team.

pairprogrammingatchitika.jpg
Image: Wikimedia/Kabren

There aren't many subjects in the programming world — and, by extension, the data science world — more controversial than pair programming. This is when two programmers or two data scientists physically work together on one code module. They sit together, share the same screen, and take turns typing (or driving). While one scientist drives, the other one overlooks to help problem-solve, and to ensure quality and best practices are followed, and parking lot tasks and ideas that will be handled later.

In the early 2000s, I was persecuted and chastised for advocating pair programming techniques in data teams. Now, with the gradual adoption of agile concepts in the data world, data science teams are exploring some of the same techniques that spurred enthusiasm back in the early 1990s with software developers who were trying to overcome the inherent liabilities of waterfall-based Software Development Lifecycles (SDLCs).

Pair programming has always been a sticking point, though, even when it was first introduced. Based on my experience in working with data teams for over 20 years, the gains are worth the challenge, but it's an estimable one. When structuring the workflows of your data science team, meet the challenge and embrace the best practice of pair programming.

Is this really a best practice?

By far, the largest challenge you'll face with pair programming is perceptual. To be clear, in the data science world, the concept can be extended to solving difficult mathematical problems or creating data visualizations; however, I haven't found the same perceptual problems in these areas.

Outsiders immediately see waste when they see two people programming together. So, if you start playing with pair programming and your sponsors don't know about it yet, they'll find out very soon. For this reason, I strongly suggest you build an ironclad business case for pair programming and have it fully endorsed by your leadership. I can assure you that pair programming does not cost twice as much to produce code, but it does carry a financial premium: somewhere on the order of 10 to 20%. Simply put, pairs code much faster but not twice as fast. Add the soft benefits of accelerated quality, high morale, and increased ideation, and you easily push the value of pair programming way over its alternative.

However, advice from a free article on TechRepublic hardly builds a tenable business case, so you'll need to experiment for yourself. Run a controlled test with pair programmers, having your traditional programming style as the control set. Your null hypothesis is that pair programming costs twice as much as normal programming — this is the default assumption. Your measure is cost per unit of code delivered, so you'll need to figure out what a unit is. This can be tricky unless you're following an SDLC that caters well to normalized code blocks like Feature-Driven Development (FDD). Most likely you're not, but it helps to frame this discussion if you understand FDD-ish concepts. Then, let the data speak for you. There's no reason to put your job on the line for an ideology.

Contending with process and workflow issues

Once you've slayed the perceptual Hydra of pair programming, you've tackled the lion's share of your challenge, but you still have to contend with process and workflow issues.

An often-overlooked land mine with pair programming is personality. Many data scientists have a problem with so much personal interaction. In the pair-programming world, your partner never leaves your side. Most data scientists need their quiet, personal time; therefore, an extended period of pair programming can be quite disturbing and disruptive. You'll need to disclose this upfront, and make sure those who participate agree. Don't ever force a data scientist to pair program with someone — this will definitely backfire.

Another process issue is pair construction. Since data science is multi-disciplinary in nature (advanced mathematics, artificial intelligence, computer programming, business intelligence, data visualization, etc.), it can be difficult to get all disciplines represented in a pair. This is easier if you have generalists (i.e., where each data scientist has multiple skills). One way to handle this is to build specialists into generalists, but this can take time. Another approach is to form nuclear teams of more than two people. There's nothing inherently wrong with triad programming or even quartet programming — just make sure you have the business case to support it.

Finally, you'll need to run interference with outsiders if you plan to adopt pair programming. Even with a strong business case, it doesn't look right, and your team might be harassed as a result. It's helpful to have at least one person assigned to monitor and intercept incoming flak. This could be your analytic manager, your team coach, or even your evangelist who helps with external change management. At the end of the day, your pair programming team should be isolated from naysayers. It's hard enough to get data scientists into the groove — they don't need outside negativity messing up their mojo.

Summary

With all the benefits I feel pair programming can bring to your team, it's one of those things you'll need to experience for yourself. The controversy over its efficacy will not end any time soon, so you'll need your own personal stories to have a say in the argument. Take some time to organize an experiment so you know your business case firsthand. Then, if it makes sense, follow through by making sure your team has the right disposition, tolerance, and protection to make it happen.

Whatever you do, don't listen to the advice of naysayers who have never practiced pair programming or done their diligence to understand its business case. How can you be so sure the world is flat if you've never seen an edge?

Also see

About John Weathington

John Weathington is President and CEO of Excellent Management Systems, Inc., a management consultancy that helps executives turn chaotic information into profitable wisdom.

Editor's Picks

Free Newsletters, In your Inbox