Years ago, Nick Carr asked "Does IT Matter?" He came to the conclusion that it didn't, that most companies spent a lot of money on undifferentiated technology that they could easily outsource.
But that was before big data hit, which has made IT matter a great deal.
Because all the best big data technology is open source, the rise of big data also signals the rise of the developer. Unlike past generations of IT, modern IT requires developers that know how to craft hardware and software to an enterprise's business goals, rather than ingesting whatever some enterprise IT company's sales rep sold your CIO on the golf course.
Capital One gets serious about IT
Capital One, a large US bank, offers a good example of what modern enterprise IT looks like.
In a conversation with The Platform, George Brady, executive vice president of technology, outlines the company's revamped approach to its software infrastructure:
"More and more of the development we are doing is for homegrown applications. Back in the early 2000s, it was really a mix of commercial software and built, and for the past seven years, there has been a pivot to more homegrown software where we think it makes sense and provides some differentiation."
Much of this "homegrown" technology depends upon open-source foundations, which includes data analytics: "Data analytics at Capital One was built on what Brady refers to as 'proprietary technologies' in the past...[but now] 'We are focused on Hadoop and Spark and we are a couple of years into that journey.'"
In fact, the company is so centered on creating its own destiny through software that it has launched an open source project, Hygieia, to track data coming from its DevOps tools. Not only is Capital One using others' open-source code, the financial services giant is contributing its own.
And what about hardware? Surely that's something the company must pick up from an HP or Oracle, right? Wrong.
"The company has historically relied on tier one system makers... for its Xeon-based server platforms, but like his prior employers Fidelity Investments and Goldman Sachs, Capital One has been actively involved with the Open Compute Project launched by Facebook and its relatively new Hadoop cluster, which weighs in at 2 PB and which is growing fast, is based on OCP-inspired servers, storage, and racks. Capital One uses the Cloudera Enterprise Hadoop distribution at the core of its new data analytics platform."
In this new world, even customized hardware becomes critical. Top to bottom, Capital One demonstrates that IT matters a great deal.
Nor is it alone. BBVA (a large Spanish bank) recently published the architecture of its big data lake that is, not surprisingly, heavily dependent on open-source technology like Apache Spark.
Developers, developers, developers
In neither of these cases, or any others, are we seeing low-level sysadmins taking on the brunt of building out such technology stacks. With so much at stake, attracting top-notch developer talent becomes critical.
This is why some industry observers like Dana Evan declare, "I've never seen the competition for developer talent as fierce as it is today."
That's true of Silicon Valley developers, of course, but it's increasingly true of Peoria, IL; Colmar, France; and Fargo, ND.
For enterprises of all shapes and sizes, getting serious about developers means they must get serious about open source. As I've written, "Not every company has an open-source page. But every organization increasingly speaks open source, because they must. Developers demand it, and organizations depend on developers to invent the future."
Or as Adrian Cockcroft of Battery Ventures (and formerly Netflix's cloud chief) puts it:
For companies looking to win in the 21st Century, they're going to have to get serious about open source, because that's the currency for attracting the best developers. Those developers, in turn, can help build software and hardware that can offer significant, enduring competitive differentiation.
It will result in fewer rounds of golf for the CIO, but that's a small price to pay for winning.
- Barrier to entry: Busting more myths about Linux
- When NoSQL may not be right for you
- End of the fail whale: How Twitter aims to help mere mortals scale like a web giant
- The truth is just a download away: Why we need open source more than ever
Matt Asay is a veteran technology columnist who has written for CNET, ReadWrite, and other tech media. Asay has also held a variety of executive roles with leading mobile and big data software companies.