Open source magic solves a months-long problem in 20 minutes

Commentary: Capventis and other system integrators increasingly depend on open source to help solve critical enterprise IT issues.


Image: iStock/undefined

As more enterprises lurch towards digital transformation and cloud-native architectures, accelerated by the urgency of business changes during the COVID pandemic, many depend on systems integrators (SIs) for help. If you look under the hood of most successful digital initiatives created by SIs (or any company), you'll find the engine involves a great deal of open source software (up to 90%, according to Sonatype data).

This rush towards digital transformation preceded the pandemic, of course, as enterprises also saw the need to move faster in their markets and accelerate the introduction of new technologies. But SIs have become key partners in driving open source deeper into enterprise IT, with projects like GraphDB increasingly important to solve data integration issues. Capventis offers a good example of how this works. 

SEE: Research: Digital transformation plans shift due to COVID-19 (TechRepublic)

Data problems

Every industry is trying to get to the future as fast as possible, and telecommunications is no different. As Iain Morris called out in a Light Reading article, in 2018 France's Orange estimated that a third of its global workforce--more than 50,000 employees--needed reskilling if the company hoped to keep up with cloud vendors. In that same article, Morris pointed out that Spain's Telefonica figured it would need nearly $2 billion in staff training and early retirement buyouts to bring in new talent with new skills to be competitive.

Such telcos often turn to SIs, like UK-based Capventis, who in turn bring domain expertise and work primarily with clients in the Business Intelligence (BI), Customer Relationship Management (CRM), and Customer Experience (CX) fields. These areas haven't traditionally been ripe for open source, but even SIs with these focus areas rely on open source software to help their clients. It's hard for even the best proprietary software vendors to keep pace with the innovation cycles of successful open source projects; so, these SIs will partner with companies like Alteryx, Qlik, Qualtrics, and Zendesk, augmenting their proprietary software with open source expertise.

In the case of Capventis, a big part of its projects involves pulling vast quantities of legacy and real-time data for its clients. It reminds me of a conversation from a decade ago with MuleSoft creator Ross Mason, who had started his career with an SI. He created MuleSoft (then called MuleSource) as something like an enterprise service bus to move lots of data around and connect it to an application or service. He kept having to do the same thing--moving lots of data--over and over for his clients, and he described it as "donkey work." So he wrote MuleSource, which eventually ended up being acquired by Salesforce for more than $6 billion.

Today, Capventis faces the same challenges but at a larger scale, as data volume, variety, and velocity grow exponentially. "A client might have Zendesk for tickets and rely on Qualtrics for surveys," said Mike Hawkes, CTO of Capventis. "But they all rest on a giant old Oracle database. There are typically many legacy systems that need to interact but none have a connector. And those that do have connectors don't handle maintaining integrity between various systems."

Capventis wrote its own data stack to manage this variety of data and integrations, giving it the appropriate name of Glü. But the company also needed some open source magic.

Integrating data the open source way

Across its client base, Capventis increasingly faced challenges with the limitations of SQL in scale-out, integration projects using legacy SQL databases with tabular structures that required large numbers of joins to integrate disparate data sources. After Capventis won a major UK government project, it struggled with how to integrate several older, proprietary databases that were still in production and being regularly updated with new information. The database project involved multiple departments that over time had merged and separated and merged again. What should have been a dozen tables had exploded into more than 300. The legacy database providers refused to collaborate on the project.

SEE: How to build a successful developer career (free PDF) (TechRepublic)

Stymied, Capventis reached into its open source software bag of tricks. Based on the scale and scope of the project, the team wanted a GraphQL-based interface with a flexible, reliable, and high-performance graph database on the backend.

For this project, Capventis selected GraphDB from Dgraph Labs after an internal technology bake-off. Dgraph outperformed the others and had the added advantage of a global developer community that the company found to be responsive and friendly. (I've written about Dgraph before.) The Capventis team used Dgraph to convert all the legacy data from multiple sources and cleansed it on the fly with no data loss, while simultaneously generating a schema read for immediate queries.

"We linked Glü to their database servers, fetched all the data, and threw it into Dgraph," Hawkes said. "Within 20 minutes we had the entire structure of the data set with all the proper interactions captured in Dgraph. We solved this problem in one hit. It was months in the planning and minutes in the execution."

That's the power of open source.

Disclosure: I work for AWS, but the views expressed herein are mine.

Also see