Just as the chief executive officer (CEO) embodied the growth of multinational corporations in the 20th century and the emergence of chief information officers (CIOs) paralleled the expansion of the internet and the World Wide Web, the rise of chief data officers (CDOs) in the 21st century reflects the central role that data now plays in every facet of society. These individuals are entrusted not just with managing information but going one layer lower in the knowledge generation and management process to raw data creation, collection, storage, and analysis.

The role fits the times. Our exabyte age has led to an explosion of data creation and storage, with associated analytical needs that “exceed the capacity of traditional processing systems.” In increasing numbers of organizations and businesses, that means the CIO tasked with managing the IT infrastructure at a government agency or enterprise is collaborating with a CDO to ensure that data is available for organizational needs and strategic decisions.

Perhaps for that reason, a 2015 report from IBM’s Center for Applied Insights (PDF) found that 61% of the 250 CIOs surveyed at large organizations wanted their employer to recruit a CDO in the next year.

Gartner estimated in January 2014 that by 2015, 25% of large global organizations would have appointed CDOs. As of March 2015, its research VP said that there are more than 100 CDOs serving in large organizations. According to a 2014 report compiled by Randy Bean, the CEO and managing partner of consultancy NewVantage Partners, 43% of the 59 Fortune 1000 companies he surveyed had appointed a CDO.

In the public sector, CDOs are making data a public asset, releasing data for job creation, applying data to policy formation, and using predictive analytics to save lives and taxpayer dollars. The first chief data scientist in the history of the US is focusing on precision medicine, data products, and ethics.

“The emergence of chief data officers at major agencies and departments is a promising sign that the federal government continues to execute on President Obama’s open data vision and his 2013 Executive Order,” said Nick Sinai, a Walter Shorenstein fellow at the Harvard Kennedy School and former deputy U.S. chief technology officer, in an interview.

“CDOs are charged with helping agencies get organized and better manage data as a strategic asset. In my mind, they are also chief evangelists for their agency’s data, responsible for increasing the feedback loops about where to focus agency data liberation efforts.

Band-aid or force multiplier?

CDOs have been getting a lot of attention in the media in recent months. This spring, Nextgov ran a special report on the “The Rise of the Chief Data Officer” that featured interviews with several of the men and women in these roles and put together the beautiful timeline embedded below.

While the overall list of federal CDOs is (slowly) growing and Government Technology found over a dozen city and state CDOs in the US last June, what people in the role should be doing, how, for whom, and with what impact is still up for debate, including whether a CDO is a meaningful addition or a band-aid.

Will adding a new person to the executive suite or further down the reporting chain in a huge bureaucracy really have a measurable difference upon how an agency or enterprise stores, manages, or shares data? For instance, hiring and installing a CDO at the Centers for Medicare and Medicaid Services (CMS) hasn’t prevented thousands of banned healthcare providers from continuing to bill state Medicare services, according to an investigation by Reuters. While the issues can’t be laid entirely at the CDO’s feet — CMS is the same agency responsible for the issues behind the failed relaunch of HealthCare.gov — data collection and sharing is clearly core to the role.

“Ultimately, CDOs should be judged on whether they make it easier for users, inside and outside of government, to find and use valuable government data to advance the public good and spur economic growth,” suggested Sinai.

Much of the work that people who are pioneering CDO positions in big bureaucracies remains necessarily “remedial,” as the UK’s first CDO told me. That means “finding out where data is, what contracts govern it, how it can be used, what standards it’s in, and getting it out of legacy IT systems,” said Mike Bracken.

Just as is the case with data-driven journalism or data science writ large, the big hurdle to gaining accountability, economic value, or insights from data is the janitorial work of cleaning it up.

At the cutting edge are CDOs who inform policy and drive insights. They’re not only cleaning and releasing data for public use but delivering improvements to performance and applying analytics to support governance and regulatory actions.

Over the last month, I’ve talked and corresponded with a number of current and former CDOs, from the national level to municipalities. Their perspectives and stories not only tell the tale of their achievements and challenges but also offer insight into those their successors and colleagues will face.

What does a CDO do?

“Generally speaking, the role of a chief data officer is to organize and manage an organization’s use of data to fulfill its mission,” said Mark Headd, developer evangelist at Accela and the first person to hold the role of CDO in the City of Philadelphia’s history. “That said, the specifics of what any CDO is charged with may depend on what a statute or policy says. My job in Philadelphia was informed by the language in the Mayor’s Open Data Executive Order.”

As more people hold the job and organizations learn from successes and failures in integrating CDOs into leadership and strategy, the role of the CDO may change. Their primary function is, however, exactly what common sense would suggest: connecting data to mission.

“For my organization, I have broken down the responsibilities into three categories: Govern, Engage, and Enable,” said Dan Morgan, CDO for the U.S. Department of Transportation (DOT), in an interview.

“Governing is advocating for policies and processes that support information management, quality, and stewardship. Engagement is connecting data producers with consumers to identify areas where we are doing well and where we can improve, celebrating successes through events like datapaloozas, and engaging around difficult problems through data jams and competitions. Enabling is diversifying the toolsets we have to acquire, store, process, analyze, and apply our data, and helping my agency more effectively accomplish its mission by extracting value from its data, other government data, and other sources.”

In large bureaucracies, like government agencies and enterprises, culture can have as much influence on how data is collected, shared, and used as technology limitations.

CDOs are responsible for “pushing the organizational culture around data and data management,” said Micheline Casey, the CDO of the Federal Reserve Board, “ensuring the alignment of the data strategy to the business, and operationalizing a data governance framework across an organization.” Casey was the first state CDO in the US, serving in Colorado before she joined the federal government.

Data officers also are responsible for ensuring that people inside of an organization can find what they need. Improving internal information retrieval rates has frequently been cited to me as a primary return on investment for open data initiatives. According to the McKinsey Global Institute, government workers spend an average of 19% of their workdays looking for information.

A CDO has to “take stock of internal data and make them accessible inside the organization,” said Peter Speyer, the chief data and technology officer at the Institute for Health Metrics and Evaluation (IHME) at the University of Washington in Seattle.

“Additionally, a CDO must understand the different internal and external audiences for the data and provide them with the right formats and data products, and engage with the broader data community, including discussions about data sharing and open data, which is key for health, where data can help save lives,” he said.

Speyer also emphasized the need for CDOs to be involved in securing data.

“The CDO should be responsible for or take a lead role in ensuring data security to ensure confidentiality and privacy, and ensuring that external stakeholders understand that the organization is treating data responsibly,” he suggested.

Finally, CDOs can inform policy and focus on performance.

“My perfect model for a CDO is someone who operates in three areas, the first being open data,” said Brett Goldstein, the first CDO for the City of Chicago, in an interview.

“That’s a relatively obvious one, and it’s a relatively common thread for the public side. The second one is performance management, where you have your standard key performance indicators and management by objectives and all of those typical things. I draw the analogy to ‘I have a 311 service. What are the metrics that I have on there?’ The third piece is the area of advanced data analytics and predictive data analytics. One of the things that worries me is that we don’t really see as many models which are covering the three. I think all three areas are critically important.”

CDO vs. CIO?

When I asked current and former CDOs if there was any difference between what a CIO and a CDO does, some themes emerged.

First, people pointed out how CDOs are similar to CIOs on a few levels.

“The job of a CIO is to manage an organization’s use of technology to fulfill its mission,” said Headd. “There is almost always an overlap between data and technology, because tech is used to produce, store, and transmit data. In other ways, it is different. The CDO role can have a significant external focus (outside the agency and the government) that CIO roles often do not have. This is usually an issue when CDO roles are placed inside the technology agency, as they often are. Central technology agencies are almost always inward-facing: they support the agencies that provide frontline services. A CDO must communicate both internally and externally.”

A significant difference in the roles may lie in the responsibilities of modern CIOs being broader within an organization than gathering, managing, publishing, or analyzing data.

“I assertively differentiate this from a CIO, even though I at one point held both roles,” said Goldstein. “The CIO should be data-informed, but whether you’re in the public or private sector, you have a vast amount of things you’re going to cover. You have everything from messaging to infrastructure to data center to ERP. It’s somewhat unreasonable to ask for that level of expertise. You do still see it in Chicago, where the CIO has a lot of passion around the data portfolio, but they need to be able to have full coverage, and with that not the depth that you need for a CDO.”

At the level of national government and agencies, CIOs are juggling multiple responsibilities beyond data warehousing, from security to data centers.

“Not only do CIOs have a responsibility to acquire, manage, and secure information technology, they also have a statutory responsibility to ensure their data and information are managed to increase program efficiency and effectiveness and improve the integrity, quality, and utility of information to all users within and outside the agency,” said Morgan. “That’s why, organizationally, I report to the Chief Technology Officer who, in turn, reports to the Chief Information Officer.”

Both CIOs and CDOs support business needs, whether in enterprise, commerce, or academia. If using performance metrics to manage IT relies upon accurate data, that means the CIO will depend on the CDO to get it right.

“The CDO is responsible for working with the business lines to define business rules and roles around data, in addition to aligning the organizational strategy around data,” said Casey. “The CIO is responsible for implementing technology architectures, infrastructures, and policies in support of business and data needs.”

Inside or outside of the public sector, the CDO can also help other executives and managers by bridging internal silos.

“The role is complementary to CIO and CTO by helping organizations capitalize on data,” said Speyer. “The CDO will rely on the CIO for IT and infrastructure support. He or she should collaborate closely with the CTO on external engagements. There is some potential overlap in responsibilities for creating data infrastructure and applying data science that need to be defined, depending on the organizational structure, skills, and experience of the different officers. The CDO is very much an interface role.”

A changing role and portfolio

What CDOs were doing years ago has shifted a bit, in the view of some of the older hands. CDOs are shaping one of the newest roles in governance to fit the needs of their organizations.

“My work in Philadelphia was heavily informed by my time as a software developer,” said Headd. “I focused a lot on providing materials for external collaborators to use, publishing data sets meant to be used by civic hackers, entrepreneurs, journalists, etc. I think Tom Schenk in Chicago is bringing a real data science focus to the role of CDO. Abhi Nemani is doing some really interesting work around performance metrics for LA government.”

At the federal level, more CDOs at agencies are thinking about making data actionable internally, not just publishing it externally.

“It’s grown beyond just having an open data focus to beginning to support central data management and governance operations,” said Casey. “Certainly, there is greater awareness for the need for the role and a maturity in thinking about the services, capabilities, and value the role can bring to organizations.”

Goldstein told me, however, that some of the changes he’s seen in the CDO role since 2011 in the public sector worry him.

“If we go back to the three verticals that I think are important, I don’t see as much of all three,” he said. “It seems that there’s a lot of effort within open data, which is great, and transparency is super important for all levels of government. We need to make sure — and I joke, since this is the name of the book we worked on, “Beyond Transparency” — that we’re getting beyond just data sets onto the Interwebs and ‘spreadsheets on the web.'”

Goldstein places informing policy at the core of the CDO role inside of government.

“Policy has a qualitative aspect, it has a public policy mindset, but we need to introduce quantitative rigor for both tactical and strategic decisions,” he said.

What should the CDO be doing?

Every CDO I spoke to, past and present, agreed that their job should focus on advancing the mission of their organization with data.

“In general, the CDO should be involved in, and in many cases responsible for, any activities along the data value chain,” said Speyer.

More broadly, some connected their work to governance, from delivering services to protecting the public or public interest. At the Department of Transportation, there’s also an explicit connection to support the missions of others.

“The CDO should be focused on helping their organization get more value and insight from the data it collects,” said Morgan, “and on helping people accomplish their goals by lining them up with the process and technology components that are critical to the organization’s strategic goals.”

Over the past year, a number of CDOs have also taken on the mantle of evangelists for the increased use of data in many contexts. That can mean “elevating the awareness and conversation internally regarding the importance of well-run data operations, supporting organizational culture change, and championing and evangelizing a data-driven culture,” suggested Casey. In her view, CDOs can be core to providing data governance and data management services to the enterprise, spanning divisional silos, setting standards and policies in the process.

If a CDO has enough statutory authority and political buy-in, setting standards for data can be a powerful forcing function for interoperability inside and outside of government and a bulwark against vendor lock-in.

“No one is in an optimal state,” said Bracken. “Many of our standards and contracts were not designed for the modern age. We will be bringing data out to have a closer look at it with my colleagues. We accept we are not currently in a great state to be interoperable. We will be setting rules of the road for government for standards internally, and externally, so that users, suppliers, and the whole ecosystem can understand them. We have 24 departments and around 300 agencies. I’m pretty sure there isn’t a single data standard across them.”

What are the primary tools that the CDO uses?

When I asked a series of data officers about their tools, I expected to hear about the “sprays, mop, and bucket” of the janitor: whatever they use to collect, clean, harmonize, and store data.

To some extent, that was true for more technically inclined people, but Casey listed also communication and education, governance, committees, budgetary levers (to the extent they exist), data management tools, policy and process engineering, and standardization.

“The CDO is part facilitator of data-related discussions and decisions, part decision maker with line management functions,” said Speyer. “The tools needed are mostly management and project management tools, although a hands-on CDO may actively use database or analytic tools, especially in smaller organizations.”

Most CDOs, however, talked about policy and people skills, driving home the point that the hardest part of their work doesn’t involve technology.

“In my short time as CDO, I have found that my most important tools are listening and convening,” said Morgan. “In practice, that means coming out from behind my computer and engaging with the leaders who are guiding the DOT’s programs to deliver a safe, efficient, accessible, and reliable transportation system. I listen to their challenges, and connect them with the tools to help them use their data to accomplish their goals.”

Part of the innovation around data often has nothing to do with data, suggested Ian Kalin, the first CDO at the U.S. Commerce Department.

“It’s what processes exist and what tools am I allowed to use,” he said. “Am I allowed to use GitHub or not? These are still questions that are being asked throughout all parts of the federal government. That’s not really a data question — that’s a governance question. You need to figure out those things first before you can even get to how you can release information in a better way. Part of that is the processes and partnerships within individual companies.”

At the municipal level, the CDO role itself has been created by executive orders and legislative actions, as in Philadelphia, or open data legislation, as in San Francisco. While there can be ambiguity, whether the men and women who occupy the role carry enough statutory authority to compel compliance is sometimes in question, depending upon where they’re placed. That said, there’s no doubt that the laws and rules are important levers.

“I used the Mayor’s Executive Order quite a lot,” said Headd. “If governments are serious about making the CDO role a success, it must have both policy and budget support. These are the tools that a CDO should use to achieve their objectives. I think this is how anything inside government — particularly culture and behavior change — ultimately happens.”

That doesn’t mean, however, that CDOs are going to flourish if they’re non-technical. Great communication skills aren’t enough to decode dirty data or see flaws in analyses.

“When you’re working in government, you’re constantly barraged by proposals and ideas,” said Goldstein. “You need to be able to rapidly vet them, know what is real, what is legitimate, what is sized correctly, and that’s important.”

He also highlighted the value of having “geeks in government” who are fluent enough to be able to engage directly with developers, particularly when quality or standards issues arise.

“We are going to get deep into these fields,” said Goldstein. “Deep into open data, deep into data analytics, deep into prediction. You need to have leadership who has depth, who knows how to call what is real versus not real, and be respected by folks out there. You need some deeper skills to get there. I am not saying we need to have super technical candidates in all cases, because I want people in these roles who can straddle the worlds, that can go from understanding the innards to sitting down with the mayor or the president or whomever and saying ‘here’s how it ties to the policy’ in a coherent, smart way.”

What is a CDO’s biggest challenge?

As is often the case with newly created roles, getting the right scope of responsibilities and authority aligned within a huge organization’s hierarchy isn’t easy.

“There are many potential issues that can make this difficult: overlap in function with CIOs and CTOs, data ownership in different parts of the organization, a lack of collaboration to agree on standards across the organization, too little authority over what happens with the data,” said Speyer. “Full backing from the CEO is essential.”

In government, full backing from a mayor or president is similarly key to driving change, particularly in the face of organizational culture.

“Getting things done inside government — especially things that are new, and where the certainty of success is not absolute — is hard,” said Headd. “Really, really hard.”

Morgan said that prioritizing, focusing, and educating people are the most challenging parts of the job. He suggested the MIT ‘cubic framework’ for CDOs as a useful resource.

“The framework points out that there are 8 different roles for CDOs,” he said. “There’s no shortage of opportunity to get more value out of our data.”

How should a CDO’s success be measured?

Given how data-driven CDOs tend to be in their work, the question of how their own impact should be quantified is relevant. To some extent, the impact can or should depend on context for their work and their mission.

“I think good open data policies are structured with milestones that can be used to evaluate the progress being made,” said Headd. “Some cities produce an annual report. I love Chicago’s report.”

Kalin was inspired by two examples of public transparency at the state and local level. “One is the citizen dashboard for Edmonton, Alberta, and the other is StateStat for recently departed Maryland Governor Martin O’Malley,” he said. “Both of those government administrations have really good dashboards. Good, highly specific, data-driven, business-owned. They have scorecards that show the underlying information behind them, with live updates, which show if I’m doing badly as much as I’m doing well. They were transparent about it, they were open about the failures as much as about the wins. That’s pretty rare.”

Morgan said that the CDO’s impact should be self-evident to the internal and external users of an organization’s data.

“Improved quality should result in increased user satisfaction, an increase in the number of people using our data, and ultimately an improved mission outcome,” he said.

Speyer aligned the impact of a CDO in an organization against specific outcomes as well.

“A CDO should maximize the value of the organization’s data,” he said. “Internally, that means transparency about data assets within the organization. Data should be available where they are needed, in the format they are needed. Data need to be secure, and confidentiality and privacy of those providing the data protected, especially where there is information about individuals. Externally, the organization should be perceived as treating data responsibly. Finally, if there is a potential for data sales or data product, their impact should be maximized, e.g., by measuring revenue, usage.”

What do people get wrong about CDOs?

“I wouldn’t say that people get things wrong about the role, but very often the discussion focuses on enabling technologies,” said Morgan. “CDOs are also leading culture change — the people aspect of the job is essential to success.”

Like Goldstein, Casey highlighted potential issues surrounded the way CDOs are positioned within a structure. “It should not report into the CIO or IT divisions,” she said. “They’re also typically underfunded and understaffed.”

Headd also thinks that placing a CDO within a technology agency can cause problems for city government.

“Most central tech agencies are inwardly focused, and a CDO role has a significant external focus,” he said. “That said, I don’t know of anyone that is doing it wrong. To some extent, we’re all still figuring out how this is supposed to work. I’m in awe and learn much from every CDO I know, and I know most of them quite well.”

How does the role of a CDO in the public sector differ from one in industry?

While the name of the title may be the same, the context for the work of the CDO in the private sector and public sector take place in different contexts, with different goals, despite the shared language around “business needs.”

There can be some similarities.

“I don’t think there’s a binary distinction here,” said Morgan. “CDOs change based on their organizational need, whether they are in the public or private sectors. The MIT ‘cubic framework’ for the CDOs is my go-to reference on this topic.”

In the view of Casey, the work of public sector CDOs are generally not tied to improving revenue or improving some measure of customer retention or service.

Public sector CDOs are also entrusted with stewardship and sharing in a way their private sector compatriots may not be.

“There is a general responsibility for the public sector to make data available to external audiences, as open data or via appropriate data sharing mechanisms,” said Speyer. “In industry, competitive considerations will make this more difficult.”

That attitude to data sharing was common among the CDOs I interviewed.

“What I think is a unique obligation, though, from the government perspective, is one of equity,” said Kalin. “Some of the information is easier to access if you have certain tools vs. another, if you have a computer or if you don’t. I think there’s also a philosophy, which I care about personally, about how the data probably doesn’t belong to the government: it belongs to the taxpayers. We have a responsibility to be good custodians of that data, to be responsible stewards, to protect privacy and confidentiality. Ultimately, it is a service that we provide. For the American taxpayers paying for the data, there should be a way and a process for the people to have the data brought back to them.”

The biggest difference, however, is politics, suggested Headd.

“Private companies that don’t use data to innovate, serve their customers better and make better decisions typically feel it in their bottom line,” he went on. “Governments don’t necessarily have this same set of incentives. If your experience paying your parking ticket is lousy, you still have to pay the next one.”

Who’s doing it right, and how?

“In the public sector, the role is relatively new,” said Morgan. “Some of my mentors are Donna Roy from the Department of Homeland Security and Micheline Casey from the Federal Reserve Board. They’re successful because they connect to their organization’s mission and are drivers for change.”

Speyer highlighted the work of former U.S. chief technology officer Todd Park.

“As CTO at HHS, he filled much of the role of a CDO, getting the conversation about data going across the complex HHS organization, and making a lot of progress in terms of releasing data and stimulating the external conversation about data and data sharing,” he said. “Niall Brennan at CMS seems to tackle the role the right way, with the support of the larger HHS open data initiative, but he is new in the position.”

At the municipal level in the US, while CDOs at Chicago and New York have been setting the pace, the work of CDOs in Los Angeles, San Francisco, and Philadelphia is also getting attention. They won’t be alone.

“I think what Laura Meixell in Pittsburgh is doing is groundbreaking,” said Headd. “The future of open data is in smaller cities, and finding ways for government to collaborate more effectively. Watch what happens there in the months and years ahead.”

Author’s note

This week I began a new job at The Huffington Post, which means that this is my last feature for TechRepublic. It’s been a pleasure and privilege contributing here and corresponding with many of you. I hope that you’ll continue to read me there: your perspectives, knowledge, suggestions, feedback, and ideas have helped to inform me and thereby inform others.

You can find me @digiphile on Twitter or facebook.com/alexhoward if you’d like to stay in touch “socially.” So long, and thanks for all the fish!