In 2009, the White House began its Open Government Initiative, an effort to increase transparency of operations and encourage collaboration with citizens. Beth Noveck, the first US Deputy chief technology officer, directed the project.

TechRepublic talked to Noveck about how her experience shaped her new book, Smart Citizens, Smarter State: The Technologies of Expertise and the Future of Governing, in which she offers insights into the process of opening up government–and why it’s essential to our democracy.

How did the rollout of the open government initiative turn out? Has it been difficult to implement?

It was intended to use technology to improve the working of government. We’ve been introducing technology into government for a long time, including the introduction of the typewriter, which had a profound effect on bureaucracy. But we wanted technology not only to make government more efficient, but more effective and innovative about how we solve problems.

Businesses are trying to use data better to understand their customers, to understand their supply inventory, too. Trying to use data to understand better the customer, the service it delivers, and how to do it better.

When it comes to data, we’ve had some great successes–ways in which data has been transformational and not just a faster way of doing the same old thing. We’ve begun to open up data, recognizing that the person who collects the data isn’t always in the best condition to use that data. For example, the government might collect data from a hospital about the rate of hospital infection, but it doesn’t mean government by itself is in the best position to make sense of that data and use it to develop ways to reduce the rate of hospital infection. When you open up that data, companies like Google and Bing can use it to power their search engine. Then, as a patient, I can search for a hospital or doctor or healthcare facility and make an informed choice about where I’m going to have a procedure done, for example. It allows government to do its job, but it also allows companies to create new tools and consumers to make more informed choices. We need to create more opportunities to participate in problem-solving.

The challenge is that we have now made communications tools available, but we haven’t had a great deal of success yet at making government more participatory and collaborative and getting smart ideas in from smart people outside of government. So I ask, “Why not?” How can we use technology to change that?

Has the move toward open government been the result of new technology, or a mindset shift in our culture?

Yes and yes. On the one hand, we have new kinds of communication tools that allow us to talk one-to-one and many-to-many. It’s more than we’ve ever been able to do to exchange and share information. We now have new tools that could allow for greater communication and collaboration. But more important than the technology is the cultural shift. The fact that, with the tools the web has given us, we have greater expectations for participation.

Companies are asking their customers to design a better Super Bowl ad instead of asking Madison Avenue to do it. Local Motors asks customers to design cars that they sell. Technology has created more opportunities for engagement, which in turn drives the demand to participate and engage. Today’s young people have the expectation that they will have a more participatory experience–I’m sure they’re going to demand it from their government.

How has the tech industry been successful in calling out for ideas through crowdsourcing?

We have lots of evidence from the tech industry about the use of technology in present communication tools as a collaboration tool to engage in crowdsourcing or open innovation, calling on the internet for people to participate. And some pharmaceutical companies have set up a platform called InnoCentive to engage a community of what they call solvers outside the pharmaceutical industry. Members of the public compete on InnoCentive to solve our R&D problems in exchange for prizes. Following that example, the federal government set up Challenge.gov, a prize-backed challenge website. Over the last five years, 450 different challenges have been set up with prizes ranging from meet the President to winning millions of dollars to engage the public in solving public problems, not just private problems.

So, when a Netflix or an InnoCentive puts out a private sector problem, Challenge.gov asks the public to come up with better ways of removing the salt from salt water to aid with agriculture in the developing world. Right now, there’s a challenge to come up with technologies to reduce the word gap between the number of words a rich child and poor child hears over the course of the first five years of life, which has a significant impact on that child on their educational attainment and other life goals.

What lessons could government learn from the tech industry?

Intelligence capital exists in our communities, and we have to have more targeted, reliable ways of getting at that expertise. We can put a challenge on Challenge.gov about reducing the word gap, but it doesn’t mean that the people most likely able to solve that problem are even going to know about the challenge. So, the question is, how do you alert people about opportunities to participate?

What government needs to learn from the tech sector is the way in which technology is helping pinpoint different types of audiences. The private sector is really good at what they call audience segmentation, using data-rich tools to determine which customer is likely to buy your tube of toothpaste at what time of day. It’s also been the private sector that can pinpoint which employees are responsible to hire by using online tools that help me know what a person’s skills are so I can make a better hiring decision.

We need to match people to opportunities so they can participate. The same technologies that we use to match people for dating, we can use to match people to opportunities to do good in the world and engage with others. The answer lies in the tools, in how we market opportunities to participate in order to match people to opportunities to participate in ways that speak to their abilities.

Technology has upended almost every other industry, including campaigning in government. Why has it been less successful in changing government?

Technology works really well when you know what the process is you’re trying to implement, when you know what your software is trying to do. If you’re trying to get people to vote or give money, it’s much easier to identify ways of using tools to accelerate those practices. If I have to go door-to-door to get people donate, it’s not going to be as effective as the ability to raise money from them on the web or by mobile phone.

The challenge with using technology to transform the way we govern is that decision-making is a very complex process, which is not well understood by most of us. We can’t use reengineering if we don’t understand how it works in the first place for what we’re trying to do. We haven’t been open to the notion of getting help from outside of government. We have a lot of examples of technology making government more effective at solving a problem or mitigating a risk that I think are very exciting.

You write that today, government faces a more varied set of challenges than in the past. How so?

The world is bigger, more populous, and more crowded. We’re also seeing the magnitude of the problem of climate change, terrorism, domestic and foreign. The scale and scope of some of the disasters, both manmade and natural, are literally more dangerous and serious than changes that we have ever faced. We have a more dense, diverse world of people within closer proximity to one another.

And now, there’s the existence of communication networks that make what happens in one place absolutely current and dependent on what happens in another place. We’ve had this view in the past you do policy intervention A and it leads to result B. I’ve come to learn it’s never so neat as that and we cannot predict the result so easily. A could lead to B, which would lead to C, which would lead to unpredictable event D.

The challenge is just the mismatch of the supply of good ideas and the demand created by the problems we’re facing, and having institutions that are more agile and more experimental in their ability to apply scientific advancements to developing challenges.

Can you use an example like recidivism to illustrate how collaboration could solve that kind of issue?

We are one of the countries with the largest prison population in the world. Whether you take the concept of how to prevent crime in the first place by trying to ensure better delivery of both mental health and education and healthcare services to vulnerable populations; whether it’s about being able to come up with better programs to help to reform people while they’re in prison to reduce their risk of recidivism; whether it’s about identifying better strategies for those most at risk of reoffending, improving our algorithmic predictions of recidivism; or whether it’s about providing better services to people after they leave prison–at any one of these stages, and plenty of places in between, there are people out there who have good ideas about how to improve or solve that problem.

It might be somebody who is working the criminal justice system who’s had really good luck with a particular algorithm used to identify risk of recidivism. Or, maybe it’s a professor at a state university somewhere who’s had really good experience working with local criminal justice coordinating counsel on perfect sources of data for how to separate out those who are mentally ill from those who are actually criminals. Or, maybe it’s somebody who runs a program in a prison that’s helped prisoners complete their GED and their college degree and get jobs when they get out of prison and reduce their risk of re-offense.

But how do you get at those ideas? We have to be open to, and willing to try, new ideas. How do I look across the great spectrum of talent available in this country and match people to opportunities?

What are the consequences if we do not become a smarter state?

We have to do more with less. We need to get better ideas to solve the problems that we face. We need to use technology to get at the ideas to help reduce the cost. We also need a diversity of approaches to managing a problem. If my goal is to reduce the rate of childhood illiteracy or decrease that word gap, the best ideas are not going to come from somebody within government. Without a diversity of approach, lots of people are going to suffer real consequences in terms of future problems, whether it’s literacy, hunger, security, or safety.

We can ill-afford to underserve people who need the services the government provides. What happens in communities where you have underperforming schools, for example, generation after generation? Consequences for whole swaths of the population, for lifetimes of people that affect whole families that lead to all kinds of complex repercussions, not just in terms of economic well-being, but increased crime, increased drugs, public health problems. All of this takes a toll on people’s lives.

It is incumbent upon us to make government work as well as it can to improve lives and improve our communities. If we fail to innovate how our government works, our government will be rendered illegitimate. That is something we cannot risk.

You know how I can do things with my smartphone with one click? It doesn’t make sense to have government agencies where I have to go to 20 different places and fill out 50 different forms to get a service that could be directly delivered to me with an app.