On Monday, OpenAI, a nonprofit AI research organization backed by Elon Musk, announced Universe, a software platform for “benchmarking the general intelligence of AI agents across the world’s supply of games, websites and other applications,” the organization said.

One of the main technical challenges facing AI research is that “AI agents require access to a large variety of realistic environments where they can learn progressively more complex skills, and where we can accurately benchmark their progress,” according to an OpenAI blog post. Currently, it said that the biggest and most popular assemblage of “reinforcement learning environments” is the Arcade Learning Environment, which is a collection of 55 ATARI games. “Over the last few years, this benchmark has led to significant progress in developing more effective and robust reinforcement learning algorithms that allow agents to learn from experience,” OpenAI said.

With Universe, OpenAI said it wants to “measure and accelerate this progress.” Right out of the box, according to OpenAI, the software contains thousands of games (such as several Flash-based ones and popular games like StarCraft), along with form-filling browser-based tasks and applications like fold.it.

“Our eventual goal is to develop a single AI agent that can flexibly apply its past experience on Universe environments to quickly master new ones, which would be a major step towards general intelligence,” OpenAI said. The organization noted that Universe is already available for research, and it is requesting assistance to scale the “environment ecosystem.”

OpenAI added that there are a number of ways other businesses and organizations can help get Universe off the ground. Among them, it is seeking companies that are willing to grant it permission to use their games, programs, websites or apps. “If your program would yield good training tasks for an AI then we’d love your permission to include it in Universe,” it said in the post. OpenAI also said that the “ideal environment candidates have an on-screen number which can be parsed as a reward, such as a computer game score, or a clear objective we can detect with a program and then send to an agent.”

Measuring progress

The San Francisco-based nonprofit, backed by Silicon Valley heavyweights including Musk, Peter Thiel and Reid Hoffman, announced its launch in December 2015, with the goal to “advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.”

Since then, OpenAl has made significant alliances. Last month, for instance, the nonprofit partnered with Microsoft and said it would make Microsoft Azure its primary cloud platform. As previously reported by TechRepublic, the chief reason OpenAI linked up with Microsoft was due to its “focus on deep learning, open source technologies, and the capabilities available in tools such as Azure Batch, Azure Machine Learning, and the Microsoft Cognitive Toolkit.”

With the announcement of Universe, OpenAI said the AI community can only be sure of progresses made by measuring performance. When it comes to developing AI agents that display general intelligence, OpenAI said its “goal is to build a single agent that can rapidly learn to perform well on a large variety of tasks.” Universe will allow the organization to measure this type of performance–referred to as an “AI Score”–and to make real progress towards such a goal.

OpenAI added that this task “may seem daunting right now,” but the project is drawing “inspiration from the historical precedent set by the ImageNet dataset in the Computer Vision community.”

In keeping with its open-source ethos, OpenAI also said it is “actively seeking strong researchers, engineers, and anyone in between” to assist with Universe.

AI experts widely praised the move. Fabio Cardenas, CEO of Sundown AI, called it a “clever idea.” “They are providing training sets for new AI platforms and benchmarking results,” he said. “These are the sorts of efforts that will generate smarter systems over time.”

And the strategy has been proven effective already. “The Atari game suite has been a valuable benchmarking tool for AI researchers,” said Marie desJardins, professor of computer science at the University of Maryland, Baltimore County, and former chair of AAAI (the National Conference of the Association for the Advancement of Artificial Intelligence). “The planned release of Universe, with a much broader and deeper collection of games and other application domains, will provide rich opportunities for researchers to evaluate and compare their systems in a wide variety of domains,” desJardins said.

Toby Walsh, professor of AI and the University of New South Wales also said “the goal of developing an imageNet for RL is a great one,” and that OpenAI should be “congratulated.”

Still, Walsh believes this move is only “a partial step” towards that goal, and “an even smaller step to general AI.”

“There’s much more to AI than playing artificial games,” said Walsh. “There’s little scope here for learning from limited data–games have proved popular as you can generate essentially unlimited amounts of training data and deep learning needs this.”

But real humans, said Walsh, are forced to learn from more limited data. “We can’t afford to make many errors,” he said. “There are many other issues that this data set will not greatly help–commonsense reasoning, planning, natural language processing. And then there’s the whole topic of unsupervised learning.”

“Many of the goals we need to address to get to general AI will not be solved,” said Walsh, “even with this software platform.”

Also see…