Imagine trying to create the computer of the future for Steven Spielberg. That was what John Underkoffler was up against when he was hired as the science and technology advisor for the tech thriller “Minority Report.”

“Spielberg wanted to see how computers would look in 50 years. Spielberg was very particular and I’m grateful for that. He said, ‘I hope we’re not going to still be using keyboards and a mouse.’ That was all the encouragement I needed. I took the ideas I’d been working on at MIT and refined them, simplified them, so that they’d be instantly recognizable on-screen,” Underkoffler said, recalling his work on “Minority Report,” which premiered in 2002. Underkoffler received his Ph.D. from the MIT Media Lab.

The result of his efforts with Spielberg was a computer designed with oversized holographic screens and operated by gestures.

“I designed all of those systems in a way naively. I designed them the only way I knew how, which was if they’d need to be built eventually in the real world. Since I wasn’t at the time a seasoned movie person, what came out the other side was something that felt very, very real. I spent days and days training the actors to use this new gestural command center,” he said. “What resulted just felt like an actual day in the life, a very exciting day in the life, but people using real technology. And I think what was significant ultimately about those scenes was not that we were suggesting there was any kind of artificial intelligence running the show. I was keenly interested in depicting that a radical user interface would let these women and men do something as humans that wouldn’t be capable without it.”

SEE: Beyond Minority Report: Why William Gibson’s Neuromancer points to the future of UI (TechRepublic)

The technology proved so popular with fans of “Minority Report” that it spurred Underkoffler to create the real thing. He began working on a prototype in 2004, and he founded his company, Oblong Industries, in 2006.

“I suppose it’s massively gratifying that this 14-year-old film continues to be a kind of touchpoint or bellwether what people expect the future to turn into, what they expect it to be like. For me, for us at Oblong, it was an incredibly important calling card in a way because I’d had a very unusual opportunity to visually prototype a bunch of ideas that I thought would be valuable in the real world. And hundreds of millions of people could see these ideas. The enthusiastic response that people had to next-gen UI computation was all the impetus I needed to get started, found Oblong, and put the ideas in practice in a real-world commercial setting.”

A real-world solution

The computer that Underkoffler created for the real world is called Mezzanine, and it is currently in its third major version with a new one in the works. It combines with an immersive collaboration solution with 360-degree screens. It works with hand gestures and screens to allow teams to work together from remote locations with multiple computer screens and be able to be broadcast at the same time.

Mezzanine offers a collaborative way for people to work, since a single computer is designed to serve one person. It provides multiple large screens, because Underkoffler believes the display space on a smartphone or tablet is too small. “It’s just not enough. You can’t see enough to get real work at any scale done. I would contend that modern day work has, depending on the task, an inherent scale, a natural scale.”

The technology has been popular, with IBM, Accenture, and Boeing among Oblong’s customers. NASA recently awarded Oblong a contract to install 10 Mezzanine systems across four NASA Aeronautics research centers in Virginia, Ohio, and California.

“IBM is a huge user. Accenture is a big user. They’ve been using Mezzanine internally with lots of units in a bunch of different countries. Late in 2015 they announced the Accenture-connected analytic experience, which basically wraps a bunch of analytics tools around the Mezzanine. A customer signs up for the connected analytics experience, and they get a Mezzanine room,” he explained.

Another client is Dentsu Aegis, a UK-based advertising agency, which uses Mezzanine in its New York, London, and Singapore offices. Employees work together remotely on advertising campaigns, he said.

“Not every use case inside a Mezzanine room is as completely urgent as the ‘Minority Room’ scenario, where they are literally going to die if they don’t figure the situation out in the next 6 minutes, but there are some scenarios being done in Mezzanine rooms that aren’t that far off,” he said.

The lure of Hollywood

After working on “Minority Report,” Underkoffler assisted with the tech seen in “Iron Man,” which came out in 2008, creating the visual images and interactive gestures that actor Robert Downey Jr. used to control the Iron Man suit.

“‘Iron Man’ was an elaboration of the gesture stuff. Whereas in ‘Minority Report’ we were showing how a gesture-controlled UI could aid collaboration, in ‘Iron Man’ it’s just him. He’s a technologist. He’s building it for one use, just for himself. It’s a different flavor, a different take on it. There was a version of the UI that appeared in his helmet as he’s flying around. That’s an interesting set of different design constraints. You don’t have a mouse or a keyboard, so how do you interact with the stuff that’s being projected into your visual field and that’s being projected on to the surface of your helmet.”

He said that Downey was so interested in learning about how his character Tony Stark, a computer scientist, should act, that he came to Oblong and spent a day hanging out to learn how they work together and behave, and even the types of things they have on their desks.

After skipping the second “Iron Man” movie, he worked some on “Iron Man 3.” He said, “I did a little quiet work on the third one because Robert Downey Jr. came by Oblong to see what we were doing and ask advice on a new interface he wanted to use to control the Iron Man suit that would depict the interface in a different way.” The result was the subtle gestures that Tony Stark uses to connect with the Iron Man suit as it flies in pieces from wherever it is being stored and assembles itself around him.

Underkoffler hasn’t worked on films since “Iron Man 3,” which came out in 2013, but he said he doesn’t miss Hollywood. “Building the stuff in the real world, getting it in the hands of customers, especially customers like NASA, is way more gratifying at the end of the day. You’re giving people something of value, you’re changing the way they work. You’re bringing the future of work to the present and bringing it to teams that desperately need it.”

The 3 top takeaways for TechRepublic readers

  1. Oblong’s Mezzanine computer is the real-life version of the immersive computer seen in “Minority Report,” which John Underkoffler created for the movie.
  2. Mezzanine provides an immersive digital experience on 360-degree screens.
  3. NASA has hired Oblong to install 10 Mezzanine systems in four locations around the US and other clients include IBM, Accenture and Boeing.