Fourteen years ago, Tom Cruise offered a glimpse of the gesture-based future of computing, sweeping his hands through the air as he navigated a dizzying amount of information in Spielberg’s Minority Report.

Today, the swiping and pinching seen in the film are the norm for controlling the mobile computers we carry in our pockets, and a multi-user version of the UI used by Cruise is available as Mezzanine, a high-end collaboration system from Los Angeles-based Oblong Industries.

But John Underkoffler, the creator of the Minority Report UI and Mezzanine, has a vision for the future of UI that is far bolder than his imaginings in the film, one that would see computer interfaces borrow from the abstract nature of art and music in an attempt to resonate more deeply with users.

Underkoffler is interested in how UI design could be informed by the ability of a piece of music or an expressionist painting to evoke a complex response. He sees great efficiency in how, with just a string a plaintive notes, music can trigger myriad emotions and thoughts in the listener. The affective nature of music and art might seem far removed from the pragmatic, functional world of user interfaces, but Underkoffler believes it deserves further study.

“I think people aren’t looking as broadly as they could be for inspiration,” he said, referencing recent trends in UI design, such as having virtual assistants, like Apple’s Siri and Microsoft’s Cortana, talk to you.

Rather than using spoken language, Underkoffler suggests machines could relay more complex information if they were to communicate using a more abstract medium, for example a language based on music.

“The thing with human language is it’s very expressive, but you’re only doing one thing at a time. Whereas humans are able to apprehend information in the audio domain that exists at different frequencies, which is why we can understand music, as it were.

“Could you make a musical language where there’s more than one idea happening in parallel? Where there are different frequencies with which you could express that?

“My guess is that humans would be capable of understanding it, it just has to be designed and it’d be really fun to prototype that again in a fictional situation.”

However, Underkoffler’s concern is that UIs could be headed in a very different direction, driven by the mass-market adoption of virtual reality headsets that some predict in the near future.

Virtual visions: Snow Crash vs Neuromancer

Science fiction has long forecast a future where virtual reality allows people to mingle in a 3D digital approximation of the real world, a concept popularised in the 1992 novel Snow Crash.

For Underkoffler, this pursuit of virtual verisimilitude would be a mistake, as it would end up restricting what’s possible in virtual spaces in the same way that having machines communicate using only human language is limiting.

“The drive to literalism is odd. So much effort is being put into stuff that to me doesn’t seem that important,” he said, referencing a drive in 3D computer graphics modelling to capture how light is refracted through translucent layers of skin when rendering a person’s face.

“Do we have sub-surface scattering in rendering so it really looks like a person on the other end? Why does it have to look like a person at all?”

Of the projections of future cyberscapes that VR might make possible, it is William Gibson’s vision of VR as a realm of abstract shapes, put forward in the 1984 novel Neuromancer, that Underkoffler finds far more compelling.

“The cyberspace representation is non-representational, it’s abstract. He talks about shifting planes of data and all these geometric shapes. For my money, those would end up being more powerful systems.

“We ought to let the real world be the place where photo-realism happens, because it happens here for free. I’m really curious why no-one is building VR and AR stuff that’s less than purely representational.”

Although it differs from the abstract type of UI Underkoffler is referring to, perhaps the most ambitious and novel UI design is coming out of the MIT Media Lab in the US, where Underkoffler previously worked as a researcher.

The labs’ Tangible Media Group has devised various ways to give physical form to digital information, from glass bottles that play music when their stopper is pulled out to flexible materials that can contort themselves into different shapes to represent different information to a sand pile that allows you to sculpt digital landscapes.

A lesson from art history

Underkoffler suggested that the design of user interfaces and virtual spaces may evolve in a similar manner to the development of painting styles, from crude prehistoric art, to the pursuit of realism, through to conceptual and abstract work.

“Maybe what’s happening here is that these technology projectors are recapitulating what’s happened in the visual arts.

“In the very beginning, let’s say for the first 800 years, painting drove ever closer to photorealism. You had folks like Caravaggio, who became obsessed with how light works and understanding how to depict stuff that looks more and more real.

“Then photography comes along and puts the painters out of business, in a certain moral sense. Finally, painters are freed from having to worry about whether it seems photo-real or not and then you get really exciting things happen.

“I guess I’m just a horribly impatient guy, because I would like to skip the part where we try to make all the computer-generated stuff look as real as possible and go straight to making it as interesting as possible.”