The team from Bjork's immersive MoMA installation talks about authoring reality at the Augmented World Expo in Santa Clara, California.
How do you split Björk in half and make her ooze electric blue lava?
This was one of many odd challenges to overcome in creating a multimedia immersive mid-career retrospective for Icelandic artist Björk, commissioned by the MoMA in New York City.
Various people involved in the making of this installation spoke at the Augmented World Expo at the Santa Clara Convention Center in Santa Clara, California on Tuesday, June 9.
The panel, titled Authoring Reality, featured Brian Pene, director of Emerging Technology at Autodesk; Andrew Melchior, chief architect at The Third Space Agency; Chris Pike, senior scientist for audio at the BBC; Tish Shute, director of product experience at THRED; Eric Hanson, partner at xRez Studio; and Varun Nair, co-founder of Two Big Ears.
Shooting the experience included an impressive mix of data capturing places like Red Rock Canyon in California and lava tubes in Iceland; designing and fabricating props and set pieces, and creating visual effects to do things like spread that blue lava across actual landscapes — and Bjork — or to have her floating in an environment with bits of fabric from her dress flying off her.
The Black Lake video was shown on two projectors in a cinema room lined with felt cones, designed and sculpted based on a spectral analysis of her music.
What all this amounts to, Third Space's Melchior, said, is the idea of creating a hybrid space — an immersive, interactive area that could blend the auditory, visual, and physical in real time. For example, sound could move around the walls of the cinema as the video moved screens as well. And what's more, as Pene said, is that it allows for new opportunities for not only storytelling, but engagement with audiences.
Melchior also talked about Stonemilker, a 360 video now available through Google Cardboard, which features Bjork on a stone beach singing the song of the same name from her recent album, Vulnicura. You can also view it on YouTube with omnidirectional scrolling.
For the exhibit, the team made their own head mounted display. One advantage of that over the Cardboard experience, Nair said, was the 3D audio. He said that though there are many great VR experiences, for the most part, the audio is still stuck in 2D.
Pike talked more about the challenges of doing audio for the VR experience. At one point in the video, there are multiple Bjorks — what does that sound like in VR? They opted to bring the audio to the forefront, go mono, and give the impression she's "in your head," he said. Also, by placing in different areas, they could nudge the viewer to turn and look in different directions.
Audio can often be the subtle marker of effectiveness in immersion. For musicians and engineers, it adds an extra layer to their workflow and processes.
The installation closed at the MoMA, but will be traveling around the world, Melchior said.