Unearthed - Interactive 360Λ sound and video in a web browser
Posted by Chris Pike on , last updated
This week we launched a new pilot on ΒιΆΉΤΌΕΔ Taster called . It's an interactive and immersive story following a day in the life of a hummingbird, featuring binaural sound and 360Λ video. The project, run through , was a collaboration between the Natural History Unit, R&D's audio team and an agency called .
We worked with sound designer Richard de Mowbray to create an immersive headphone mix for Unearthed, using our binaural production system. If you've not heard of binaural sound before, . In short, it is a way of recording or processing sound to create immersive 3D scenes specifically for headphone listening.
Most of Unearthed uses normal 2D video, albeit with some nice interactive touches. For these scenes we created a binaural mix from stereo and mono recordings, much as we did for the radio drama production last year. We used various tools to construct a 3D soundscape from these conventional recordings, including 3D panning, upmixing, and 3D reverb. These tools are important when working on real productions, as you can't always capture the ideal source material for 3D audio; as much as I'd like to have taken a microphone rig like to the jungle.
For us the most exciting part of this project is the 360Λ interactive video scene, because it also has 360Λ interactive audio! Meaning that when you rotate the scene to look around, the sound rotates with it. This is when binaural sound technology really works well, when you can interact with the scene. In our lab we've been using head tracking with our binaural production system to do this for some time. We showed in an last year that this can lead to a highly plausible impression of sounds coming from outside of the headphones in the room around you.
For this reason, binaural sound will feature in virtual reality systems when they hit the market in the near future. We've already got some experience working with VR and immersive audio. We've been doing production experiments ever since we got our hands on an . Earlier this year I worked with to create a for her song , which was . The video is now available on , although unfortunately YouTube doesn't yet support 360Λ sound. So that's why we're so excited about Unearthed, as far as we know, it's the first time that interactive binaural sound has been presented with 360Λ video in a web browser without needing to install a plug-in.
It's all made possible through an emerging technology called Web Audio, which allows developers to efficiently process audio signals in a web application. The Web Audio API is rapidly moving towards standardisation, that is making this happen. Several major browsers already the draft specification and many already exist. This year also saw the first held in Paris, at which our team presented several papers and demos.
To produce the sound for this 360Λ scene, we used a novel tool we've developed that combines our real-time binaural rendering software with 360Λ video rendering for the Oculus DK2 headset. The sound source positions are displayed as overlays on the video, which is presented in sync to the audio from our workstation as we're working. This allows the engineer to be immersed in the scene whilst mixing it and to interactively explore it as they work.
The next challenge is to export the scene so that it can be rendered by the web player. We've helped to standardise a meta-data model for representing advanced audio content called the . This can be used to translate the experience between different systems, such as our native production renderer and the web player. We also applied some pre-processing to reduce the complexity of the scene for the web player, this is one aspect that we plan to research further.
Web Audio is exciting because it makes experiences such as interactive binaural sound accessible to a large number of people without them needing specialist equipment or software. It does not yet offer the same sound quality and efficiency as can be achieved with a native application, but as the standard develops we believe the gap can be narrowed. It's important for the ΒιΆΉΤΌΕΔ that new content experiences are available to as many people as possible. We will continue to work on the production and client-side technology for binaural sound and 360Λ video, and hope to bring you more immersive and interactive content experiences soon.
I'd like to thank everyone at the NHU, Realise, Connected Studio and R&D who worked on this project. Please do and give us some feedback. I hope you enjoy it!
P.S. Check out on the ΒιΆΉΤΌΕΔ News website.
-
More on Virtual Reality and 360 Video:
from day
This post is part of the Immersive and Interactive Content section