Â鶹ԼÅÄ

So we were happy to help out when Cathy Robinson and the Digital Marketing team at Â鶹ԼÅÄ Wales got in touch and suggested making an immersive version of a Six Nations match. This felt particularly fitting as fans are not allowed into the stadiums for the tournament - so anything we can do to try to give people some of the sense of immersion of being part of a crowd and right next to the action is a step in the right direction. Also, as an orchestrated audio experience with accompanying video, this was a nice use case for driving our software framework's development.

The experience

We decided to keep things relatively simple for our first attempt at orchestrated audio for sport. We designed an experience based on the Scrum V highlights package of the Scotland v Wales Six Nations match on Saturday 13th February 2021 (no spoilers, but it was a close match with loads of action).

On the main device, the listener can watch the video and hear the crowd noise and commentary (which can be toggled on and off). When extra devices are connected, the listener can hear additional crowd noise to give an immersive experience and can also choose to add sound effects track consisting of the referee's microphone and pitch sounds (e.g. ball kicks and players shouting).

Screenshots of the experience showing the connection screen on a laptop with two phones connected, playing additional crowd and referee noise.

Our previous audio orchestrator trials:

Â鶹ԼÅÄ R&D - The Vostok-K Incident / Try it on Â鶹ԼÅÄ Taster

Â鶹ԼÅÄ R&D - Decameron Nights / Try it on Â鶹ԼÅÄ Taster

Â鶹ԼÅÄ R&D - Monster / Try it on Â鶹ԼÅÄ Taster

Â鶹ԼÅÄ R&D - Pick A Part  / Try it on Â鶹ԼÅÄ Taster

Technical details

There were a few technical challenges in pulling this project together. First, we needed to create the content. Being able to send different parts of the audio to different connected devices relies on having access to the separate audio tracks, not just the complete mix. We got hold of the highlights video with the commentary separated from the crowd noise but only had access to the referee and sound effects as a separate recording from the whole match - so it was necessary to find and extract the highlight clips. This was achieved with a process that was part-manual, part-automated. The approximate time of the highlight in the full match was found by hand, then a signal processing technique called cross-correlation (which works by determining the similarity between two audio signals) was used to find the exact clip.

Digital audio workstation session with audio tracks and start points for each highlight clip.

With the content in place, we needed to expand our orchestration framework to replay video. To do this, we modified the Â鶹ԼÅÄ Standard Media Player plugin that was used to drive the synchronisation in Â鶹ԼÅÄ Together (our experimental tool for remote shared viewing) so that we could hook it up to the audio device orchestration template. We also made some changes to the user experience, for example, providing a prompt to let viewers know that they can connect extra devices, a button to show the connection instructions, and a switch for turning the commentary on and off.

Standard media player with synchronisation plugin and user interface components for connecting extra devices and switching audio tracks on and off.

Next steps

We're interested to see how this trial of orchestrated audio for sports content is received. If you get a chance to try it out, please take a minute to complete the short Â鶹ԼÅÄ Taster survey to let us know about your experience.

We're only just scratching the surface of what we could possibly do with sports audio and device orchestration. We're using quite a simple sound scene for this project - crowd noise, commentary, and sound effects/referee microphone. But with more access to different microphone feeds, there would be increased possibilities for creating really immersive experiences. Of course, this will be easier to do once stadiums are full of fans again.

It would also be great to try out different sports. How about Formula 1, where you could listen to a team's radio communications on your mobile? Or a tennis match where you can really hear the ball as it bounces on each side of the court.

One of the joys of watching sport is that it's happening live, and the outcome is unknown - so trialling this technology with a live match rather than highlights would be another good step. There will be more technical challenges to overcome for synchronised multi-device playback of live content, so testing out the user experience with 'on-demand' playback is a good start.

-

Â鶹ԼÅÄ MakerBox - Audio Orchestrator

Â鶹ԼÅÄ R&D - Vostok K Incident - Immersive Spatial Sound Using Personal Audio Devices

Â鶹ԼÅÄ R&D - Vostok-K Incident: Immersive Audio Drama on Personal Devices

Â鶹ԼÅÄ R&D - Evaluation of an immersive audio experience

Â鶹ԼÅÄ R&D - Exploring audio device orchestration with audio professionals

Â鶹ԼÅÄ R&D - Framework for web delivery of immersive audio experiences using device orchestration

Â鶹ԼÅÄ R&D - The Mermaid's Tears

Â鶹ԼÅÄ R&D - Talking with Machines

Â鶹ԼÅÄ R&D - Responsive Radio