Audio Augmented Reality (AAR) technologies are now coming to market, and this technology presents . Collaborative or multiplayer experiences in AAR are potentially an exciting opportunity as they allow users to interact in real-time and space with each other, without the barrier of a screen. However, opportunities for multiple participants nevertheless remain under-researched. As part of their PhD programme in Media and Arts Technology at Queen Mary University London, Anna Nolda Nagele and Valentin Bauer undertook a placement with ΒιΆΉΤΌΕΔ Research & Development. In the first of two posts, Anna writes about how the pair developed a four-player performance-based experience in AAR: "Please Confirm You Are Not A Robot".
The game is set in a not-so-distant future, where participants are invited to deprogram themselves from their digital rituals and practice real human interactions, to listen to each other and the world around them. A voice assistant called "Pi" guides the group through four exercises, which gradually increase the level of human interaction. As the story goes on, Pi displays more and more human features and discovers the players' feelings and fears. The experience ends by Pi trying to find their voice through one of the participants, prompting the participants to tell the others "Enjoy reality as long as it lasts" and to free them by taking each player's glasses off their heads.
Players should not look at their phones during the game as instructions and information are delivered as sound. This allows players to move around as part of the game.
We used as they leave the ears completely free and allow the user to stay connected to their real auditory surroundings and are an interesting technology to design a collaborative experience around.
We tested this experience in a user experience study and came up with some design recommendations for developing collaborative AAR experiences.
Collaborative storytelling is used as a tool in theatre and performance-making to communicate different layers of a story. Participants become the actors as well as the audience for each other. There is room for them to collaborate and communicate - each bringing their ideas and experiences to the story - and to create a unique, communal experience.
- ΒιΆΉΤΌΕΔ R&D - Recommendations for Designing Audio Augmented Reality Experiences
- ΒιΆΉΤΌΕΔ R&D - Designing and Developing Ideas for Audio AR Sunglasses
The use of digital media, especially handheld devices and their applications, affect mental and physical well-being. Contemporary technologies fulfil human needs such as competence, control, social connection and emotional desires. However, they can also bring negative consequences with their use such as distraction, behavioural addiction, or a loss of control.
This experience draws attention to how digital devices and services impact people's lives. In a series of games, choreographies and exercises, participants are prompted to reenact, observe and subvert their digital practices.
Sound Design
The location-based AAR prototype '' showed that 3D audio environments highly contribute to the feeling of immersion for players in comparison to audio-only games in stereo audio.
Other early designs of interactive audio-only experiences highlight the importance of sound design for a virtual world with no visual component. It is essential to put sounds in the context of other sounds or narration to make them meaningful and to establish a cause and effect relationship between the actions and sounds. Without visual cues, a dense soundtrack is difficult to interpret. This effect may increase with perfect auditory transparency as augmented sounds and real-life sounds have to be understood. Our sound design leaves enough room for real and virtual sounds to exist and complement each other. We supplement the silence with a minimal, glassy ambient soundtrack which feels as if it is emerging from the space around the player.
The best way to tell a story in AAR is to combine the narration with well-designed ambient cues. Consequently, our narrator Pi guides each participant and introduces sounds, activities and conversations. The AAR game might be played in noisy environments, and so sounds have to be exaggerated, alongside clear narration.
Audio-only sound design has to be created for the player's imagination and their level of understanding. Every player will interpret the sounds differently according to their expectations and previous experiences in life. Consequently, the experience will not be the same for each participant, but there is room for interpretation and to allow the participants to behave however they wish.
- ΒιΆΉΤΌΕΔ R&D - Audio AR: Geolocated Sound
- ΒιΆΉΤΌΕΔ R&D - Virtual Reality Sound in The Turning Forest
Design Process
The design and development process started with a range of mini-games, all considering different aspects of digital well-being, 3D audio, and what was possible with the Bose Frames. Those activities were scripted in a linear way, including the narrator and sound design elements.
To conduct some initial user testing for the ideas, we developed a set of Wizard-of-Oz prototypes. The narration was recorded using . We arranged the recordings on a linear timeline (one for each player if asymmetric information was required). This meant the sound could be played back in stereo audio to four players at a time, testing the ideas and game mechanics and also allowed us to adapt the script and the experience quickly.
We narrowed a wide range of prototype games down to four, taking into account how well they worked with the Frames and their potential to generate interesting insights during the empirical study. This way of prototyping allowed us to test the game mechanics between participants and their interaction with each other. However, it possibly also led to a less well-developed spatial sound design in the final prototype as we considered spatial sound cues but could not test them in the early prototypes.
Once we decided on the mini-games, the narrative was expanded to connect four games into one storyline. The narrator Pi became a character, explaining the world in which the story was set, and outlining each scene and the interactions. Users would receive information and feedback about when they score points, and after each game would find out how far they are from completing the experience.
When testing the interactions, we found that the gesture controls β nodding and head-shakes β felt particularly unnatural to users and were also badly recognised by the technology. Ultimately, we replaced all but one of the interactions with a simple tap of the arms of the Frames.
Up until the last version of the prototype before the study, the narrator's voice was synthesised using Polly, which lacked excitement and sometimes clear pronunciation. From the tests, users told us that a more human voice would be better and create more empathy for the character Pi. For the final prototype, Catherine Robinson, a ΒιΆΉΤΌΕΔ radio producer, recorded the voice of Pi and provided us with a sound design for the character's virtual embodiment.
This process of iterative testing and adjusting based on feedback allowed us to improve the narrative and interactions in the final prototype incrementally.
Evaluation of the Experience
We tested the final prototype with five groups of four in a user experience study. Everyone was excited to try the new technology, and they found the experience thought-provoking.
Users have differing experiences with spatial sound, so one aim of the experience was to raise awareness and introduce listeners to 3D audio. It was complicated and frustrating for people to localise sounds except for users who had experience using the Frames. Introductory training to spatial listening before starting the game and a clear distinction between feedback sounds and spatial sounds would be useful and could enhance the effectiveness of this game.
Participants were most excited about the introduction sounds and where spatial audio cues were strong and felt integrated with the environment. Players described the social interactions as the best parts of the experience and saw significant potential in multiplayer AAR to break down social norms and barriers. It was enjoyable for most to interact with other people physically.
- In our second post we compile a set of design recommendations for people designing augmented reality experiences on smart headphones.
- -
- ΒιΆΉΤΌΕΔ R&D - Recommendations for Designing Audio Augmented Reality Experiences
- ΒιΆΉΤΌΕΔ R&D - Designing and Developing Ideas for Audio AR Sunglasses
- ΒιΆΉΤΌΕΔ R&D - On Our Radar: Audio AR
- ΒιΆΉΤΌΕΔ R&D - Audio AR: Geolocated Sound
- ΒιΆΉΤΌΕΔ R&D - Audio AR: Sound Walk Research - The Missing Voice
- ΒιΆΉΤΌΕΔ R&D - Virtual Reality Sound in The Turning Forest
- ΒιΆΉΤΌΕΔ R&D - Binaural Sound
- ΒιΆΉΤΌΕΔ R&D - Spatial Audio
- ΒιΆΉΤΌΕΔ Academy - Spatial Audio: Where Do I Start?
- ΒιΆΉΤΌΕΔ R&D - What Do Young People Want From a Radio Player?
- ΒιΆΉΤΌΕΔ R&D - Prototyping, Hacking and Evaluating New Radio Experiences
- ΒιΆΉΤΌΕΔ R&D - Better Radio Experiences: Generating Unexpected Ideas by Bringing User Research to Life
-
Internet Research and Future Services section
The Internet Research and Future Services section is an interdisciplinary team of researchers, technologists, designers, and data scientists who carry out original research to solve problems for the ΒιΆΉΤΌΕΔ. Our work focuses on the intersection of audience needs and public service values, with digital media and machine learning. We develop research insights, prototypes and systems using experimental approaches and emerging technologies.