Developing methods of providing extra data about the content of a TV programme using Heads-Up Display, Augmented Reality-style applications
Project from 2012 - present
What we are doing
The Βι¶ΉΤΌΕΔ already uses Augmented Reality applications, such as the Piero Sports Graphics system, to provide enhanced coverage of sporting events. These are used to add tied-to-scene 2D and 3D graphics to broadcast content before it is delivered to the user. The allowed a user to choose to view sporting data and statistics in windows at the edges of the video. We are looking at combining these approaches: to draw data registered to objects in the scene directly into the video in the player, at the control of the user.
The application will be similar to iPlayer, with the addition of a suite of options about which data overlays the user would like to see. For example, in a football match we could use player tracking data to draw the name of each player above their heads. We could also highlight members of one team, or overlay a bigger ball, to help viewers with poor eyesight.
The player-tracking information, along with other relevant data about the scene (such as the position and orientation of the camera) is sent directly to the user, and a web-browser application on the user’s computer draws the graphics on top of the video. Because the graphic drawing is all done on the user’s machine, the user has much greater control about which data overlays he would like to see. In the football example, the user would have a choice of which, if any, of the player names to show, but we can also add data about how fast a player is moving, where the ball is, how often the player has possession of the ball, whether a player is offside and many other details that are useful for match analysis.
An alternative use-case would be to add enhanced information to science or nature programmes. For a programme such as Blue Planet, we could add several extra layers of data to the programme. One layer could present enhanced zoological information about the species being shown, whilst another layer adds text highlights to particularly interesting physiological features displayed by the animal. These might be of particular interest certain sections of the audience, while others could choose to turn these displays off.
How it works
In order to make this system a reality, we are building a production tool to allow the creation of 2D & 3D overlays, developing video analysis tools for generating the overlay data and writing a web-browser based video player that allows frame-accurate synchronisation between video and data. Current web-browser video players do not guarantee frame-accurate reporting of playback time, something which is vital for tied-to-scene graphics to look correct. To deal with this, we have created a WebGL shader-based video fingerprinting system that allows exact detection of a specific video frame, ensuring that the overlay graphics remain perfectly synchronised.
Project Team
-
Immersive and Interactive Content section
IIC section is a group of around 25 researchers, investigating ways of capturing and creating new kinds of audio-visual content, with a particular focus on immersion and interactivity.