Â鶹ԼÅÄ

When discussing or reading about artificial intelligence, I get the feeling that AI is simultaneously being over- and underestimated. On the one hand, recent projects like or prompt some people to proclaim the imminent advent of the Singularity, a sentient AI that surpasses human intelligence. On the other hand, failures of AI systems that range from to reassure sceptics that these technologies are not yet fit for purpose. As so often is the case, the truth is somewhere in-between; AI is already used successfully for a large variety of tasks across businesses, but it also requires human guidance to function properly.

Recommender systems for online streaming platforms are a perfect example of this. If you are unfamiliar with recommenders, you can imagine them as a system that reorders a list of items (for example movies) according to one person’s preferences. Usually that list would not consist of all possible items but would be a subset selected by a human. At the Â鶹ԼÅÄ, iPlayer Curation, a team within the editorial wing, puts together content following the Â鶹ԼÅÄ mission to entertain, inform and educate. But modern audiences that are used to streaming platforms like Netflix do not only expect content to be high quality, but also relevant to their personal taste. This is where things can get messy.

During this research project, I conducted eight interviews with members of iPlayer Curation, iRex, the team responsible for iPlayer recommendations in production, and individuals from R&D Automation who develop new AI systems at the Â鶹ԼÅÄ. I found that while everyone agrees that recommenders are important, they are understood in very different ways. Editorial stakeholders must juggle a variety of objectives in their work, for example ensuring that iPlayer content is diverse and universal to align with Â鶹ԼÅÄ values, whilst promoting shows to specific audiences. A lot of thought goes into content curation, but it can be difficult to anticipate what audiences will end up seeing because recommenders, at least in part, reorganise the content.

Curators have a small number of test accounts that they use to regularly check iPlayer. Sometimes they spot recommendations that do not ‘look quite right’, which is communicated to iRex and sometimes R&D, especially when a show does not perform as well as expected. However, data scientists think about recommenders as a numbers game: if, overall, recommenders make the audience’s experience better, it does not matter if some individual recommendations appear to be ‘off’. What ‘better’ means must be defined by a quantitative (measurable in numbers) metric. The go-to metric is consumption, meaning how much content is watched, which has been shown to increase when recommenders are in use. On the one side, this can lead to frustrated curators who want to promote a new show, but the recommender instead pushes older popular content to increase consumption. On the other side, it can lead to frustrated data scientists who want curators to tell them in quantifiable terms what to optimise the recommenders for.

This issue cannot be solved in the three months that were planned for this project, but I want to provide a starting point. From my research it became clear that better communication is key to enabling curators and data scientists to work towards the same goals. But better communication does not mean having more meetings or presentations. Relevant teams already speak to one another, but the conversation can easily get lost in translation. Together with R&D Automation we designed an interactive tool that visualises how data scientists view recommenders that other stakeholders can explore. This tool can help stakeholders identify a specific, tangible issue. For example, the tool clearly visualises the trade-offs in recommender systems that data scientists continuously talk about, where improving recommendations for one group of users might reduce the performance in another group.

The interface of the interactive tool. This is a prototype that focuses on recommendations in the New & Trending rail on the iPlayer homepage. It is meant as a starting point for further development.

All too often, discussions both between and within teams lead nowhere because problems are not formulated in a concrete way that ensures everyone is on the same page. This tool will not ‘solve’ communication, but it is an approach that takes the discussion from an abstract space to something tangible, which might prove essential to effectively tackle the challenges surrounding personalised iPlayer content.