麻豆约拍

Jim Simmons from D+E Platform Media Services, discusses our experience of delivering live Ultra High Definition at scale for the World Cup and Wimbledon on a round-by-round basis. 

Qualification

We started discussing the possibility of bringing the World Cup and Wimbledon to the audience in UHD towards the end of 2017. A cross-divisional team was set up with members from 麻豆约拍 Research & Development, Design & Engineering Platform, and Online Technology Group (OTG) who manage internet distribution, as well as the engineering and support teams in Broadcasting House and Salford, Sport Outside Broadcast (OB) who work closely with our commercial OB facility providers, our TV Platform team who test all the devices that use iPlayer, not to mention the iPlayer and Sport teams themselves.

If you want lots more technical detail of our set up, please check by our amazing developer, Lloyd Wallis. 

In short, the outside broadcast team encodes the UHD at a very high bit rate on site using a contribution encoder and gets it to Broadcasting House in London via IP where it goes in to the video router. This feeds the UHD distribution encoders which generate four bit rates of HEVC video up to 36 Mb/s (iPlayer HD is 5Mb/s for comparison). This is then packaged using the DVB-DASH standard so that 4 second long chunks of audio and video can be requested over the internet via a playlist called a manifest. This is served via an origin server in “the cloud”, which is then cached by servers in a 麻豆约拍 data centre to protect the origin. There is a further level of distribution provided by CDNs (content delivery networks) whose job is to serve very large amounts of traffic to very many of people across many ISPs who deliver the chunks to the connected TV. Easy.

We began building this chain in the spring but there were many unknowns. We needed to test contribution and distribution encoders, work out the best bit rates, try and bring all the networking together, organise the operational and support infrastructure - who pressed what button when etc. make sure we had levels of redundancy, test lots of different TVs and devices, ensure there was sufficient internet capacity. There was a long to do list. While we were building, we were also preparing our core services on iPlayer and iPlayer Radio for what we were expecting to be our biggest ever live streaming events.

There weren’t many opportunities to test our workflow. There are not that many UHD capable OB vans in the UK and they are often busy providing sports coverage for BT Sport and Sky so we only had a couple of events to test. We didn’t publicise these because we had no idea if they’d work at all.

We covered the FA Cup game between Southampton and West Brom (which I’ve now seen about fifty times), the Rugby League and some of the FA Cup final. By the end of this testing, we had a good feel for the workflow but we hadn’t put any significant load through the systems. We had only had a couple of thousand viewers.

We did discover that the bit rate for UHD averaged out at around 25Mb/s per viewer. We added this in to the total bandwidth predictions for the World Cup across the 麻豆约拍 to see how many concurrent users we could comfortably support and found that we would need to limit access to the trial. The standard HD and mobile streams were our first priority so we needed to ensure the trial didn’t take bandwidth needed for normal iPlayer and 麻豆约拍 Sport viewing.

We decided we’d be comfortable with a maximum number of around 60,000 concurrent UHD viewers, but that the number would be continually re-assessed during the tournament depending on overall performance. We also estimated there were maybe 2 million devices that might be capable of playing the UHD streams, hence the warnings about the streams being available on a first come, first served basis.

The Group Stages

The World Cup kicked off for us on the Friday 15th June as Egypt took on Uruguay. Trivia fans will want to know that the first ever UHD World Cup goal on the 麻豆约拍 was scored by Gimenez for Uruguay in the 89th minute.

The first couple of games ran relatively smoothly from a production point of view. We were closely monitoring feedback from audiences, via a survey and also through social media. The audience response was generally very positive, especially with regard to the picture quality. There were some comments about stuttering and juddering and speech getting out of sync despite viewers having high speed internet.

In the France v Australia game on the Saturday, some video didn’t arrive in time to be served to the audience. Unfortunately, the audio did arrive as DASH separates the two. This led to the audio becoming one segment out of sync - about four seconds. We failed over to the back-up. If you left the stream and came back, the issue was fixed, but if you stayed on the stream it remained faulty.

In the later game between Germany and Mexico, we had a router switch fail which caused more issues.

The quality of service that UHD demands of network infrastructure quickly roots out any problems, or components that are introducing latency. We replaced some optical connectors and noticed significant improvements in our logs and a reduction in complaints.

While we were fixing switches and networks, we took some other precautionary measures. We removed the lowest bit rate to give the distribution encoders less data to generate in each time period and we increased the wait time on the packagers. This increased our latency by a further 10 seconds. Not ideal but we wanted to keep the audience experience as smooth as possible.

We’d also noticed a number of comments on the size of the “麻豆约拍 UHD” logo in the top right hand corner of the screen. This was set to a DPP (Digital Production Partnership) guideline but we all agreed it did look a bit big, so we reduced the size by 10 percent. We changed that on the Tuesday of the first week.

On the day of the first England game, 18th June, one of our contribution encoders had an issue during the preceding Belgium vs Panama match so we removed it from the chain and switched to the back-up. We used that for the rest of the tournament.

The England vs Tunisia game proved to be our most streamed live event up to that date. We stopped access to UHD in the first half for a short period of time as we re-balanced traffic across our CDNs to optimise performance. Things calmed down a bit and we re-enabled UHD access at half time. We had just over 35,000 UHD viewers at this point but were shipping around 5Tbits of data per second across all services.

We continued to work on our distribution and found and fixed further issues with networks and infrastructure along the distribution chain during the next few matches as we discovered them.

As the technical situation settled down, we felt confident enough to let the presentation team on 麻豆约拍 One mention the fact that there was UHD available. This was done by the commentators and an on-screen graphic. Doing so resulted in an immediate spike of 10,000 new viewers within a few seconds.

Two days later (27th of June) we had a serious external networking incident that caused a problems for viewers such as buffering and streams failing. We let the stream run, but there were only a few thousand watching. Things recovered for the second half of the South Korea vs Germany game. We asked Sport not to mention us for a little while just in case.

Round of 16/Wimbledon

As we moved in to the round of 16 in the World Cup, we added Centre Court coverage from Wimbledon. This added a bit of complication as we didn’t have enough UHD encoders to provide 2x resilience for both events. This meant a degree of source swapping for the back-ups.

As the network issues had now been resolved, we put back the lowest bit rate and reduced our packager waiting time by 20 seconds. This got our latency down to around 45 seconds - but it was very dependent on what device you were using.

Audiences started to increase, both for UHD and in general. For the Belgium vs. Japan game, we peaked at 37,300 concurrent UHD viewers, including football and Wimbledon.

Quarter Finals

Things started to hot up now. For the quarter final between Brazil and Belgium on Friday 6th July, we had 41,600 concurrent UHD viewers at the peak (including Wimbledon viewers). We still hadn’t hit our theoretical maximum limit by full time though.

England vs. Sweden was on Saturday 7th June with a 3pm kick off. We hit our UHD cap of just over 60,000 concurrent viewers half way through the first half. We pretty much stayed at this level for the rest of the match.

Semi Finals

We ran UHD for the France vs. Belgium semi final with no issues and just over 48,000 concurrent UHD viewers. The men’s semi-finals at Wimbledon provided a good soak test for our encoders and staff with play continuing until just after 11pm.

The Finals

The tennis final on Saturday 14th, and tennis and football finals on Sunday 15th July ran smoothly. A peak of 44,300 viewers saw Novak Djokovic and France win their respective tournaments, including the first penalty awarded with the help of VAR in a World Cup final. We don’t know if the VAR was in UHD or not.

We stopped the last stream around 7pm and put live UHD to bed for now.

Extra Time

As well as the technical issues we addressed at the time, there were a couple of other common comments that I’d like to address now.

Latency

The elephant in the room for live internet streaming is latency. Well, it’s in the room if you’re watching on Freeview. If you’re online it’ll be a minute or two yet.

Latency is less of an issue for everyday viewing, but it comes in to sharp focus when we look at sport. The usual latency for online streaming is between 30 and 90 seconds depending on your viewing device. For UHD we got latency down to between 45 seconds and a few minutes but it was very variable, again depending on the device. Interestingly, when we asked viewers about latency in our survey, while they wanted it to be as low as possible, most said they wouldn’t trade it off against picture quality.

The problem is one of scale versus speed. It is possible to stream video very quickly over the internet. It’s used all the time for video calling. The problems with low latency streaming are that it generally only supports small numbers of viewers, or it requires very proprietary hardware and software, or there is a significant reduction in quality or resilience. As more and more audiences move online this is becoming a significant issue for the whole industry.

Fortunately, it is also an issue the industry is beginning to address, both through developments by hardware and software vendors, and through the work of standards groups which the 麻豆约拍 is contributing to. Ultimately, we need to get the latency of IP delivery comparable to that of linear distribution and we, in 麻豆约拍 Platform and OTG have already started working through those issues with our partners in the industry.

It’s too dark

A couple of themes that came out of the user feedback were around the picture being too dark or the colours not being vibrant enough. Having worked with the production teams we think we have managed to improve some of the issues we found e.g. ensuring graphics are inserted at the correct level. For a more detailed assessment of some of the possible causes at the home end then please see this blog written by my R&D colleague Andrew Cotton.

It’s Coming 麻豆约拍

The UHD trial was a brilliant learning experience for us so we definitely consider it a success. It’s clear there’s lots more to learn, work on and improve, and massive events like the World Cup and Wimbledon really put our systems through their paces. We hope those that watched it had a good experience, and sincerest apologies for when it didn’t go so well. Our TV in the office buffered just as France scored in the semi-final and I was livid so I do understand how it can be frustrating at times. Thank you for trying it if you did.

We built the trial in such a way that it can be re-used and improved upon. This will allow the next trial, whatever it may be, to take advantage of all our experiences and fixes we’ve implemented this time around. Live streaming Ultra HD over the internet is at a very early stage, but we’ve taken an important step with this trial. We will continue to build our capability and make sure future audiences have a free-to-air option for live Ultra HD and HDR.