Picture Quality on Βι¶ΉΤΌΕΔ HD: a Viewers' Group Visit (part 2)
Blog 2: The Research and Development Visit
Editor's note: This is the second post of three that have been written by Paul Eaton and a group of visitors to the Βι¶ΉΤΌΕΔ regarding picture quality on Βι¶ΉΤΌΕΔ HD. The first part of the post is here. (PM)
"Present were: Andy Quested; Matthew Postgate (Controller, Research & Development); Phill Layton (leads the work for the Βι¶ΉΤΌΕΔ digital transmission chain) and Andrew Cotton (a lead technologist). As mentioned yesterday, a key part of our visit was the time we spent at R&D, and so we wanted to devote a whole Blog to what we did, saw and learned there.
ITU Tests
Our time in R&D began with a set of subjective tests comparing various clips of HD material.
The complete test was based on ITU-R Recommendation BT.500 (Methodology for the subjective assessment of the quality of television pictures), double-stimulus continuous quality-scale (DSCQS).
The method requires the assessment of two versions of each test clip. One of each pair of test clips is unimpaired while the other presentation may or may not contain impairment. The unimpaired picture is included to serve as a reference, but the observers are not told which the reference picture is. In the series of tests, the position of the reference picture is changed in pseudo-random fashion. The observers are simply asked to assess the overall PQ of each presentation by inserting a mark on a vertical scale. See ITU BT.500: .
In our case, we discovered the actual content of the test after it had taken place. We were viewing three versions of each test clip. The 'unimpaired' original source, a version played through the old encoder and bit-rate set-up, and a version played through the current set-up. In each of some 29 tests we were comparing the source against one or the other encoder set-up, or in some cases against itself. This whole procedure was preceded by a rudimentary eye-test using a Snellen chart in which we were simply asked to record the smallest line of text we could read on our test sheets as an indication of our eyesight. The tests were recorded anonymously.
The clips were viewed on calibrated 42" Full HD Plasma displays. The tests were conducted in a darkened, windowless room. The actual test set-up is shown in the picture (and schematic) below:
The test clips were part of the 'SVT High Definition Multi Format Test Set' (taken from the programme Fairytale, made by Swedish Television (SVT)) that are routinely used for this testing. The clips can be downloaded from the .
In addition, a short broadcast on Βι¶ΉΤΌΕΔ HD, which we had identified as suffering from PQ issues, was also used.
The test clips that were used are listed in the table below. The 'coding difficulty' as according to the ITU test requirements is also shown, since the tests are meant to use a range of material on this basis. The first three are EBU assessments.
The test clips were preceded by some other clips to 'acclimatise' us for the testing process and included examples of very high quality - a clip of source material from the Children in Need concert at the Royal Albert Hall - as well as of very poor quality - a clip from Bleak House demonstrating inappropriate use of progressive filming mode that literally juddered across the screen. At the time of writing, we are still awaiting the results of this testing from the Βι¶ΉΤΌΕΔ. [Please see the Postscript to Part 3 of this Blog.]
Comparisons Using our Selected Clips
The viewing of the subjective test clips was followed by an opportunity to compare a selection of clips of our choosing. We selected them, after considerable discussion, well in advance of the visit and were shown them through the same set-up used for the subjective tests and in the same three versions - source, current and old encoders.These tests were conducted in a less formal way than the subjective test, with the complete group free to roam around in front of the three monitors. Each version was shown side by side on three displays, and the group members were able to discuss amongst themselves, and with Andy and the R&D team, what they thought of each version.
The versions were not swapped around, so all the source clips, for example, were shown on the same display. However, we did not know which display was showing which version until the end of the sequences, when we were invited to offer our opinions as to which display was showing which version.
The test clips used are detailed in the table below, together with an indication of the issue that that had been raised in the Βι¶ΉΤΌΕΔ HD blogs by group members and others, plus any comment or explanation made on the day by the Βι¶ΉΤΌΕΔ. It should be noted that some suggested clips, showing issues that have not been acknowledged by the Βι¶ΉΤΌΕΔ, were not made available.
Clips 1-3 were broadcast originally with the old encoder set-up and had been identified by the group to have degraded when broadcast again using the current encoder set-up.
Clips 4, 5, 6 and 8 have only been broadcast with the current encoder set-up but had been identified by the group as exhibiting certain problems.
It's worth stating here that the test conditions were appalling. The set-up of the screens and the group meant that all of us were getting in each other's way as we moved between the displays, it was virtually impossible to see all three at the same time, and the clips were only shown once.
This made it difficult to do meaningful comparisons, and we feel a more rigorous set-up along the lines of the ITU test would have been preferable. Nevertheless, the majority of the group correctly identified the old and new set-ups, and all bar one identified the source. After the source was rapidly identified we were asked as a group two questions in quick succession: which was the old encoder and which the new, and which did we prefer. That some people didn't correctly identify the old and new set-ups may have been partly due to the physical layout of the test, but also because of the ambiguity of these questions since in some circumstances the new encoder performed better.
For example, in the Survivors clip a car's motion across the screen was jerky with the old encoder and handled much better by the new. Further, it took a minute or two for people to decide. This may have been partly because there was a lot of information to process but group dynamics played a role here as well. Undoubtedly some were reluctant to be the first to say anything in case they were wrong, as is natural in group situations. However, others later reported that they were immediately certain which was which, but left it a little while from a sense of fairness, not wanting to deny others a chance to voice an opinion.
Further reflections on encoder performance
In this comparison we could see the encoders making different decisions; they looked different depending on the source material and the scene content of the moment. This shouldn't really be of any surprise. At times the new encoder did look superior, but its efficacy did vary.However, the new encoder does have overt specific problems. Its blocking was not purely related to the acknowledged mix/fade issue; it was observed on demanding scenes without particular variances in luminance (brightness).
Noise handling is also an interesting area. The source material is often more noisy than many would expect. However, when viewed on the high bit-rate source, it looks natural, one might even say "analogue" in nature. As such it did not seem particularly distracting. However, once it has been passed through the encoder its nature can change. It can look "digital" and unnatural. Hence people commenting on the backgrounds in some material looking "unstable" or even "crawling". This is distracting to the viewer. The old encoder does not produce these artefacts to the same extent; it is less overt and therefore less distracting.
In the simplest terms, if we used a scale of 1 to 10 to judge the respective encoder's performance, with 1 being poor/ showing annoying distracting artefacts and 10 being excellent/no annoying distracting artefacts, the performance range of the new encoder would straddle the old. This is a completely arbitrary scale and must not be taken literally, but one might place the old encoder's performance with a variety of material ranging between say 5 and 7. The new however might range 3 to 8. The old encoder handles the problems presented by the more difficult material in a far more palatable way.
Finally, while the encoder comparisons were very valuable and helped us consolidate a number of insights, the group would not want to subscribe to the implication arguably inherent in the two test set-ups that parity with the old encoder should be the benchmark against which success for Βι¶ΉΤΌΕΔ HD is measured. As our own experiences attest, the old encoder had its weaknesses as well, as was perhaps evidenced in the welter of complaints on the Βι¶ΉΤΌΕΔ HD blogs when the bit-rate was previously reduced from 19 to 16 Mbps approx.
Demonstration of the Latest Encoder Fix
The Βι¶ΉΤΌΕΔ has already admitted the problems of the mix/fade issue with the current encoder that are visible in the Waterloo Road clip, for example, and they have also stated that they are working on a fix. Consequently, we were given an opportunity at the end of the visit to R&D to see the Waterloo Road clip when played through the current encoder set-up but with the latest fix applied to address this issue. This was played alongside the set-up without the fix so that we could judge the improvements.We note that the fix is still in testing, so it would be inappropriate for us to attempt to make comment on its efficacy yet. However, what we can say is that we are all very disappointed that this resolution, to a problem evident in many programmes, has still not been implemented after 9 months (at the time of writing)
Failure to Demonstrate the New Encoder at the Old Bit-rate
Unfortunately, for reasons not disclosed, we were not able to see a demonstration of the new encoder working at the older (i.e. higher) bit-rate. Andy Quested insisted, several times during the day, that matching the new encoder with a higher bit-rate would make no noticeable difference to the PQ of the Βι¶ΉΤΌΕΔ's HD channel's transmissions. However, when challenged, he did acknowledge that a bit-rate in the range of 12-13 Mbps would probably produce a slight improvement but he qualified that by stressing that this would be insufficient justification to warrant a change. (N.B. This position directly contradicts the statement, commented on in the previous Blog, by the Acting Head of Distribution, Stephen Baily, who said viewers would be disappointed if the Βι¶ΉΤΌΕΔ increased the bit-rate only to reduce it later when the bandwidth was needed for other purposes.)Considering the effort that the Βι¶ΉΤΌΕΔ, and particularly its R&D team, put into the day in order to show us all other aspects pertinent to our PQ complaint we were surprised by the omission of what might be considered the one demonstration that could have proven to us, once and for all, that the difference made by a higher bit-rate is insignificant.
Thus ends our blog on the visit to R&D. Tomorrow will be where we draw conclusions from the whole day, and make some recommendations to the Βι¶ΉΤΌΕΔ."
Read part one of this post: Picture Quality on Βι¶ΉΤΌΕΔ HD: a Viewers' Group Visit.
Comment number 1.
At 5th Jun 2010, Briantist wrote:To be honest, that the panel were allowed to discuss their choices of "old", "new" and "source" rather nullifies the results somewhat.
Surely the tests should be taken "blind" with each member of panel recording their results during the tests?
I am a little disappointed by phrases like "handles the problems presented by the more difficult material in a far more palatable way", with no actual evidence presented defining "palatable", and also "one might even say 'analogue' in nature", whereas there is no "Full HD" analogue system in existence.
"Unfortunately, for reasons not disclosed ... the older (i.e. higher) bit-rate".
Why no reference to the amount of bandwidth two Βι¶ΉΤΌΕΔ HD channels will take up on Freeview HD?
Complain about this comment (Comment number 1)
Comment number 2.
At 5th Jun 2010, Andy Quested wrote:Dear Briantist - the test were "blind" and conducted according to ITU-R BT.500 specifications as many have asked for. After the group visit we conducted the same test was conducted with other groups to give about twice the minimum recommended test size.
The results of the groups tests (alone) and the combined tests were sent to the group before these blogs were published.
Andy
Complain about this comment (Comment number 2)
Comment number 3.
At 5th Jun 2010, BikeNutt wrote:Thank you team. Very eloquently written.
So the one test that would have conclusively proved whether or not using a higher bit-rate in conjunction with the new encoders has an effect on picture quality in the user's home was not offered. I think that says it all doesn't it?
Complain about this comment (Comment number 3)
Comment number 4.
At 5th Jun 2010, tagmclaren wrote:Hi Briantist,
Just in case there is any confusion, the ITU test and informal viewing comparison were separate tests. No discussion of the viewed material was had during the ITU test. It was performed in accordance with the test standards. The results are discussed in tomorrows blog.
Regarding the informal comparison test, it was still blind, we did not know which was which. Although I take your point regarding the possiblity of group dynamics, the encoders and were identifiable from their respective picture quality and the specific problems the new suffers from when presented with the right (wrong!?) material.
IMHO on of the problem with digital systems is that the defects tend to be more objectionable to viewers. Most will tolerate a bit of "snow" on an analogue picture. It doesn't distract too much from the important bit - the content. However a burst of blocking, or the scene backgrounds "squirming" around (see Garrows law comments above) due to the encoder not being able to deal in a graceful way with the noise in the source, is far more distracting.
So the use of the phrase "palatable" simply is indicating the new encoder blocked overtly where you might say the old, although also struggling, "disguised" the problem in a way that wasn't as overtly obvious. Reducing detail maybe. Sometimes it is difficult to articulate subjective observations.
Complain about this comment (Comment number 4)
Comment number 5.
At 5th Jun 2010, tagmclaren wrote:Andy,
I haven't managed to see much content with the latest software, but I did catch the testcard just now. Whats the issue with the frequency gratings?
There are some bands in the 25 and (aliased) 30 which were not there before.
Thanks
Complain about this comment (Comment number 5)
Comment number 6.
At 5th Jun 2010, Andy Quested wrote:Dear tagmclaren - you win the prize!! I had a bet with R&D if one of the group or another would spot it first but I won't say who won!
The good news is that unless a producer decides to use a shot with a 25MHz grating on top of a 30MHz grating you won't see it on pictures. We think it's something to do with the Nyquist limit and are looking at a fix with the manufacturer
Andy
Complain about this comment (Comment number 6)
Comment number 7.
At 5th Jun 2010, Peter wrote:I've been tracking the conversations on bit rates, encoders for sometime, and find these blog posts interesting, so thank you to the the Βι¶ΉΤΌΕΔ and the viewers panel.
For me, one day I noticed that the PQ of Βι¶ΉΤΌΕΔ HD just didn't seem to be as good as before, it was a few days later that I read about the change of encoder and bit rate. Therefore I had seen in effectively a blind test a degradation.
Although not HD, but still relevant, the effect of reducing the bit rate can be very easily demonstrated by watching the different Βι¶ΉΤΌΕΔ1 region feeds on the Sky platform. For instance Βι¶ΉΤΌΕΔ1 London (973) and Βι¶ΉΤΌΕΔ1 Northern Ireland (974). The former looks obviously much better than the latter.
With all good scientific research you never change more than one variable at a time. The test as documented here do not follow this mandate, i.e. both the encoder and the bit rate varied between the two tests. I believe that all four combinations would need to be seen before conclusive observations can be drawn, i.e.
old encoder, higher bitrate
old encoder, lower bitrate
new encoder, lower bitrate
new encoder, higher bitrate
Would it be possible for the Βι¶ΉΤΌΕΔ use some of the off air time on the HD channel to broadcast these test samples via the two encoders, and even the two bitrates, to a larger audience?
Complain about this comment (Comment number 7)
Comment number 8.
At 5th Jun 2010, citizenloz wrote:Andy, as you seem to be more actively monitoring this blog, can you respond to the post I made @1347 in the old Salmon Style blog.
I didn't see the Βι¶ΉΤΌΕΔ2 version, so I don't know how the SD version compares, but this scene on Βι¶ΉΤΌΕΔ HD was very poor IMO.
To repeat:
There was another example this evening of SD material being handled very poorly when used in an HD programme. Similar comments were made about programmes like How Earth Made us.
I understand the need to use SD source material at time, I don't object to that.
But in Timewatch Atlantis: The Evidence, there were scenes for example at 11.00 minutes in on board a ship that appeared to have been sourced in SD, but were simply full of artifacts, often on edges of objects. I have no way of knowing if the source was like that in the first place - if so, it was a poor choice of material to include in an HD programme.
It seems to me like the new encoders struggle with SD material as it can't make its mind up how it should deal with 'soft' images (in comparison to HD)
Complain about this comment (Comment number 8)
Comment number 9.
At 5th Jun 2010, HD wrote:Why were the tests done only with Plasma HDTVs, when much more viewers will be watching on LCD HDTVs? Is it because Plasma HDTVs hide artefacts more than LCD HDTVs?
Complain about this comment (Comment number 9)
Comment number 10.
At 5th Jun 2010, tagmclaren wrote:Andy ~ 6
Well, the test card seems a logical place to start when having a look at potential changes in PQ.
Just for fun I will look out for finely checked shirts! ;-)
Complain about this comment (Comment number 10)
Comment number 11.
At 5th Jun 2010, Andy Quested wrote:Dear Peter - Thanks for the comments. The tests are valid and done in accordance to the ITU document. They do NOT test one encoder against the other though, the scoring is one clip against the original. Of the two clips is always the original (either first or second) and the other clip is either the new encoder or the old encoder or the original. Scoring is absolute not relative - during the process each test is shown twice. The first five are discarded to remove "settling in" variations
For the four different sequences there were in total 24 different tests scored.
Unfortunately we can't broadcast the test as you would need a 440Mbs link to get the un-prosessed version.
Dear citizenloz thanks for the post - unfortunately there will always be SD material in some programmes and some of it will be poor quality SD. I don't know the reason for the shots in question but as part of the 25% SD limit almost any camera can be used.
Also some non-broadcast digital recording systems (especially some CCTV style installations) or some of the consumer recorders introduce visible digital artifacts themselves so no matter what the transmission bit rate, the process of editing, colour correction, playout and transmission can only make them worse.
Dear HD1080 - Plasma or LCD it doesn't matter as long as all the displays used (we used four) are set up identically and do not produce unacceptable artifacts themselves.
Complain about this comment (Comment number 11)
Comment number 12.
At 5th Jun 2010, citizenloz wrote:@ 11 Andy Quested wrote:
Dear citizenloz thanks for the post - unfortunately there will always be SD material in some programmes and some of it will be poor quality SD. I don't know the reason for the shots in question but as part of the 25% SD limit almost any camera can be used.
Andy, please read my post instead of just giving a stock answer. I already said I knew that SD is used occasionally, I am not complaining about that.
The POINT is, that this clip, like others that have been mentioned before is particularly bad - it isn't just that it is SD, it that it is badly artifacted, and I am questioning whether the new encoder is to blame.
The clips in How Earth Made Us, were also particularly bad, but you never replied to that either, nor made the clips available for review on our visit as we did ask. It wasn't that the clips in HEMU were SD, it was they appeared worse than SD.
Hence the question is in both cases, was the source material really that bad? Was it like that in the Βι¶ΉΤΌΕΔ2 broadcast? Or is there something specific going in the HD broadcast.
You need to go look at the clips, not give a stock answer...
Complain about this comment (Comment number 12)
Comment number 13.
At 5th Jun 2010, Dean Davidson wrote:Hey, I wonder if you'll be able to tell me all the resident programs related to the interaction channel on mheg-5.
I currently have detailed information on SHF, RDa in a book S&T provided me with last year - I'm a studen and i've been selfteaching myself MHEG-5.
I know, I'll most likly get no result from here..
~ But, there is no where else I can ask.. Since neither PressRed / Βι¶ΉΤΌΕΔi blogs exist anymore (Which, is a real shame!)
- Thanks, Dean.
Complain about this comment (Comment number 13)
Comment number 14.
At 5th Jun 2010, burnlea wrote:#11 Andy Quested
"The tests are valid and done in accordance to the ITU document. They do NOT test one encoder against the other though, the scoring is one clip against the original. Of the two clips is always the original (either first or second) and the other clip is either the new encoder or the old encoder or the original. Scoring is absolute not relative - during the process each test is shown twice. The first five are discarded to remove "settling in" variations
For the four different sequences there were in total 24 different tests scored."
And as we will see tomorrow from these 'valid' tests, they are not conclusive.
To avoid any confusion about your comment "Scoring is absolute not relative", the absoluteness of the scoring was for Clip A against Clip B within each of the 24 tests, but not absolute across the 24 tests as that was not possible to achieve.
Complain about this comment (Comment number 14)
Comment number 15.
At 5th Jun 2010, Andy Quested wrote:Dear citizenloz - I'm sorry you thought it was a stock answer - I intended it to be a generic answer about any low quality SD in an HD programme.
1. SD in HD programmes is either archive in which case i could be high quality (digibeta recorded) or shot on low cost or specialist or low quality cameras, usually because there is no HD camera that can do the job or the HD cameras are too big or the shots come from other providers (not shot by the programme)
2. Because of 1. above often the recording format may also be lower quality than usual or domestic (for reasons of size, power or ruggedness).
Generally the lower quality will stand out far more against native HD and digital artifact invisible in SD become far more visible. An example would some of the (just) underwater shots of fish in the bear sequence of salmon run.
Andy
Complain about this comment (Comment number 15)
Comment number 16.
At 5th Jun 2010, Andy Quested wrote:Dear all (again) - Just a quick note, can you let me know which platform you are commenting on? Either use the name - Sky/Freesat or Freeview or Virgin or Sat, DTT, Cable is OK. I saves me having to ask before replying - I can't assume you are all satellite any more!!
Andy
Complain about this comment (Comment number 16)
Comment number 17.
At 5th Jun 2010, 128fish wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 17)
Comment number 18.
At 6th Jun 2010, Al wrote:Andy I'm disappointed that you wouldn't allow a back to back compatrison of 16mbs vs 9.6mbs. If you have nothing to hide, what wsa the problem in allowing this?
Also, really would have liked to have seen South Pacific - the surfing epsiode used in the comparison as it was absolutely terrible quality with the inboat sequences etc until the slow motion kicked in and then low and behold, prefect HD quality when the frame changes slow down. Can there be any better demonstration of how the encoder is failing to cope? I would doubt so. Best example within a single programme I've seen so far.
Complain about this comment (Comment number 18)
Comment number 19.
At 6th Jun 2010, Andy Quested wrote:Dear Alsone - thanks for the post. We did show 16Mbs and 9.5Mbs throughout the ITU test and the examination of the selected clips. Re South Pacific - I'm afraid no one asked to see the surfing episode, but I will check to see if it is a good candidate for the test roll I mentioned earlier
Andy
Complain about this comment (Comment number 19)
Comment number 20.
At 6th Jun 2010, burnlea wrote:#19 Andy Quested
Your response to Alsone #18, was disengenuous. You well know that he was referring to the fact that you didn't show us side by side comparisons of the new encoder at 9.7 Mbps and at 16 Mbps. The only clips we saw at 16 Mbps were via the old old encoder.
Complain about this comment (Comment number 20)
Comment number 21.
At 6th Jun 2010, burnlea wrote:The third section of our visit report, containing conclusions and recommendations, has now been posted here:
/blogs/bbcinternet/2010/06/picture_quality_on_bbc_hd_a_vi_2.html
Complain about this comment (Comment number 21)
Comment number 22.
At 6th Jun 2010, Nick Mason wrote:Gents, thanks to all that have spent a great deal of their time in (a) visiting the Βι¶ΉΤΌΕΔ and (b) doing the write up afterwards.
However, I am sad to say that because of this HUGE omission "Failure to Demonstrate the New Encoder at the Old Bit-rate", I conclude that all your efforts were entirely wasted. This surly must have been the cornerstone of the whole debate.
AQ – this smacks of a deliberate mis-direction on the part of the Βι¶ΉΤΌΕΔ team – shame.
Complain about this comment (Comment number 22)
Comment number 23.
At 6th Jun 2010, Andy Quested wrote:Dear burnlea - sorry you think my response to Alsone was not straight forward. When I contacted Paul about the visit I did say this was all about the new encoder vs the old and that we had chosen to the settings on the new to match or exceed the quality of the old (as stage 1 of the new encoders development)).
The complaint to the Βι¶ΉΤΌΕΔ Trust and the No. 10 petition is all about the bit rate drop causing a degradation in quality after switching to the new encoder.
We have acknowledged the new encoder had a problem with mixes and fades and we would be working to fix this as fast as possible. The first stage of that fix (other than the temporary patch) was uploaded to the encoders on Friday afternoon. Also some of the other work to improve picture quality over and above the old encoder is nearly ready to be uploaded. I will keep you posted about further updates.
Andy
Complain about this comment (Comment number 23)
Comment number 24.
At 6th Jun 2010, burnlea wrote:#23 Andy Quested
"When I contacted Paul about the visit I did say this was all about the new encoder vs the old and that we had chosen to the settings on the new to match or exceed the quality of the old"
But that didn't preclude you showing us the new encoder at a higher bitrate. As so many people have now commented, that would have ended the bitrate argument one way or the other. I think it is clear now what it would have demonstrated, as you yourself stated under questioning on the day that there could be a noticeable improvement in PQ if the bitrate was increased.
As we had confirmed on the visit, it was platform neutrality with Freeview, and spectrum planninng (the need to fit in a second HD channel on satelite) that determined the current bitrate, and not "match or exceed the quality of the old" as that hasn't been achieved over the complete range of source material.
Complain about this comment (Comment number 24)
Comment number 25.
At 6th Jun 2010, Andy Quested wrote:Dear burlea - thanks for the post. During the session at R&D, they did say the bit rate was chosen to matched (or exceeded) the then current quality over the majority of the output - the ITU test did confirm this position and even a clip chosen to look worse on the new encoder scored hardly any different new to old. I hope no one can deny I have said this since the beginning of the debate.
Having watched a lot of the promo today I can say the upgrade has made this position even better.
There was an opportunity to see clips chosen by yourselves via the new and old encoder with the original running too. These were observed at less than 1h and were examined for specific know artifacts. The displays used were the ones the ITU tests had been conducted on so were well lined up.
If you are looking for 100% then I cannot help yet, what I can offer now is an improvement on majority of the material shown, to be at least as good as on the major part of the rest and (acknowledged from the start) a couple of problem areas that we committed to work on as quickly as possible.
Andy
Complain about this comment (Comment number 25)
Comment number 26.
At 6th Jun 2010, burnlea wrote:#25 Andy Quested
"During the session at R&D, they did say the bit rate was chosen to matched (or exceeded) the then current quality over the majority of the output
But we don't dispute that for a proportion of the Βι¶ΉΤΌΕΔ HD output, it does match the pre August 2009 quality. It is that portion where the current quality doesn't match the pre-August 2009 picture quality due to demanding source material that the current encoder/bitrate can't cope with satisfactorily, that is the subject of our complaints, and the even larger portion that doesn't match the Βι¶ΉΤΌΕΔ HD trial picture quality.
We are not looking for 100% as you put it. There is ample ground between the current picture quality and what was achieved during the HD trial period that could be recovered, with for example a change of encoder, a move from 1440 x 1080 to 1920 x 1080 or an increase in bitrate. Contentious I know, but refusal to demonstrate the current encoder at a higher bitrate than 9.7 Mbps will continue to leave this option, which has science on its side, open to speculation.
Complain about this comment (Comment number 26)
Comment number 27.
At 6th Jun 2010, Andy Quested wrote:Dear burnlea thanks for the post can I just say there is no refusal - it's more that as we intend to move to VBR and then into a sta mux we would rather put the effort into achieving that as soon as we can. Also the proportion of material that's better is very high and the areas where the new has problems is extremely limited and as the ITU test showed not that different to the old.
Andy
Complain about this comment (Comment number 27)
Comment number 28.
At 6th Jun 2010, tagmclaren wrote:Andy,
You say:-
"and the areas where the new has problems is extremely limited and as the ITU test showed not that different to the old"
I think we would disagree with that statement. The problems are far more pervasive than you acknolwledge (ignoring latest software update).
The ITU test we took part in only had one clip that showed the deficiencies of the new encoder. Why didn't you add in the Waterloo road clip? What would results shown then? 4 clips are not representative of the very broad range of Βι¶ΉΤΌΕΔHD output. Did the EBU clips have mixes or fades? How many programs in reality have mixes and fades or other changes in luminance that kick the problem off?
The other big problem with that statement is that you are again using the old outdated encoder as the benchmark. The objective should have been to noticeably improve on the old with all material, not "most" as you put it. The objective should have been to offer the highest possible quality.
With the problems the new encoder has, which you have only just started to improve after 9 months, this was not achieved.
Moving on from that, you have previously stated that you want to move to 1920 when you can, so must feel it is desireable/of benefit.
What is stopping you doing so now?
Could you also give some detail on the upcoming set of improvements and timescales for implementation?
Thanks
Complain about this comment (Comment number 28)
Comment number 29.
At 6th Jun 2010, Andy Quested wrote:Dear tagmclaren - last post before bed!!
The reason for choosing a JR sequence was:
1. The number of posts about the programme - remember too, after the first programme I investigated the light levels and camera gain and distributed new delivery guideline later that week.
2. Flashing lights are far more common - Jools, JR wvery week Glastonbury etc...
3. It's not often you see a blow torch level flame so close to two people having dinner - it was pretty "bold" on the original.
4. Each new clip adds six tests (orig to new, orig to old, orig to orig and repeat) that's another six mins and you reached the maximum duration of a test with four.
The clips chosen were - two x difficult EBU, one x easy (but with lots of detail) and the JR clip - I know you rated it as "moderate" but as there were high colours, noisy black level, changing light levels and continuious movement - I would say for the set up when you saw it, it was a "challenging" sequence.
If we had used WR it would guess the original would have been marked low (for the resons above) the old very low and the new lower but not the lowest!
No more tonight - I have to get the teenager to school early (after half term) as it's exam week!
Andy
Complain about this comment (Comment number 29)
Comment number 30.
At 7th Jun 2010, Bill Taylor wrote:Thanks for the blog.
The Βι¶ΉΤΌΕΔ did not offer a test viewing of the new encoders at the old bit-rate. I can only infer that this was down to their awareness that the difference would be noticeable and undermine their decision to reduce the bit-rate.
I have had to accept that the Βι¶ΉΤΌΕΔ can no longer be the Gold Standard for HD broadcasting and I have to go elsewhere for Stunning HD PQ and sound.
Complain about this comment (Comment number 30)
Comment number 31.
At 7th Jun 2010, ChrisK wrote:#22 Nick Mason wrote:
".. Failure to Demonstrate the New Encoder at the Old Bit-rate, I conclude that all your efforts were entirely wasted. This surly must have been the cornerstone of the whole debate."
This entire debate was triggered by some viewers who perceived a change in PQ with the change from Old/high bit-rate to New/Lower bit-rate. This is what was put to the test in the viewing comparisons. I can't see how that makes the day wasted. If these blind comparisons show that there is little perceptible difference then the Βι¶ΉΤΌΕΔ have proved their point - you essentially get the same PQ and a saving in bandwidth.
By the way, thanks to the group and the Βι¶ΉΤΌΕΔ for doing this. I think the openness is to be commended.
Complain about this comment (Comment number 31)
Comment number 32.
At 7th Jun 2010, burnlea wrote:#31 ChrisK
"If these blind comparisons show that there is little perceptible difference then the Βι¶ΉΤΌΕΔ have proved their point - you essentially get the same PQ and a saving in bandwidth"
But the blind comparisons didn't show that. For the EBU material which was all naturally lit outdoor clips, yes, but for the 'real program clip' Lady Gaga on the Jonathan Ross show, no. The old encoder/higher bitrate was noticeably better for this clip. As mentioned many times, it is the really demanding material that shows up the deficiencies of the new encoder/lower bitrate, and that was demonstrated with the Lady Gaga clip.
So, no, you don't get the same PQ and a saving in bandwidth for the most demanding material.
Complain about this comment (Comment number 32)
Comment number 33.
At 7th Jun 2010, Andy Quested wrote:Dear all - this is just a very quick post to say VBR started this afternoon on DSat
Dear ChrisK and burnlea
ChrisK - you are correct in the comment about the debate, that is exactly what the day and the blogs are about. burnlea - sorry to disageree but the EBU material is challenging and the difference was not minor. As for the JR clip the difference was very small in favour of the old but both new and old were very close to the original.
Andy
Complain about this comment (Comment number 33)
Comment number 34.
At 7th Jun 2010, burnlea wrote:#33 Andy Quested
Yes the EBU material was challenging, but in a different way to the Lady Gaga clip which clearly showed up the deficiencies of the then encoder/bitrate, to the visiting viewers group and the wider group.
On a positive note, the Tennis from Queens looks remarkably good, no doubt as a result of VBR. Thank you for this change. What upper and lower bitrates have been set as they won't be available on Linowsat until tommorow?
Complain about this comment (Comment number 34)
Comment number 35.
At 7th Jun 2010, tagmclaren wrote:Sorry Andy,
I have to disagree, I don't think JR clip was that close to the original. The problems were obvious.
Complain about this comment (Comment number 35)
Comment number 36.
At 7th Jun 2010, tagmclaren wrote:Andy,
Regarding the tennis, when I was watching the first match this afternoon the stripes in the grass directly behind the net seemed to sort of "dislocate" as the main camera panned left and right. I also thought my video processor breifly jumped into film mode deinterlacing a few times
Problems seem to have gone away now though.
Is the main camera as well focussed as it could be?
Complain about this comment (Comment number 36)
Comment number 37.
At 7th Jun 2010, citizenloz wrote:@36 tagmclaren wrote:
Is the main camera as well focussed as it could be?
I was wondering the same myself. Anything from the net onwards seems distinctly soft.
Note, it's exactly the same feed on Eurosport HD, and any problems are apparent on both.
Complain about this comment (Comment number 37)
Comment number 38.
At 7th Jun 2010, burnlea wrote:As Andy continues to not accept that the Lady Gaga clip was more challenging to the encoder than the other 3 clips we were comparing, see for yourself. The particular segment we were shown can be seen in this Youtube clip between 2.05 and 2.15 minutes in:
As mention previously, the other 3 clips were all taken outdoors in good natural lighting conditions, but with other characteristics to challenge the encoder.
Complain about this comment (Comment number 38)
Comment number 39.
At 7th Jun 2010, Andy Quested wrote:Quick note to all re Queens - I was on the phone as soon as play started, we are looking at a possible configuration issue with the link back to Television Centre.
Dear tagmclaren - thanks for the post - for the group the JR clip measured very little difference between new and old encoder and relatively close to the original
Andy
Complain about this comment (Comment number 39)
Comment number 40.
At 7th Jun 2010, Andy Quested wrote:Dear burnlea - thanks for the post - It was the group who designated the JR clip as "moderate" - see the table in the blog - and I said it challenged the our current (then) encoder set-up because it had lighting changes, high colour, noise in the dark areas and movement across lighting/smoke effects.
Also, I still don't understand why the use of EBU test material is being challenged after saying for so long we do not follow EBU guidelines. The EBU material was shot to challenge encoders - just as the JR clip challenged an acknowledged deficiency in our encoder.
Complain about this comment (Comment number 40)
Comment number 41.
At 7th Jun 2010, burnlea wrote:#40 Andy Quested.
I don't really understand your response, but thank you for it.
"It was the group who designated the JR clip as "moderate" - see the table in the blog"
Did anyone suggest otherwise? Just confirms that the then encoder/bitrate setup had problems with 'moderate' material
"- and I said it challenged the our current (then) encoder set-up because it had lighting changes, high colour, noise in the dark areas and movement across lighting/smoke effects."
Agreed - but challenged it more than it challenged the pre August 2009 setup, which coped better with it.
"Also, I still don't understand why the use of EBU test material is being challenged after saying for so long we do not follow EBU guidelines. The EBU material was shot to challenge encoders - just as the JR clip challenged an acknowledged deficiency in our encoder."
I don't see anyone challenging the use of the EBU test material, only commenting that it challenges the encoder in a different way to way the JR material does, which exposes issues (other than your acknowledged deficiency) that the EBU material doesn't.
Can we move on to the welcome introduction of VBR and let us know what bitrate parameters have been set for it?
Complain about this comment (Comment number 41)
Comment number 42.
At 7th Jun 2010, Nick Mason wrote:AQ - I take your point that your future efforts will go into VBR and sta mux. BUT - you really did miss a HUGE opportunity to draw a line under the whole bit-rate debate. Own goal? Or didn't you fancy the odds on the outcome?!
Complain about this comment (Comment number 42)
Comment number 43.
At 8th Jun 2010, Andy Quested wrote:Dear burnlea - thanks for the comment. In post 38 you said "As Andy continues to not accept that the Lady Gaga clip was more challenging to the encoder..."
You also said "As mention previously, the other 3 clips were all taken outdoors in good natural lighting conditions, but with other characteristics to challenge the encoder." The EBU clips were shot to challenge encoders and demonstrate shortcomings - lighting, location or content aside.
The results of the ITU test confirm this
Dear Nick Mason - I am not sure what you mean by own goal???
Andy
Complain about this comment (Comment number 43)
Comment number 44.
At 8th Jun 2010, citizenloz wrote:It is a pity that we were not given the opportunity to make the "comparisons using our selected clips" using the same ITU testing methodology.
As it is, it seems increasingly likely that the Βι¶ΉΤΌΕΔ Trust will make decisions on the appeal based on the tests using the EBU clips, but not consider how that might have turned out if Βι¶ΉΤΌΕΔ's own clips as selected by viewers had been used.
Complain about this comment (Comment number 44)
Comment number 45.
At 8th Jun 2010, jTemplar wrote:I have to say I think the sound on Luther this evening is very good ... although the very low bit-rate readings I am getting suggest this programme is more SD than HD.
Complain about this comment (Comment number 45)
Comment number 46.
At 8th Jun 2010, Andy Quested wrote:Dear John Temperley - thanks for the post - we will be checking the lower levels over the next few weeks but if the encoder has it right adding more bit will do nothing other than sharpen the noise.
Andy
Complain about this comment (Comment number 46)
Comment number 47.
At 9th Jun 2010, Nick Mason wrote:#43 AQ - #42 NM Andy, if you take out my phrases of 'own goal' and fancy the odds', would you then see the point I'm making?
Wow, I see we have to blunt with you, else you go off on a tangent :)
Complain about this comment (Comment number 47)
Comment number 48.
At 9th Jun 2010, Andy Quested wrote:Dear Nick Mason - thanks for the post - i did get the football connection but I'm not sure how to draw the line! Happy to oblige though:)
Andy
Complain about this comment (Comment number 48)
Comment number 49.
At 11th Jun 2010, airjaff wrote:watching the world cup match right france vs uruguay bbc hd not to impressed the itv hd quality was much better for me! shame on the bbc
Complain about this comment (Comment number 49)
Comment number 50.
At 15th Jun 2010, Nick Reynolds wrote:Just to let you know that I am closing this blog for comments. Please go to blog 3 if you wiah to carry on the conversation.
Complain about this comment (Comment number 50)