Subject: [rmcat] PSNR and draft-singh-rmcat-cc-eval-02

From: Michael Welzl <[email-masked]>
Date: Thu, 25 Apr 2013 14:31:54 +0200

Hi all,

draft-singh-rmcat-cc-eval-02 contains the following text:

***
      [Editor's Note: Should the algorithm compare average PSNR of test
      video sequences or what other video quality metric can be used?
      If Quality is used as a metric, it should not be the only metric
      used to compare rate-control schemes.  Also, algorithms using
      different codecs cannot be compared].
***

Here's a proposal:

In a recent email dialogue, my colleague Carsten Griwodz mentioned to me that "using PSNR for assessing the quality of adaptive video must be forbidden" (and this isn't the first time I hear such statements from multimedia people). He went on to write, about the document I'm attaching:

"See the attached PDF, Appendix I, the tables in section I.3.1. Networking people use PSNR all the time, and it is absolutely bogus. The table shows that it is crap for stable video quality. For actively adapted video (e.g. not by dropping packets but by changing compression), it predicts subjective quality correctly in 30% of the cases. Papers that use it as an optimization goal cannot be trusted."

Of course one reason we're using it is its simplicity; if we want to *truly* assess quality, many more factors come into play, making it quite impossible to take this route for most of us. However, it seems that almost everything else is better than PSNR, so we could resort to at least a simple alternative.

Can we settle for VQM? It is easily available: http://www.its.bldrdoc.gov/resources/video-quality-research/software.aspx
Register and receive a download URL. You can access Matlab code and Windows executables for batch processing (bvqm_*) and command line processing (cvqm_*). One of those is enough, the other tools aren't required.
There's a more convenient Windows implementation of VQM from MSU, but it costs money.

VQM does OK for resolutions up to VGA / SD video (no known results for higher resolutions), but the most important point is that it considers changes over time, i.e. it is made for video, not for images. It's feature set is described in ITU-T Rec. J.144 Annex D. Some more reports about VQM are here:
http://www.its.bldrdoc.gov/publications/2574.aspx
http://www.its.bldrdoc.gov/publications/2572.aspx
http://www.its.bldrdoc.gov/publications/1882.aspx

While VQM is not the best known objective test (PEVQ is), VQM is the best known test with a free implementation and no strings attached.

Cheers,
Michael

---

From: Kevin Gross <[email-masked]>
Date: Thu, 25 Apr 2013 08:57:14 -0600

My proposal, included in my recent comments on
draft-singh-rmcat-cc-eval-02, is simpler: "media quality is at best
indirect and at worst and unreliable indicator of network performance. I
would like to see it removed."

Kevin Gross
+1-303-447-0517
Media Network Consultant
AVA Networks - www.AVAnw.com <http://www.avanw.com/>, www.X192.org

On Thu, Apr 25, 2013 at 6:31 AM, Michael Welzl <[email-masked]> wrote:

> In a recent email dialogue, my colleague Carsten Griwodz mentioned to me
> that "using PSNR for assessing the quality of adaptive video must be
> forbidden" (and this isn't the first time I hear such statements from
> multimedia people). He went on to write, about the document I'm attaching:
>
> "See the attached PDF, Appendix I, the tables in section I.3.1. Networking
> people use PSNR all the time, and it is absolutely bogus. The table shows
> that it is crap for stable video quality. For actively adapted video (e.g.
> not by dropping packets but by changing compression), it predicts
> subjective quality correctly in 30% of the cases. Papers that use it as an
> optimization goal cannot be trusted."
>
> Of course one reason we're using it is its simplicity; if we want to
> *truly* assess quality, many more factors come into play, making it quite
> impossible to take this route for most of us. However, it seems that almost
> everything else is better than PSNR, so we could resort to at least a
> simple alternative.
>
> Can we settle for VQM? It is easily available:
> http://www.its.bldrdoc.gov/resources/video-quality-research/software.aspx
> Register and receive a download URL. You can access Matlab code and
> Windows executables for batch processing (bvqm_*) and command line
> processing (cvqm_*). One of those is enough, the other tools aren't
> required.
> VQM does OK for resolutions up to VGA / SD video (no known results for
> higher resolutions), but the most important point is that it considers
> changes over time, i.e. it is made for video, not for images. It's feature
> set is described in ITU-T Rec. J.144 Annex D. Some more reports about VQM
> are here:
> While VQM is not the best known objective test (PEVQ is), VQM is the best
> known test with a free implementation and no strings attached.
>
> Cheers,
> Michael
>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 25 Apr 2013 17:23:17 +0000

I agree with Kevin. Media quality metrics are not relevant for RMCAT. They include estimates of intrinsic media quality without transmission impairments, which is obviously not relevant for us, since we are not chartered to evaluate encoders or decoders. Even if we define new quality metrics which somehow exclude intrinsic quality and only estimate the impact of transmission impairments, this extremely extrapolates the transmission impairment into the uncontrolled domain of codec error resilience (decoder error concealment and encoder error recovery) which is highly dependent on codec implementation and configuration. No useful or reliable data can come from such metrics.

Transport performance metrics like IPPM may be a better starting point for us than any form of media quality metrics like VQM/PEVQ/PSNR.

The tricky part is defining what transport metrics really matter to media applications, and how they impact the applications. Another tricky part is separating the transport impairments that we intentionally impose for test purposes from the transport performance metrics that result from those impairments. We want to measure how the candidate algorithms respond to our test stimulus, not measure our test stimulus directly (which would be useless regurgitation).

Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Kevin Gross
Sent: Thursday, April 25, 2013 10:57 AM
To: Michael Welzl
Cc: [email-masked] WG; Carsten Griwodz
Subject: Re: [rmcat] PSNR and draft-singh-rmcat-cc-eval-02

My proposal, included in my recent comments on draft-singh-rmcat-cc-eval-02, is simpler: "media quality is at best indirect and at worst and unreliable indicator of network performance. I would like to see it removed."

Kevin Gross
+1-303-447-0517
Media Network Consultant
AVA Networks - www.AVAnw.com<http://www.avanw.com/>, www.X192.org<http://www.X192.org>

On Thu, Apr 25, 2013 at 6:31 AM, Michael Welzl <[email-masked]<mailto:[email-masked]>> wrote:
Hi all,

***

Here's a proposal:

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Thu, 25 Apr 2013 21:04:34 +0200

I agree, too, in fact I always had this opinion (but maybe I only said it in hallway discussions).

My point was rather, if we must have a quality metric, let it not be PSNR, but I'm all for removing this altogether. It makes any evaluation codec-dependant and rather unnecessarily complicates things, I think.

Cheers,
Michael

On Apr 25, 2013, at 7:23 PM, Mo Zanaty (mzanaty) <[email-masked]> wrote:

> AVA Networks - www.AVAnw.com, www.X192.org
>  
>

---

From: Joerg Ott <[email-masked]>
Date: Fri, 26 Apr 2013 08:25:38 +0300

I agree with Mo in that we need to find transport metrics that matter.
As long as we lack those we need an approximation and PSNR was a,
albeit quite crude, approximation, with all the known deficits.

I also agree that codecs matter.  But who says that an RMCAT congestion
control algorithm will be completely codec-independent?  If RTCWEB
is going to make a codec recommendation, it appears natural that
RMCAT would compare algorithms with this specific codec in mind.
So, the need to put things into a codec context could be part of such
an evaluation framework.  Not to mandate or endorse a codec here,
but maybe using a wording that makes exactly your (and our) concerns
explicit.  We could do this until we come up with a better way so
that this basic idea does not get lost.

We surely welcome proposals for which metrics matter and why.

Cheers,
Jörg

---

From: Michael Welzl <[email-masked]>
Date: Fri, 26 Apr 2013 09:37:08 +0200

Hi,

My concern is that  including a quality metric makes things more complicated, for little benefit. Folks can still write papers using them, just because the evaluation document doesn't prescribe any such thing - but if that document is meant as a basis for making decisions in the WG, I'd say we'd be better off if this was removed. Just my 2c.

To be a little more concrete here: the overall goals for a congestion control mechanism aren't sooo hard to define - try to get a high sending rate, but try to be able to track the application's send rate well, and try to minimize variations; try to minimize packet loss and variations thereof; try to minimize delay and variations thereof; be fair.

We're looking at these factors anyway, and it's probably correct to assume that, as a general rough rule, the more of these things a mechanism does right, the better the quality gets.

What a quality metric then buys us is to conflate these factors into a single one, which is not 100% trustworthy as well as codec-dependent. The gain that we get is that we can, as a function of the codec, assign different levels of importance to the various factors above. Personally, I don't think that's a big gain.

This being said, I'm not opposed to having quality metrics mentioned in the document, along with the concerns we're having about them, as long as this text is not very prescriptive.

Cheers,
Michael

---

From: Kevin Gross <[email-masked]>
Date: Fri, 26 Apr 2013 08:29:47 -0600

Devils advocate here.

In an ideal world, a quality metric goes directly at what end users care
about and would be a good cross-check to make sure we don't get so buried
in network performance metrics that we lose sight of the resulting user
experience.

We don't live in an ideal world. Existing quality metrics respond
inconsistently to packet loss and delay variation and do not capture the
importance of latency. We could give them a try to confirm this and then
try to improve them but this would greatly expand the scope of our work.

Kevin Gross
+1-303-447-0517
Media Network Consultant

On Fri, Apr 26, 2013 at 1:37 AM, Michael Welzl <[email-masked]> wrote:

> Hi,
>
> My concern is that  including a quality metric makes things more
> complicated, for little benefit. Folks can still write papers using them,
> just because the evaluation document doesn't prescribe any such thing - but
> if that document is meant as a basis for making decisions in the WG, I'd
> say we'd be better off if this was removed. Just my 2c.
>
> To be a little more concrete here: the overall goals for a congestion
> control mechanism aren't sooo hard to define - try to get a high sending
> rate, but try to be able to track the application's send rate well, and try
> to minimize variations; try to minimize packet loss and variations thereof;
> try to minimize delay and variations thereof; be fair.
>
> We're looking at these factors anyway, and it's probably correct to assume
> that, as a general rough rule, the more of these things a mechanism does
> right, the better the quality gets.
>
> What a quality metric then buys us is to conflate these factors into a
> single one, which is not 100% trustworthy as well as codec-dependent. The
> gain that we get is that we can, as a function of the codec, assign
> different levels of importance to the various factors above. Personally, I
> don't think that's a big gain.
>
> This being said, I'm not opposed to having quality metrics mentioned in
> the document, along with the concerns we're having about them, as long as
> this text is not very prescriptive.
>
> Cheers,
> Michael
>
>
> >> My point was rather, if we must have a quality metric, let it not be
> PSNR, but I'm all for removing this altogether. It makes any evaluation
> codec-dependant and rather unnecessarily complicates things, I think.
> >>
> >> Cheers,
> >> Michael
> >>
> >>
> >>> I agree with Kevin. Media quality metrics are not relevant for RMCAT.
> They include estimates of intrinsic media quality without transmission
> impairments, which is obviously not relevant for us, since we are not
> chartered to evaluate encoders or decoders. Even if we define new quality
> metrics which somehow exclude intrinsic quality and only estimate the
> impact of transmission impairments, this extremely extrapolates the
> transmission impairment into the uncontrolled domain of codec error
> resilience (decoder error concealment and encoder error recovery) which is
> highly dependent on codec implementation and configuration. No useful or
> reliable data can come from such metrics.
> >>>
> >>> Transport performance metrics like IPPM may be a better starting point
> for us than any form of media quality metrics like VQM/PEVQ/PSNR.
> >>>
> >>> The tricky part is defining what transport metrics really matter to
> media applications, and how they impact the applications. Another tricky
> part is separating the transport impairments that we intentionally impose
> for test purposes from the transport performance metrics that result from
> those impairments. We want to measure how the candidate algorithms respond
> to our test stimulus, not measure our test stimulus directly (which would
> be useless regurgitation).
> >>>
> >>> Mo
> >>>
> >>>
> >>> In a recent email dialogue, my colleague Carsten Griwodz mentioned to
> me that "using PSNR for assessing the quality of adaptive video must be
> >>> "See the attached PDF, Appendix I, the tables in section I.3.1.
> Networking people use PSNR all the time, and it is absolutely bogus. The
> table shows that it is crap for stable video quality. For actively adapted
> video (e.g. not by dropping packets but by changing compression), it
> predicts subjective quality correctly in 30% of the cases. Papers that use
> it as an optimization goal cannot be trusted."
> >>>
> >>> While VQM is not the best known objective test (PEVQ is), VQM is the
> best known test with a free implementation and no strings attached.
> >>>
> >>> Cheers,
> >>> Michael
> >>
>
>

---

From: "Bill Ver Steeg (versteb)" <[email-masked]>
Date: Fri, 26 Apr 2013 14:42:48 +0000

There is a bit of subtlety to what we need to do here. I agree that the run-time algorithm that decides how to shape the traffic should not be calculating MOS/PSNR scores. There is simply too much complexity. However, when analyzing the effectiveness of two proposed shaping algorithms, it may be instructive to use traditional video quality methods to evaluate the proposals. These methods include the usual mix of golden-eye tests, end user tests, PSNR estimates, MOS scores, etc. There is a correct way to do video quality evaluations, and there is an incorrect way. 

What we want to avoid is using non-scientific video quality assessment methods as part of the evaluation. Putting two grainy canned videos up on an uncalibrated projector with a random aspect ratio and asking a large audience to say which one looks better should be strongly discouraged. Such exercises invariably end up in a goat rodeo. If we get to a point in the process in which video quality is a part of the evaluation, we need to be rigorous in the methods used.  

In the past, it has often instructive to a designer of an error control method (let's say Reed Solomon plus trellis coding on MMDS, or vanilla Reed Solomon on HFC) to run tests over some representative systems using a mix of flows and proposed FEC settings. The video quality is evaluated across a range of network configurations, and the correct FEC is determined for a given network. This is how we figured out Trellis coding (which spreads the data out in time a bit in order to protect from impulse noise) was the best FEC for MMDS. This is how we figured out what the Reed Solomon protection rate should be for an HFC network with a given power/SNR/cabling type. This is how OFDM was selected for DVB.

We may be able to get RMCAT done without using video quality as a factor, as there may be a clear protocol path. However, if we need to include video quality in the analysis - we can't allow people to cherry pick streams and just give their impressions. If anybody invokes video quality as a differentiator for a given algorithm, they should be required to describe the tests they used and the quantitative results. In the absence of real test results, the statement "the video looks better with algorithm X" should be banned.

Bvs

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Welzl
Sent: Friday, April 26, 2013 3:37 AM
To: Joerg Ott
Cc: [email-masked] WG; Mo Zanaty (mzanaty); Kevin Gross; Carsten Griwodz
Subject: Re: [rmcat] PSNR and draft-singh-rmcat-cc-eval-02

Hi,

Cheers,
Michael

On 26. apr. 2013, at 07:25, Joerg Ott wrote:

> I also agree that codecs matter.  But who says that an RMCAT 
> congestion control algorithm will be completely codec-independent?  If 
> RTCWEB is going to make a codec recommendation, it appears natural 
> that RMCAT would compare algorithms with this specific codec in mind.
> an evaluation framework.  Not to mandate or endorse a codec here, but 
> maybe using a wording that makes exactly your (and our) concerns 
> explicit.  We could do this until we come up with a better way so that 
> this basic idea does not get lost.
> 
>>> http://www.its.bldrdoc.gov/resources/video-quality-research/software
>>> .aspx Register and receive a download URL. You can access Matlab 
>>> code and Windows executables for batch processing (bvqm_*) and command line processing (cvqm_*). One of those is enough, the other tools aren't required.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 26 Apr 2013 14:22:27 -0700

Bill Ver Steeg (versteb) wrote:
> We may be able to get RMCAT done without using video quality as a
> factor, as there may be a clear protocol path. However, if we need to
> include video quality in the analysis - we can't allow people to
> cherry pick streams and just give their impressions. If anybody
> invokes video quality as a differentiator for a given algorithm, they
> should be required to describe the tests they used and the
> quantitative results. In the absence of real test results, the
> statement "the video looks better with algorithm X" should be banned.

Amen.

 From experience, I've often found that even rigorous tests may not be 
testing what you think they're testing. I've seen codec comparisons in 
the past where the results were actually dominated by the color 
conversion used, or because one path was correcting for a pixel aspect 
ratio near (but not exactly) 1.0 and the other wasn't... I expect 
network-layer feedback to encoder rate control is an interaction complex 
enough to present similar challenges (especially once you add 
resolution, framerate, FEC, etc. as parameters).

It may be useful to try to measure quality. Having done so, the 
important thing becomes figuring out what the results are really telling 
you, and whether that's useful. Even if you do everything right, that 
may not be obvious. Having tests that others can reproduce is the best 
chance you've got.