Subject: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt

From: Al Morton <[email-masked]>
Date: Wed, 01 Aug 2012 17:01:00 -0400

Regarding:
http://tools.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-02.txt

>    Calculation Algorithm (CALg):3 bits
>
>          000 - ITU-T P.564 Compliant Algorithm [P.564] (Voice)
>          001 - G.107 [G.107] (Voice)
>          010 - ETSI TS 101 329-5 Annex E [ ETSI] (Voice)
>          011 - ITU-T P.NAMS [P.NAMS] (Multimedia)
>          100 - ITU-T P.NBAMS [P.NBAMS] (Multimedia)
>          101~111 - Reserved for future extension.
>
>       G.107 and P.564 and ETSI TS101 329-5 specify three Calculation
>       algorithms or MoS algorithms that are used to estimate speech
>       quality or conversation quality.  P.NAMS and P.NBAMS specify two
>       MoS algorithms that are used to estimate multimedia quality
>       including video quality, audio quality and audio-video quality.

Specifying the standard for the calculation is a good start,
but these standards all have options and input parameters
that must also be specified in order to know what the MOS means.

For example, we need to know what codec was used, because G.726
carries an equipment Impairment factor that limits the upper
bound on MOS, while G.711 does not.

Some of the input parameters will be assumed (e.g., at default values)
while others will be measured, and the distinction between measured
and assumed parameters should also be made.

As I mentioned, metrics with many optional parameters are somewhat
difficult to identify in a simple way, as we found with the IPPM
Metrics Registry (which we withdrew when we found it was insufficient
to describe the measurement results in an exact way).

Colin Perkins suggested the possibility a registry for the calculation alg.
A registry could provide a single value index to a complicated set of
input parameters and other assumptions, and this might work for you.

hope this helps,
Al

---

From: Qin Wu <[email-masked]>
Date: Thu, 16 Aug 2012 10:33:44 +0800

Hi, Al:
Thank for your general comments, I take some time to look at the standards for calculation again. 
please see my replies inline.

Regards!
-Qin
----- Original Message ----- 
From: "Al Morton" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, August 02, 2012 5:01 AM
Subject: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt

> bound on MOS, while G.711 does not.

[Qin]: I agree MoS value depends on the codec that is in use. That's why, in the current draft,
We use 7-bit payload Type in the metric block to signal what codec or payload format is
 in use for reporting interval. 
Please see the definition of Payload Type (PT) in the section 3.2.1.

> and assumed parameters should also be made.

[Qin]: 
The input parameters may come from Delay metric block, Discard metric block,
Burst Gap metrics block, JB metrics block, loss concealment metric block, concelament seconds
metrics block, these metrics blocks are defined in other XR Block drafts by XRBLOCK WG .

In QoE metric block, 4 or 5 pameters are defined, i.e.,
a.Segment Type
b.MoS Type
c.MoS algorithm
d. MoS value
In these parameters, a~c can be counted as default values while d is counted as measured value.
The other default values rely on specific computation model defined in each standard, e.g., 
P.564, G.104, P.NAMS.
Assume each standard only provide one algorithm or one model, we can use  the standards name and payload type(i.e.,codec) 
to identify each MoS algorithm used or each MoS model used.

> input parameters and other assumptions, and this might work for you.

[Qin]: Yes,depends on the input parameters and option we got, we can calculate
the different MoS values however the computation model is same and will not be 
affected by the input parameters and options.
So maybe rather than registering MoS algorithm, we should register assessment model or computation model.
We can use computation model defined in the standard and payload type as single value index. 

> hope this helps,
> Al
> 
> 
> _______________________________________________
> xrblock mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xrblock
>

---

From: Alan Clark <[email-masked]>
Date: Thu, 16 Aug 2012 10:58:39 -0400

Some further work is definitely needed on this draft with regard to the MOS
terminology. The various calculation methods for MOS either already do, or
soon will, provide MOS values for narrowband, wideband, super wideband and
fullband codecs; these MOS values occupy the same range and hence the QoE
block needs to indicate what the MOS reference is.

In addition there are national standards (e.g. Japan) that use a different
MOS scaling.

Alan

On 8/15/12 10:33 PM, "Qin Wu" <[email-masked]> wrote:

> Assume each standard only provide one algorithm or one model, we can use  the
> standards name and payload type(i.e.,codec)

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 27 Aug 2012 16:21:25 +0200

Hi Alan,

In the meeting in Vancouver a proposal was made to create a IANA
registry for the calculation algorithms. Assuming that we go this way
(starting from Al's initial list or something derived from it) do you
believe that we need a second registry for the codec type? 

Dan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Alan Clark
> Sent: Thursday, August 16, 2012 5:59 PM
> To: Qin Wu; [email-masked]; Al Morton
> Subject: Re: [xrblock] comment on
draft-ietf-xrblock-rtcp-xr-qoe-02.txt
> 
> 
> Some further work is definitely needed on this draft with regard to
the
> MOS terminology. The various calculation methods for MOS either
already
> do, or soon will, provide MOS values for narrowband, wideband, super
> wideband and fullband codecs; these MOS values occupy the same range
and
> hence the QoE block needs to indicate what the MOS reference is.
> 
> In addition there are national standards (e.g. Japan) that use a
> different MOS scaling.
> 
> Alan
> 
> 
> 
> >> Specifying the standard for the calculation is a good start, but
> >> these standards all have options and input parameters that must
also
> >> be specified in order to know what the MOS means.
> >
> >> carries an equipment Impairment factor that limits the upper bound
on
> >> MOS, while G.711 does not.
> >
> > [Qin]: I agree MoS value depends on the codec that is in use. That's
> > why, in the current draft, We use 7-bit payload Type in the metric
> > block to signal what codec or payload format is  in use for
reporting
> > interval.
> >> Some of the input parameters will be assumed (e.g., at default
> >> values) while others will be measured, and the distinction between
> >> measured and assumed parameters should also be made.
> >
> > [Qin]:
> > The input parameters may come from Delay metric block, Discard
metric
> > block, Burst Gap metrics block, JB metrics block, loss concealment
> > metric block, concelament seconds metrics block, these metrics
blocks
> > are defined in other XR Block drafts by XRBLOCK WG .
> >
> > In QoE metric block, 4 or 5 pameters are defined, i.e., a.Segment
Type
> > b.MoS Type c.MoS algorithm d. MoS value In these parameters, a~c can
> > be counted as default values while d is counted as measured value.
> > The other default values rely on specific computation model defined
in
> > each standard, e.g., P.564, G.104, P.NAMS.
> > Assume each standard only provide one algorithm or one model, we can
> > use  the standards name and payload type(i.e.,codec) to identify
each
> > MoS algorithm used or each MoS model used.
> >
> > [Qin]: Yes,depends on the input parameters and option we got, we can
> > calculate the different MoS values however the computation model is
> > same and will not be affected by the input parameters and options.
> > We can use computation model defined in the standard and payload
type
> > as single value index.

---

From: Alan Clark <[email-masked]>
Date: Mon, 27 Aug 2012 13:03:44 -0400

Hi Dan

A registry would be needed for the MOS definition - not for the codec type.
The MOS definitions would need to include both the reference and the scale,
for example:

1. Reference:
- Narrowband (8kHz) speech
- Wideband (16kHz) speech
-....

Note that you can represent a Narrowband codec on both Narrowband and
Wideband scales but a Wideband codec has to be represented on a Wideband
scale.

Current thought within the industry is to map Superwideband and Fullband
codecs onto the Wideband scale however if this changes then there may be
additional references needed

For video - the issue of MOS reference is still a discussion item.

2. Scaling:
This is a slightly more complicated issue. G.107 was developed in the mid
1990's based on subjective test data from the 1980's and early 90's; a
"good" G.711 call would have a MOS of 4.45 as calculated by G.107.  If you
look at subjective test data from the later 1990's and 2000's you would find
that G.711 MOS scores tend to be around 4.1-4.2.  This may be due to
subjective test reference condition changes or possibly to listeners being
more used to digital audio and hence more critical.  Also the Japanese
national standard uses a lower MOS scaling that would give a "good" G.711
score of 3.8.
Telchemy uses the terms "ITU Scaled" to mean the scaling used for G.107,
"ACR Scaled" to mean - consistent with typical subjective test data, and
"Japanese TTC Scaled" to mean - consistent with the Japanese national
standard.  All of our customers seem to use the ACR Scaled range as this
aligns with typical quoted MOS scores for codecs. I'm not aware of ITU-T
SG12 having worked on this specific topic of scaling, although they are of
course very aware of the subject matter.

It would of course be possible to include a reference MOS in the QoE report
as an alternative to a more complex registry.

Best Regards

Alan

---

From: Qin Wu <[email-masked]>
Date: Tue, 28 Aug 2012 10:33:35 +0800

Hi, Alan:
Given the popularity of the Japan national standards (i.e.,TTC 201.01) in Japan, 
we can add TTC 201.01 based algorithm as one new calculation algorithm  in the
QoE Metric Block.
Currently we still allow support additional 5 new calculation algorithms using 3bit CAlg.

Regarding MoS Reference, are you suggesting we should differentiate low bit rate from high bit rate
or we should differeniate narrowband from wideband?
It looks to me narrowband and wideband only can be used with audio stream and don't apply to video?

Another question is I am not familiar with the terminology "MoS reference".
Would you like to point me a reference to explain what "MoS Reference" is?
Thanks!

Regards!
-Qin

---

From: Qin Wu <[email-masked]>
Date: Tue, 28 Aug 2012 10:52:49 +0800

Hi,
Does "the reference" you refer to for the MoS definitions apply to video?
Are there any standards we can refer to for the concept of "reference" 
and "scaling"?

Regards!
-Qin

---

From: Alan Clark <[email-masked]>
Date: Tue, 28 Aug 2012 09:17:33 -0400

Hi Qin

The idea of "reference" is incorporated into subjective test methodologies
and into specific subject test experiments. For example when a codec is
evaluated it is typically compared against other codecs with the same sample
rate (i.e. a narrowband codec is compared against other narrowband codecs) -
this means that the resulting MOS scores have a narrowband reference.

I am not aware of a specific standard related to "reference" and "scaling"
other than the definition of the subjective test methodologies that define
anchor conditions (e.g. MNRU's).

The same general idea applies to video - video on a mobile handset would
typically be evaluated against other mobile handsets - so reference/scaling
may apply to image resolution, screen size or both. Telchemy's technology
uses the terms Absolute and Relative MOS-V, we have described this in
contributions to ITU-T SG12 however nothing has been incorporated into a
standard as yet.

Alan

On 8/27/12 10:52 PM, "Qin Wu" <[email-masked]> wrote:

> To: "Dan (Dan)" <[email-masked]>; "Qin Wu" <[email-masked]>;
> <[email-masked]>; "Al Morton" <[email-masked]>

---

From: Alan Clark <[email-masked]>
Date: Tue, 28 Aug 2012 09:22:15 -0400

Qin

Wideband codecs are often used on conference phones - also AMR-WB is the
default codec for 4G LTE.

We have some application notes and standards contributions related to this
on our web site - I'll forward a copy of relevant documents to you "off
list"

Alan

On 8/27/12 10:33 PM, "Qin Wu" <[email-masked]> wrote:

>>> Thank for your general comments, I take some time to look at the standards
>>> for
>>> calculation again.
>>> We use 7-bit payload Type in the metric block to signal what codec or
>>> payload
>>> format is
>>> calculate
>>> So maybe rather than registering MoS algorithm, we should register
>>> assessment
>>> model or computation model.