Subject: RE: [AVT] - Some - More Comments on draft-clark-avt-rtcpxr-video-02

From: "Mark Watson" <[email-masked]>
Date: Fri, 14 Jul 2006 11:59:31 -0700

All,

I'd like to add my support to Riccardo's suggestion of some additional
information which can better capture the character of the losses in
terms of patterns and correlations amongst losses. This is very
important for assessing the impact of different loss recovery
techniques, especially application layer FEC.

Generally, when FEC is applied for IPTV it is applied over short blocks
of packets - almost certainly blocks less than 500ms but preferably not
much less than about 50-100ms (otherwise the FEC overhead required
becomes rather high, whatever FEC code you use).

What is important for understanding how to select and engineer FEC
technologies for an IPTV network is the occurrence of multiple losses
within these blocks.

What would be extremely useful, therefore would be a metric which
counted events in which multiple losses were observed within time
periods of this order and, importantly, separate counts for events with
n packet losses within a block for various values of n: 2, 3, 4, ...
(n=1 could be derived from the other information and the overall loss
rate, or could be provided explicitly). i.e. the client would report the
number of (say) 200ms periods within which it observed 2 packet losses
as well as the number of 200ms periods in which it observed 3 packet
losses etc.

Once you know how often these different events occur it's a question of
comparing this with your end-user quality ambitions in order to
determine whether protection against such events is required or whether
they can be ignored as too rare to be worth considering.

I would suggest that the block size for the endpoint to measure against
(e.g. 100ms, 200ms etc.) is something which could be configured at the
endpoint and included within the report.

If there is interest in providing for this kind of report then I'd be
happy to draft the message coding for that.

Best regards,

Mark Watson

-----Original Message-----
From: Fiandra Riccardo [mailto:[email-masked]] 
Sent: Friday, July 14, 2006 10:59 AM
To: Alan Clark; [email-masked]
Subject: R: [AVT] - Some - More Comments on
draft-clark-avt-rtcpxr-video-02

Gentlmen,
I have been following a little the discussion on this video metrics
reporting, I would like to make some comments hoping these would be of
some help for the work.
First point is that this is very useful stuff and the use of STB or any
end-point reports is, in my opinion, one of the most powerful way to
measure network and system delivery chain end-to-end performance. As I
think known to all of you, using correlation methods, STB reports can be
processed to indentify source of impairment (access, backbone, head-end,
a certain link, etc) and assess service performance on a continuos way.
 
Here are my comments:
 
-IP Loss metrics.
I think that the number LOST packets and Expected packets should be two
known values.
Beyod that, some other information are very useful:
 - Number of loss events with the definition of a loss event as in 3557
(or similar)
 - Number of loss events that violates a minimum defined distance (I
think this also is a concept highlighted in 3557)
 -Optionally minimum and maximum loss periods within the report interval
would help.
 
Reason for this is that suppose you have 4 (or any little number of )
lost packets in one minute of a video stream (that is roughly 20k
packets for 4 Mbps stream) and that you find this number (4) often in
your reports. You are very interested in knowing how spread are these
lost packets (period an distance), for example to extimate the loss of
quality due to the IP network or  to design and dimension a possible FEC
system for video.
I think that the parameters I listed let estimate quite effectively
these valuable information leaving the implemention simple. If these
information are already available by means of currently defined
parameters then it would be OK as it is. I am not very familiar with Gap
and Bursts, but it looks these parameters are very useful and needed, on
the end: would it be possible to add the parameters above to the
currently defined one?
 
- Pre FEC - Post FEC
Does this refer to a FEC scheme at above the IP layer? Generally I would
define an IP/RTP lost packets and  then the information on how much was
recovered by means of a FEC (numeber of FEC corrected packets). Since
still FEC for IPTV it is not standard nor widely available I would make
it as optional.
 
- within the ITU-T J.241 we stated some of the principles included in
this draft text, that is measurement reporting by the STB for IP
transport, frame rate and buffer level, are reccomended way of
maeasuring IPTV service performance. Even though we did not go into big
detail on what and how to report, it definitevely help into stressing
that passive in-service feedback-reporting based measurement are greatly
helpful for IPTV.
 
I hope this helps somehow,
 
Rgds,
 
Riccardo

________________________________

Da: Alan Clark [mailto:[email-masked]]
Inviato: ven 14/07/2006 13.26
A: 'Fardid, Reza'; [email-masked]
Oggetto: RE: [AVT] More Comments on draft-clark-avt-rtcpxr-video-02

Reza

Thanks for the comments - some of which would apply equally to the RTCP
HR
draft.

> Sec 3.1:
> 
> RTP and MPEG Metrics:
>  -Average Jitter is not a useful statistic for real-time
> management of Voice over IP sessions compared
>    to Jitter Percentile.  The IPPM WG is also de-emphasizing
> its use. Has its significance been proven in
>    the context of Video over IP to include it in these
> metrics sub-blocks?

We intend that RTCP XR Video be aligned with RTCP HR in the area of
jitter
metrics (where it makes sense).

Also - MAPDV is a very useful metric for VoIP applications, it
correlates
well with jitter buffer behavior (e.g. is a good predictor of optimum
jitter
buffer size)

> Video/Audio Metrics:
> -Overwhelming network management with too many Video Service
> Quality metrics is not likely to be   productive. In
> particular, when these metrics are expected to be correlated
> to one another and to IP/RTP metrics. A single composite A/V
> quality metric and a control plane quality metric will be
> sufficient for most practical purposes.

These metrics are in discussion in other standards groups that are
heavily
service provider oriented - if anything some of these groups are
generating
even more metrics :-(

> Sec 3.2:
> 
> It may useful to have an indication of inactive media based
> on a configurable timer to suppress sending the sub-blocks.
> For instance, Block Type = mmm+10 can indicate Null metrics.

Good idea - we can accomodate this or some equivalent

> Sec 3.3:
> 
> Is the Number of IP Packets Expected estimated from the last
> M sessions, as an average or smoothed average?  I wonder
> about the usefulness of this estimate.  Can you please clarify?

This is as per RTCP HR - and is simply to allow you to put some metrics
in
context.  I can see a potential issue with multiplexed MPEG streams over
a
single IP path .. needs further clarification

Regards

Alan

> Regards,
> Reza Fardid
>
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/avt
>

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Hedayat, Kaynam" <[email-masked]>
Date: Wed, 19 Jul 2006 10:01:39 -0400

Mark,

I fully support your recommendation. You should take a look at RFC3357
section 6.1. It is taking on the same concept based on number of packets
vs. time (what you are suggesting). My suggestion is to take on the
metrics from 3357, if need be extend them to IPTV (e.g. time vs.
packet), and include them in XR. I'd be happy to help on drafting the
approach.

Regards,
Kaynam

> From: Mark Watson [mailto:[email-masked]] 
> Sent: Friday, July 14, 2006 3:00 PM
> To: Fiandra Riccardo; Alan Clark; [email-masked]
> Subject: RE: [AVT] - Some - More Comments on 
> I'd like to add my support to Riccardo's suggestion of some 
> additional information which can better capture the character 
> of the losses in terms of patterns and correlations amongst 
> losses. This is very important for assessing the impact of 
> different loss recovery techniques, especially application layer FEC.
> 
> Generally, when FEC is applied for IPTV it is applied over 
> short blocks of packets - almost certainly blocks less than 
> 500ms but preferably not much less than about 50-100ms 
> (otherwise the FEC overhead required becomes rather high, 
> whatever FEC code you use).
> 
> What is important for understanding how to select and 
> engineer FEC technologies for an IPTV network is the 
> occurrence of multiple losses within these blocks.
> 
> What would be extremely useful, therefore would be a metric 
> which counted events in which multiple losses were observed 
> within time periods of this order and, importantly, separate 
> counts for events with n packet losses within a block for 
> various values of n: 2, 3, 4, ...
> (n=1 could be derived from the other information and the 
> overall loss rate, or could be provided explicitly). i.e. the 
> client would report the number of (say) 200ms periods within 
> which it observed 2 packet losses as well as the number of 
> 200ms periods in which it observed 3 packet losses etc.
> 
> Once you know how often these different events occur it's a 
> question of comparing this with your end-user quality 
> ambitions in order to determine whether protection against 
> such events is required or whether they can be ignored as too 
> rare to be worth considering.
> 
> I would suggest that the block size for the endpoint to 
> measure against (e.g. 100ms, 200ms etc.) is something which 
> could be configured at the endpoint and included within the report.
> 
> If there is interest in providing for this kind of report 
> then I'd be happy to draft the message coding for that.
> 
> Best regards,
> 
> Mark Watson
> 
> I have been following a little the discussion on this video 
> metrics reporting, I would like to make some comments hoping 
> these would be of some help for the work.
> First point is that this is very useful stuff and the use of 
> STB or any end-point reports is, in my opinion, one of the 
> most powerful way to measure network and system delivery 
> chain end-to-end performance. As I think known to all of you, 
> using correlation methods, STB reports can be processed to 
> indentify source of impairment (access, backbone, head-end, a 
> certain link, etc) and assess service performance on a continuos way.
>  
> Here are my comments:
>  
> -IP Loss metrics.
> I think that the number LOST packets and Expected packets 
> should be two known values.
>  - Number of loss events with the definition of a loss event 
> as in 3557 (or similar)
>  - Number of loss events that violates a minimum defined 
> distance (I think this also is a concept highlighted in 3557) 
>  -Optionally minimum and maximum loss periods within the 
> report interval would help.
>  
> Reason for this is that suppose you have 4 (or any little 
> number of ) lost packets in one minute of a video stream 
> (that is roughly 20k packets for 4 Mbps stream) and that you 
> find this number (4) often in your reports. You are very 
> interested in knowing how spread are these lost packets 
> (period an distance), for example to extimate the loss of 
> quality due to the IP network or  to design and dimension a 
> possible FEC system for video.
> I think that the parameters I listed let estimate quite 
> effectively these valuable information leaving the 
> implemention simple. If these information are already 
> available by means of currently defined parameters then it 
> would be OK as it is. I am not very familiar with Gap and 
> Bursts, but it looks these parameters are very useful and 
> needed, on the end: would it be possible to add the 
> parameters above to the currently defined one?
>  
> - Pre FEC - Post FEC
> Does this refer to a FEC scheme at above the IP layer? 
> Generally I would define an IP/RTP lost packets and  then the 
> information on how much was recovered by means of a FEC 
> (numeber of FEC corrected packets). Since still FEC for IPTV 
> it is not standard nor widely available I would make it as optional.
>  
> - within the ITU-T J.241 we stated some of the principles 
> included in this draft text, that is measurement reporting by 
> the STB for IP transport, frame rate and buffer level, are 
> reccomended way of maeasuring IPTV service performance. Even 
> though we did not go into big detail on what and how to 
> report, it definitevely help into stressing that passive 
> in-service feedback-reporting based measurement are greatly 
> helpful for IPTV.
>  
> Thanks for the comments - some of which would apply equally 
> to the RTCP HR draft.
> 
> > Sec 3.1:
> > 
> > RTP and MPEG Metrics:
> >    to Jitter Percentile.  The IPPM WG is also 
> de-emphasizing its use. 
> > Has its significance been proven in
> >    the context of Video over IP to include it in these metrics 
> > sub-blocks?
> 
> We intend that RTCP XR Video be aligned with RTCP HR in the 
> area of jitter metrics (where it makes sense).
> 
> Also - MAPDV is a very useful metric for VoIP applications, 
> it correlates well with jitter buffer behavior (e.g. is a 
> good predictor of optimum jitter buffer size)
> 
> > Video/Audio Metrics:
> > particular, when these metrics are expected to be correlated to one 
> > another and to IP/RTP metrics. A single composite A/V 
> quality metric 
> > and a control plane quality metric will be sufficient for most 
> > practical purposes.
> 
> These metrics are in discussion in other standards groups 
> that are heavily service provider oriented - if anything some 
> of these groups are generating even more metrics :-(
> 
> 
> > Sec 3.2:
> > 
> > It may useful to have an indication of inactive media based on a 
> > configurable timer to suppress sending the sub-blocks.
> > Is the Number of IP Packets Expected estimated from the last M 
> > sessions, as an average or smoothed average?  I wonder about the 
> > usefulness of this estimate.  Can you please clarify?
> 
> This is as per RTCP HR - and is simply to allow you to put 
> some metrics in context.  I can see a potential issue with 
> multiplexed MPEG streams over a single IP path .. needs 
> further clarification
> 
> Regards
> 
> Alan
> 
> 
> > Regards,
> > Reza Fardid
> >