Subject: RE: [AVT] - Some - More Comments on draft-clark-avt-rtcpxr-video-02

From: "Mark Watson" <[email-masked]>
Date: Wed, 19 Jul 2006 09:43:58 -0700

Kayam,

Yes, the P-One-Way-Loss-Noticable-Rate in RFC3357 is similar to one of
the counts I was suggesting: specifically the count of the events where
there are exactly 2 packet losses within the period of interest.

However, to be interesting for IPTV we would also like to know the
number of events in which there were 3, 4, 5, etc. losses within the
period of interest. So, it would be better to have several counters.

Given this, the term 'noticable' is probably not appropriate - since
whether the losses are noticeable depends on lots of things.

The other question is whether the measurement period should be defined
as a number of packets or a period of time. When applying FEC protection
to a video stream then an FEC buffer is needed at the receiver and the
receiver must delay the playout by the duration of this buffer (in
time). Subsequently, FEC protection cannot be applied to blocks larger
than this (int time), since that would either starve the player or mean
the FEC data would arrive too late at the decoder. So, for FEC it's a
time period which is most interesting.

Of course, for constant rate streams, then a number of packets and a
time period are roughly equivalent, so it is only for variable rate
streams where this becomes interesting.

Finally, I'd just like to take the opportunity to point out a
(non-technical) problem which I have observed with RFC3357: Whilst the
loss period and loss distance *streams* and other metrics defined there
completely describe the losses that were seen in the stream, I have
often seen people consider 'loss period' and 'loss distance' as just two
numbers (presumably averages) which are used to characterize the whole
stream. Of course these two numbers give very limited information about
the loss patterns observed and are certainly not sufficient for
understanding the QoE that might be obtained for a video stream or what
kind of loss mitigation is required.

Best regards,

Mark

-----Original Message-----
From: Hedayat, Kaynam [mailto:[email-masked]] 
Sent: Wednesday, July 19, 2006 7:02 AM
To: Mark Watson; Fiandra Riccardo; Alan Clark; [email-masked]
Subject: RE: [AVT] - Some - More Comments on
draft-clark-avt-rtcpxr-video-02

Mark,

I fully support your recommendation. You should take a look at RFC3357
section 6.1. It is taking on the same concept based on number of packets
vs. time (what you are suggesting). My suggestion is to take on the
metrics from 3357, if need be extend them to IPTV (e.g. time vs.
packet), and include them in XR. I'd be happy to help on drafting the
approach.

Regards,
Kaynam

> -----Original Message-----
> From: Mark Watson [mailto:[email-masked]] 
> Sent: Friday, July 14, 2006 3:00 PM
> To: Fiandra Riccardo; Alan Clark; [email-masked]
> Subject: RE: [AVT] - Some - More Comments on 
> draft-clark-avt-rtcpxr-video-02
> 
> All,
> 
> I'd like to add my support to Riccardo's suggestion of some 
> additional information which can better capture the character 
> of the losses in terms of patterns and correlations amongst 
> losses. This is very important for assessing the impact of 
> different loss recovery techniques, especially application layer FEC.
> 
> Generally, when FEC is applied for IPTV it is applied over 
> short blocks of packets - almost certainly blocks less than 
> 500ms but preferably not much less than about 50-100ms 
> (otherwise the FEC overhead required becomes rather high, 
> whatever FEC code you use).
> 
> What is important for understanding how to select and 
> engineer FEC technologies for an IPTV network is the 
> occurrence of multiple losses within these blocks.
> 
> What would be extremely useful, therefore would be a metric 
> which counted events in which multiple losses were observed 
> within time periods of this order and, importantly, separate 
> counts for events with n packet losses within a block for 
> various values of n: 2, 3, 4, ...
> (n=1 could be derived from the other information and the 
> overall loss rate, or could be provided explicitly). i.e. the 
> client would report the number of (say) 200ms periods within 
> which it observed 2 packet losses as well as the number of 
> 200ms periods in which it observed 3 packet losses etc.
> 
> Once you know how often these different events occur it's a 
> question of comparing this with your end-user quality 
> ambitions in order to determine whether protection against 
> such events is required or whether they can be ignored as too 
> rare to be worth considering.
> 
> I would suggest that the block size for the endpoint to 
> measure against (e.g. 100ms, 200ms etc.) is something which 
> could be configured at the endpoint and included within the report.
> 
> If there is interest in providing for this kind of report 
> then I'd be happy to draft the message coding for that.
> 
> Best regards,
> 
> Mark Watson
> 
> -----Original Message-----
> From: Fiandra Riccardo [mailto:[email-masked]]
> Sent: Friday, July 14, 2006 10:59 AM
> To: Alan Clark; [email-masked]
> Subject: R: [AVT] - Some - More Comments on
> draft-clark-avt-rtcpxr-video-02
> 
> Gentlmen,
> I have been following a little the discussion on this video 
> metrics reporting, I would like to make some comments hoping 
> these would be of some help for the work.
> First point is that this is very useful stuff and the use of 
> STB or any end-point reports is, in my opinion, one of the 
> most powerful way to measure network and system delivery 
> chain end-to-end performance. As I think known to all of you, 
> using correlation methods, STB reports can be processed to 
> indentify source of impairment (access, backbone, head-end, a 
> certain link, etc) and assess service performance on a continuos way.
>  
> Here are my comments:
>  
> -IP Loss metrics.
> I think that the number LOST packets and Expected packets 
> should be two known values.
> Beyod that, some other information are very useful:
>  - Number of loss events with the definition of a loss event 
> as in 3557 (or similar)
>  - Number of loss events that violates a minimum defined 
> distance (I think this also is a concept highlighted in 3557) 
>  -Optionally minimum and maximum loss periods within the 
> report interval would help.
>  
> Reason for this is that suppose you have 4 (or any little 
> number of ) lost packets in one minute of a video stream 
> (that is roughly 20k packets for 4 Mbps stream) and that you 
> find this number (4) often in your reports. You are very 
> interested in knowing how spread are these lost packets 
> (period an distance), for example to extimate the loss of 
> quality due to the IP network or  to design and dimension a 
> possible FEC system for video.
> I think that the parameters I listed let estimate quite 
> effectively these valuable information leaving the 
> implemention simple. If these information are already 
> available by means of currently defined parameters then it 
> would be OK as it is. I am not very familiar with Gap and 
> Bursts, but it looks these parameters are very useful and 
> needed, on the end: would it be possible to add the 
> parameters above to the currently defined one?
>  
> - Pre FEC - Post FEC
> Does this refer to a FEC scheme at above the IP layer? 
> Generally I would define an IP/RTP lost packets and  then the 
> information on how much was recovered by means of a FEC 
> (numeber of FEC corrected packets). Since still FEC for IPTV 
> it is not standard nor widely available I would make it as optional.
>  
> - within the ITU-T J.241 we stated some of the principles 
> included in this draft text, that is measurement reporting by 
> the STB for IP transport, frame rate and buffer level, are 
> reccomended way of maeasuring IPTV service performance. Even 
> though we did not go into big detail on what and how to 
> report, it definitevely help into stressing that passive 
> in-service feedback-reporting based measurement are greatly 
> helpful for IPTV.
>  
> I hope this helps somehow,
>  
> Rgds,
>  
> Riccardo
> 
> 
> ________________________________
> 
> Da: Alan Clark [mailto:[email-masked]]
> Inviato: ven 14/07/2006 13.26
> A: 'Fardid, Reza'; [email-masked]
> Oggetto: RE: [AVT] More Comments on draft-clark-avt-rtcpxr-video-02
> 
> 
> 
> Reza
> 
> Thanks for the comments - some of which would apply equally 
> to the RTCP HR draft.
> 
> > Sec 3.1:
> > 
> > RTP and MPEG Metrics:
> >  -Average Jitter is not a useful statistic for real-time 
> management of 
> > Voice over IP sessions compared
> >    to Jitter Percentile.  The IPPM WG is also 
> de-emphasizing its use. 
> > Has its significance been proven in
> >    the context of Video over IP to include it in these metrics 
> > sub-blocks?
> 
> We intend that RTCP XR Video be aligned with RTCP HR in the 
> area of jitter metrics (where it makes sense).
> 
> Also - MAPDV is a very useful metric for VoIP applications, 
> it correlates well with jitter buffer behavior (e.g. is a 
> good predictor of optimum jitter buffer size)
> 
> > Video/Audio Metrics:
> > -Overwhelming network management with too many Video Service
> > Quality metrics is not likely to be   productive. In
> > particular, when these metrics are expected to be correlated to one 
> > another and to IP/RTP metrics. A single composite A/V 
> quality metric 
> > and a control plane quality metric will be sufficient for most 
> > practical purposes.
> 
> These metrics are in discussion in other standards groups 
> that are heavily service provider oriented - if anything some 
> of these groups are generating even more metrics :-(
> 
> 
> > Sec 3.2:
> > 
> > It may useful to have an indication of inactive media based on a 
> > configurable timer to suppress sending the sub-blocks.
> > For instance, Block Type = mmm+10 can indicate Null metrics.
> 
> Good idea - we can accomodate this or some equivalent
> 
> > Sec 3.3:
> > 
> > Is the Number of IP Packets Expected estimated from the last M 
> > sessions, as an average or smoothed average?  I wonder about the 
> > usefulness of this estimate.  Can you please clarify?
> 
> This is as per RTCP HR - and is simply to allow you to put 
> some metrics in context.  I can see a potential issue with 
> multiplexed MPEG streams over a single IP path .. needs 
> further clarification
> 
> Regards
> 
> Alan
> 
> 
> > Regards,
> > Reza Fardid
> >
> > _______________________________________________
> > Audio/Video Transport Working Group
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/avt
> >
> 
> 
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/avt
> 
> 
> 
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/avt
> 
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/avt
> 

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Alan Clark" <[email-masked]>
Date: Mon, 24 Jul 2006 18:57:43 -0400

Mark

The loss model used in RFC3611, which is also contained in the video metrics
draft, uses a sparse burst model that may also be a useful model to consider
in the context of FEC.  This model divides the session into burst and gaps -
where a gap is defined as a period of time during which any lost or
discarded packet must be preceded and followed by at least Gmin received
packets (where Gmin is typically 16).  This essentially means that during
burst periods lost packets are separated by less than Gmin received packets
- and the model measures the length and average loss density of burst and
gap periods.  
 
If on some stream the following parameters were measured:

Gap length 10,000 ms
Gap density 0.01 (i.e. 1%)
Burst length 1,200 ms
Burst density 0.23 (i.e. 23%)

Then you would know that on average, packet loss was occurring in bursts of
23% loss rate for periods of (on average) 1.2 seconds, separated by gaps of
10 seconds during which around 1% of packets were lost.  It should be fairly
easy to relate this to FEC performance.

Regards

Alan

> Sent: Wednesday, July 19, 2006 12:44 PM
> To: Hedayat, Kaynam; Fiandra Riccardo; Alan Clark; [email-masked]
> Yes, the P-One-Way-Loss-Noticable-Rate in RFC3357 is similar 
> to one of the counts I was suggesting: specifically the count 
> of the events where there are exactly 2 packet losses within 
> the period of interest.
> 
> However, to be interesting for IPTV we would also like to 
> know the number of events in which there were 3, 4, 5, etc. 
> losses within the period of interest. So, it would be better 
> to have several counters.
> 
> Given this, the term 'noticable' is probably not appropriate 
> - since whether the losses are noticeable depends on lots of things.
> 
> The other question is whether the measurement period should 
> be defined as a number of packets or a period of time. When 
> applying FEC protection to a video stream then an FEC buffer 
> is needed at the receiver and the receiver must delay the 
> playout by the duration of this buffer (in time). 
> Subsequently, FEC protection cannot be applied to blocks 
> larger than this (int time), since that would either starve 
> the player or mean the FEC data would arrive too late at the 
> decoder. So, for FEC it's a time period which is most interesting.
> 
> Of course, for constant rate streams, then a number of 
> packets and a time period are roughly equivalent, so it is 
> only for variable rate streams where this becomes interesting.
> 
> (non-technical) problem which I have observed with RFC3357: 
> Whilst the loss period and loss distance *streams* and other 
> metrics defined there completely describe the losses that 
> were seen in the stream, I have often seen people consider 
> 'loss period' and 'loss distance' as just two numbers 
> (presumably averages) which are used to characterize the 
> whole stream. Of course these two numbers give very limited 
> information about the loss patterns observed and are 
> certainly not sufficient for understanding the QoE that might 
> be obtained for a video stream or what kind of loss 
> mitigation is required.
> 
> Best regards,
> 
> Mark
>