Subject: RE: [AVT] draft-clark-avt-rtcpxr-video-02

From: "Hedayat, Kaynam" <[email-masked]>
Date: Wed, 12 Jul 2006 15:15:00 -0400

Alan,
 
The argument is for not tying the model to the metrics reported by XR or
at a minimum have an option that will allow multiple models. RFC3357
provides metrics that provide packet loss behavior in terms of loss
periods and loss distance, both applicable to real-time applications. 
 
I disagree with your point that 3357 does not consider packet discard.
For the application, loss is loss, regardless of its source (network or
endpoint). Both kinds of loss can be described in terms defined by 3357.
 
A lot of good work and time has been put into RFC3357 and its metrics
are widely used in the field today, especially for Video over IP
applications (IPTV, VoD, etc.). I can't imagine any argument against
using these metrics.
 
Regards,
Kaynam

________________________________

	From: Alan Clark [mailto:[email-masked]] 
	Sent: Wednesday, July 12, 2006 2:29 PM
	To: Hedayat, Kaynam; [email-masked]
	Subject: RE: [AVT] draft-clark-avt-rtcpxr-video-02

	Kaynam
	 
	You are quite incorrect.
	 
	(i) The burst and gap metrics were originally defined in RFC3611
(RTCP XR) and, as they are extremely helpful in understanding the impact
of IP impairments on VoIP, were then adopted in eight other standards
related to VoIP performance.    Hence the original work was done in AVT
and then adopted across the industry
	 
	(ii) The concepts used in RFC3611 are quite different to those
in RFC3357 for good reason.  We analyzed the distribution of lost
packets on many traces and found that "burstiness" tended to be
exhibited as periods of high loss and NOT as periods of consecutive
loss. We also found that periods of high PDV typically coincide with
periods of high loss, and hence see merit in measuring the impact of
loss and PDV together (i.e. loss/discard distribution).
	 - The loss period you refer to from RFC3357 defines only
consecutive loss and not the sparse bursts that typically occur
(particularly at the network edge), whereas RFC3611 describes sparse
bursts
	 - RFC3357 only applies to lost packets, and not to packets
discarded due to late arrival, whereas the model from RFC3611 applies to
the combined loss/discard distribution
	 
	A good summary of the rationale behind the burst/gap model is
http://www.telchemy.com/reference/ITUSG12_PacketLossDistribution.pdf

	I totally disagree with your assertion that RFC3357 metrics are
better defined and more accurately represent packet loss behavior - you
are entitled to hold any opinion you like however if you want to express
an opinion should explain the justification for it.  Any of these
measurements essentially comprise parameters of some assumed model that
describes the behavior of the system being measured.  You should try and
ensure that the model is relevant with regard to the aspect of the
system behavior you are interested in.  You could argue that the assumed
model is or is not a good fit for the aspect of system behavior that is
of interest, or that the method for parameter estimation is more or less
accurate.
	 
	Alan

________________________________

		From: Hedayat, Kaynam [mailto:[email-masked]] 
		Sent: Tuesday, July 11, 2006 11:49 PM
		To: [email-masked]
		Subject: [AVT] draft-clark-avt-rtcpxr-video-02

		I have concerns with the packet loss metrics reported in
the proposed blocks (RTP and MPEG TS). The Burst and Gap models are
based on the VoIP metrics block of RTCP-XR that were used for
consistency with other standards for voice quality algorithms. I suggest
either replacing burst and gap with packet loss period and packet loss
distance based on RFC3357 or augmenting them with these metrics. RFC3357
metrics are better defined and more accurately represent packet loss
behavior.

		Regards, 
		Kaynam 

		----------------------------------------- 
		Kaynam Hedayat 
		Chief Technology Officer 
		Vice President, Engineering 

		Brix Networks 
		285 Mill Road 
		Chelmsford, MA 01824 
		tel. 978.367.5611 
		fax. 978.367.5700 
		[email-masked] 
		www.brixnet.com <http://www.brixnet.com>  

		Service Quality Matters. Test the performance and
quality of your VoIP or IP video service at http://www.TestYourVoIP.com
<http://www.testyourvoip.com/>   and http://www.TestYourIPVideo.com
<http://www.testyouripvideo.com/>  

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Alan Clark" <[email-masked]>
Date: Wed, 12 Jul 2006 17:03:26 -0400

Kaynam
 
RFC3357's ideas on packet loss distribution seem largely based on work done
in the mid-1990's.  Several of the studies of packet loss burstiness "only"
considered the notion that bursts were consecutive - and not surprisingly if
you only look for some specific characteristic then that is what you find.
I re-analyzed a number of the traces used in some of these studies (e.g. Sue
Moon's traces) however used a sparse burst model instead of a consecutive
loss model - and found significant presence of sparse bursts.  Across a
number of traces we've seen bursts of 20-30% loss or loss/discard density
that range up to several seconds in length.
 
If you consider the impact of a consecutive loss burst vs a sparse burst
then the issue becomes apparent.  A burst of 2 or 3 lost packets will cause
a short term degradation lasting a few tens of milliseconds for voice or
audio, and several hundred milliseconds for video.  A burst of 20% packet
loss lasting for 2 seconds will completely trash the decoded video for 2-3
seconds.  The burst model used in RFC3611 can recognize either type of burst
whereas a consecutive loss model can only recognize consecutive loss.
 
If I were a service provider, I would not like the idea of a measurement
system that was insensitive to sparse bursts.
 
With regard to field deployment -  there are over 70 equipment vendors that
have/are deploying RFC3611 style metrics in their products, with an
installed base of several million units as of late 2005 - expected to grow
to 10-20 million units by the end of 2006. 
 
The video metrics draft specifies loss/discard metrics for "transport units"
and not "packets".   Video systems can use various multiplexing algorithms,
including packing multiple transport units for the same video stream into
one packet.  Are you suggesting that RFC3357 should be used as specified or
should be re-interpreted to mean that packets are transport units?
 
Alan

  _____  

From: Hedayat, Kaynam [mailto:[email-masked]] 
Sent: Wednesday, July 12, 2006 3:15 PM
To: Alan Clark; [email-masked]
Subject: RE: [AVT] draft-clark-avt-rtcpxr-video-02

Alan,
 
The argument is for not tying the model to the metrics reported by XR or at
a minimum have an option that will allow multiple models. RFC3357 provides
metrics that provide packet loss behavior in terms of loss periods and loss
distance, both applicable to real-time applications. 
 
I disagree with your point that 3357 does not consider packet discard. For
the application, loss is loss, regardless of its source (network or
endpoint). Both kinds of loss can be described in terms defined by 3357.
 
A lot of good work and time has been put into RFC3357 and its metrics are
widely used in the field today, especially for Video over IP applications
(IPTV, VoD, etc.). I can't imagine any argument against using these metrics.
 
Regards,
Kaynam

  _____  

Kaynam
 
You are quite incorrect.
 
(i) The burst and gap metrics were originally defined in RFC3611 (RTCP XR)
and, as they are extremely helpful in understanding the impact of IP
impairments on VoIP, were then adopted in eight other standards related to
VoIP performance.    Hence the original work was done in AVT and then
adopted across the industry
 
(ii) The concepts used in RFC3611 are quite different to those in RFC3357
for good reason.  We analyzed the distribution of lost packets on many
traces and found that "burstiness" tended to be exhibited as periods of high
loss and NOT as periods of consecutive loss. We also found that periods of
high PDV typically coincide with periods of high loss, and hence see merit
in measuring the impact of loss and PDV together (i.e. loss/discard
distribution).
 - The loss period you refer to from RFC3357 defines only consecutive loss
and not the sparse bursts that typically occur (particularly at the network
edge), whereas RFC3611 describes sparse bursts
 - RFC3357 only applies to lost packets, and not to packets discarded due to
late arrival, whereas the model from RFC3611 applies to the combined
loss/discard distribution

I totally disagree with your assertion that RFC3357 metrics are better
defined and more accurately represent packet loss behavior - you are
entitled to hold any opinion you like however if you want to express an
opinion should explain the justification for it.  Any of these measurements
essentially comprise parameters of some assumed model that describes the
behavior of the system being measured.  You should try and ensure that the
model is relevant with regard to the aspect of the system behavior you are
interested in.  You could argue that the assumed model is or is not a good
fit for the aspect of system behavior that is of interest, or that the
method for parameter estimation is more or less accurate.
 
Alan

  _____  

I have concerns with the packet loss metrics reported in the proposed blocks
(RTP and MPEG TS). The Burst and Gap models are based on the VoIP metrics
block of RTCP-XR that were used for consistency with other standards for
voice quality algorithms. I suggest either replacing burst and gap with
packet loss period and packet loss distance based on RFC3357 or augmenting
them with these metrics. RFC3357 metrics are better defined and more
accurately represent packet loss behavior.

Regards, 
Kaynam 

Brix Networks 
285 Mill Road 
Chelmsford, MA 01824 
tel. 978.367.5611 
fax. 978.367.5700 
[email-masked] 
 <http://www.brixnet.com> www.brixnet.com 

Service Quality Matters. Test the performance and quality of your VoIP or IP
video service at  <http://www.testyourvoip.com/> http://www.TestYourVoIP.com
and  <http://www.testyouripvideo.com/> http://www.TestYourIPVideo.com