Subject: [AVT] Comments on draft-ietf-avt-rtp-hdrext-01.txt

From: Stephen Casner <[email-masked]>
Date: Mon, 10 Jul 2006 13:37:39 -0700 (PDT)

A while ago, I asked Ron Frederick, one of the co-authors of the RTP
RFC, if he had any comments on draft-ietf-avt-rtp-hdrext-01.txt.  He
replied but then I neglected to forward them to the WG mailing list.
Here they are now.  To the extent that these comments may be
out-of-date with respect to the current revision, that's my fault.

My own position is that I have significant concerns about opening up
the header extension mechanism.  It is a genie that can't be put back
in the bottle.

                                                        -- Steve

---------- Forwarded message ----------
Date: Tue, 04 Apr 2006 17:08:38 -0700
From: Ron Frederick <[email-masked]>

[snip]

I just read through this. It generally looks ok, but I do have a few
thoughts on it...

First, there's no alignment on the individual header extensions. We
might want to provide a mechanism to allow extensions to be aligned if
they need to be. We've already used a byte of 0x00 (type=0, len=0) as
padding at the end of the extensions section. We could potentially use
0xff (type=15, len=15) as inter-extension padding, or use 0x00 in both
cases and say that 0x00 bytes within the extension header should just be
ignored, allowing any number of them between valid extensions within the
region indicated by the overall extension length.

Second, I think the proposed 'toffset' extension could use a bit more
discussion around it. In particular, it strikes me that the way it is
defined today would suggest that it would always be reporting a non-zero
number even in cases where packets were evenly spaced, because of the
nature of the RTP timestamp representing the timestamp of the first
sample in the packet, not the timestamp of the last one. Is that really
the intention, though? Giving a concrete example here might be helpful.
Also, it seems that one of the main uses of this extension is to improve
the accuracy of a receiver's jitter calculation. If that's the
intention, it might be good to specifically raise that point here and to
propose specifically how the jitter calculation should be changed at
receivers that understand this option. If an example is provided, it
might also be good to separately cover audio and video data, since the
common case for each is slightly different. With video data in
particular, there'd be the issue of multiple packets being sent at
different times with a common RTP (frame) timestamp.

Finally, I noticed in the BNF section that there's something defined
there called "mapattrs", but it doesn't seem to be referenced anywhere
else in the document. Where exactly would this fit in? Was it supposed
to appear after extensionname in the definition of "extmap"?

[snip]
-- 
Ron Frederick
[email-masked]

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Desineni, Harikishan" <[email-masked]>
Date: Wed, 12 Jul 2006 21:38:35 -0700

I don't know what was there in -01 and -02 version of this draft. I
looked at the -03 version of the draft. The concept of "offset" is not
clear enough to me from the description provided in section 6.

[1a] For each media type (audio,video & text), how common is it that a
sender does not transmit the RTP packets strictly according to their
relative RTP timestamps? 
[1b]Is that not a bug in the implementation of the RTP transmitter?
[1c] Does an ideal RTP transmitter always output offset value of 0?

[2] When would the sender know that it is not transmitting the RTP
packets strictly according to their relative RTP timestamps?
Is it an implementation issue? 

[3] What is the purpose of knowing "effective" RTP transmission time of
the packet? Is it just for much accurate inter arrival jitter value
reported in RTCP receiver reports? 

"offset" is fixed 24-bit signed integer.
[4] When would the offset be +ve and what does that mean? Delayed
transmission?

[5] When would the offset be -ve and what does that mean? Too early
transmission? Can an early RTP transmission ever take place?

[6] Should an RTP transmitter try its best to keep the value of offset
as close as possible to 0?

My apologies if I am trying to understand the issues which were already
discussed on this list.

Thanks,
Hari

-----Original Message-----
From: Stephen Casner [mailto:[email-masked]] 
Sent: Monday, July 10, 2006 1:38 PM
To: AVT WG
Cc: Ron Frederick
Subject: [AVT] Comments on draft-ietf-avt-rtp-hdrext-01.txt

                                                        -- Steve

---------- Forwarded message ----------
Date: Tue, 04 Apr 2006 17:08:38 -0700
From: Ron Frederick <[email-masked]>

[snip]

[snip]
-- 
Ron Frederick
[email-masked]

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 13 Jul 2006 18:20:26 -0400

> sender does not transmit the RTP packets strictly according to their

It is quite common, but is both media and application dependent. In 
streaming applications that use transmission scheduling, basically all 
packets will have some offset to produce a constant bit-rate stream from 
variable bit-rate encoded media.

For video there are many examples where a single video frame will result 
in several packets. These packets can't be and really shouldn't be 
transmitted at the same time instance. Thus there will be some 
transmission times differences in these cases.

> [1b]Is that not a bug in the implementation of the RTP transmitter?

No, as it is commonly done on purpose and otherwise by necessity.

> [1c] Does an ideal RTP transmitter always output offset value of 0?
I don't think so. Depending on in which aspect you define an ideal 
transmitter.

> Is it an implementation issue? 

Yes, and application dependent. In many cases one can measure the 
deviations in transmission times compared to sampling time of the media 
content.

> reported in RTCP receiver reports? 

It is beneficial in buffer handling and transmission rate adaptation.

> [4] When would the offset be +ve and what does that mean? Delayed

To me the offset is actually the deviation compared to a static delay 
between the sampling instant and the transmission instant. Thus the 
transmission may happen both slightly later and earlier than what the 
fixed delay indicates.

> transmission? Can an early RTP transmission ever take place?

Yes, see above.

> as close as possible to 0?

No, as explained for certain application do benefit from moving the 
transmission time.

Cheers

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]

---

From: Dave Singer <[email-masked]>
Date: Tue, 18 Jul 2006 18:43:16 +0200

Magnus has it, as usual.  The only thing I'd add is that definition 
of the absolute offset values is probably not needed.  Normally you'd 
expect a packet with timestamp R2 to arrive after a packet with 
timestamp R1 after an interval of R2-R1.  However, if the 
transmissions were offset, for traffic smoothing etc., then you'd 
expect them to arrive now at (R2+toffset2) - (R1+toffset1).

At 18:20  -0400 13/07/06, Magnus Westerlund wrote:
>streaming applications that use transmission scheduling, basically 
>all packets will have some offset to produce a constant bit-rate 
>stream from variable bit-rate encoded media.
>
>For video there are many examples where a single video frame will 
>result in several packets. These packets can't be and really 
>shouldn't be transmitted at the same time instance. Thus there will 
>be some transmission times differences in these cases.
>
>deviations in transmission times compared to sampling time of the 
>media content.
>
>>
>>[3] What isFrom [email-masked] Tue Jul 18 13:01:05 2006
Date: Tue, 18 Jul 2006 18:43:16 +0200
To: Magnus Westerlund <[email-masked]>,
	"Desineni, Harikishan" <[email-masked]>
From: Dave Singer <[email-masked]>
Subject: Re: [AVT] Comments on draft-ietf-avt-rtp-hdrext-01.txt
Cc: Stephen Casner <[email-masked]>, Ron Frederick <[email-masked]>,
	AVT WG <[email-masked]>

Magnus has it, as usual.  The only thing I'd add is that definition 
of the absolute offset values is probably not needed.  Normally you'd 
expect a packet with timestamp R2 to arrive after a packet with 
timestamp R1 after an interval of R2-R1.  However, if the 
transmissions were offset, for traffic smoothing etc., then you'd 
expect them to arrive now at (R2+toffset2) - (R1+toffset1).

At 18:20  -0400 13/07/06, Magnus Westerlund wrote:
>streaming applications that use transmission scheduling, basically 
>all packets will have some offset to produce a constant bit-rate 
>stream from variable bit-rate encoded media.
>
>For video there are many examples where a single video frame will 
>result in several packets. These packets can't be and really 
>shouldn't be transmitted at the same time instance. Thus there will 
>be some transmission times differences in these cases.
>
>deviations in transmission times compared to sampling time of the 
>media content.
>
>>
>To me the offset is actually the deviation compared to a static 
>delay between the sampling instant and the transmission instant. 
>Thus the transmission may happen both slightly later and earlier 
>than what the fixed delay indicates.
>
>>

-- 
David Singer
Apple Computer/QuickTime

---

From: Dave Singer <[email-masked]>
Date: Tue, 18 Jul 2006 18:46:11 +0200

At 13:37  -0700 10/07/06, Stephen Casner wrote:
>the header extension mechanism.  It is a genie that can't be put back the purpose of knowing "effective" RTP transmission time of

-- 
David Singer
Apple Computer/QuickTime

---

From: Dave Singer <[email-masked]>
Date: Tue, 18 Jul 2006 18:46:11 +0200

>the header extension mechanism.  It is a genie that can't be put back

Yes, I agree.  That's why I haven't been trying to fast-track it 
through the process...

>region indicated by the overall extension length.

This is a good idea, thanks.

>different times with a common RTP (frame) timestamp.

Yes, thanks.  Colin asked for the toffset to be in a separate draft 
and separately processed, also.

>to appear after extensionname in the definition of "extmap"?

This was an oops.

-- 
David Singer
Apple Computer/QuickTime

>in the bottle.

Yes, I agree.  That's why I haven't been trying to fast-track it 
through the process...

>region indicated by the overall extension length.

This is a good idea, thanks.

>different times with a common RTP (frame) timestamp.

Yes, thanks.  Colin asked for the toffset to be in a separate draft 
and separately processed, also.

>to appear after extensionname in the definition of "extmap"?

This was an oops.

-- 
David Singer
Apple Computer/QuickTime

---

From: Ron Frederick <[email-masked]>
Date: Tue, 18 Jul 2006 15:07:15 -0700

Dave Singer wrote:
> Magnus has it, as usual.  The only thing I'd add is that definition of 
> the absolute offset values is probably not needed.  Normally you'd 
> timestamp R1 after an interval of R2-R1.  However, if the transmissions 
> were offset, for traffic smoothing etc., then you'd expect them to 
> arrive now at (R2+toffset2) - (R1+toffset1).

RTCP provides a mapping from RTP timestamps to an absolute NTP time, and 
RFC 3551 specifically mentions what absolute time the RTP timestamp 
should correspond to. For frame-based audio encodings for instance, 
section 4.4 of RFC 3551 states:

     "The RTP timestamp reflects the instant at which the first sample in
      the first frame was sampled, that is, the oldest information in the
      packet."

So, if the transmission timestamp is computed by adding the RTP 
timestamp in the packet and this offset value, the resulting value will 
correspond to a specific absolute time, in addition to being usable in 
the way you describe above to determine expected interarrival times for 
things like jitter calculations.

Magnus said in his comments that the offset may represent a deviation 
compared to a static delay between the sampling instant and the 
transmission instant. If this is the case, though, it means that we can 
no longer know the absolute transmission time. We can only know 
something which may be an unknown fixed offset off of this time. The 
advantage to this is that we can ignore things like computational 
overhead in generating and sending the packet if we have some reason to 
believe it is constant for all packets, but that seems like a dangerous 
assumption to make. We can also ignore other constant offsets like the 
fact that the RTP timestamp in audio packets corresponds to the first 
sample but we don't send these packets until after we capture the last 
sample. However, it would not be difficult at all to factor this into 
the calculation and make this into a true offset. This could make it 
useful for things like round-trip calculations between two hosts if they 
were both synchronized to NTP. With a "deviation compared to a static 
delay", we would not be able to claim this.

For video encodings, the RTP timestamp in a packet represents the 
sampling instant of the video image being encoded. If the compression 
processing took zero time and the host sent all of the packets in the 
video frame immediately, it would be possible in this case for the 
offset values in all these packets to be zero. However, a real system 
would probably have some delay between sampling the frame and sending 
the first packet, and this delay should really be reflected in this 
offset value. Also, if there are any additional delays before sending 
other packets associated with this frame, these delays should be 
accumulated and included in the offset values in each of the later 
packets, since all of the packets in the frame have the same RTP 
timetstamp. This is true regardless of whether the offset is a true 
offset or just a deviation. So, making it a true offset for video 
doesn't really increase the complexity much.

It would be good to capture some of these details in section 6 in the 
latest version of the hdrext Internet Draft.

I've looked over the -03 version of this draft and most of my original 
comments still apply. I'd still like to see something which supports 
alignment of individual extensions within the header extension block by 
allowing some form of internal padding in addition to the padding at the 
end, and I think the 'toffset' extension could use some examples. The 
issue I raised with the BNF has been addressed already, though.

As for the point being raised in this mail thread expressing concerns 
about using the RTP header extension instead of something like a 
profile-specific fixed header extension, I would also add my 
reservations about this. While this design seems reasonable for RTP 
header extensions, I'm in no hurry to encourage their use. I'd much 
prefer to see a new profile defined which adds an appropriate fixed 
payload header for anything that profile really needed.

>> packets will have some offset to produce a constant bit-rate stream 
>> from variable bit-rate encoded media.
>>
>> result in several packets. These packets can't be and really shouldn't 
>> be transmitted at the same time instance. Thus there will be some 
-- 
Ron Frederick
[email-masked]

---

From: "Desineni, Harikishan" <[email-masked]>
Date: Wed, 19 Jul 2006 13:47:24 -0700

When R2 == R1 (Two RTP packets with same TS), the expected difference in
arrival time would be (toffset2 - toffset1). It appears that this would
be true only if each offset is "true offset" as defined by Ron. If the
offset is just a variation from a fixed delay (as defined by Magnus), it
may not accurately reflect the expected difference in the arrival time
of these packets.

Other than traffic smoothing, what other strong factor could
offset/deviate the tranmission of an RTP packet? 

It looks like this extension may not be of much use for audio/text media
types. I'd like to know if there are any good use cases for audio/text
media types. I don't think it is very common that a single speech/audio
frame consume multiple RTP packets.

Thanks,
Hari 

> From: Ron Frederick [mailto:[email-masked]] 
> Sent: Tuesday, July 18, 2006 3:07 PM
> To: Dave Singer
> Cc: Magnus Westerlund; Desineni, Harikishan; Stephen Casner; AVT WG
> RTCP provides a mapping from RTP timestamps to an absolute 
> NTP time, and RFC 3551 specifically mentions what absolute 
> time the RTP timestamp should correspond to. For frame-based 
> audio encodings for instance, section 4.4 of RFC 3551 states:
> 
> So, if the transmission timestamp is computed by adding the 
> RTP timestamp in the packet and this offset value, the 
> resulting value will correspond to a specific absolute time, 
> in addition to being usable in the way you describe above to 
> determine expected interarrival times for things like jitter 
> calculations.
> 
> Magnus said in his comments that the offset may represent a 
> deviation compared to a static delay between the sampling 
> instant and the transmission instant. If this is the case, 
> though, it means that we can no longer know the absolute 
> transmission time. We can only know something which may be an 
> unknown fixed offset off of this time. The advantage to this 
> is that we can ignore things like computational overhead in 
> generating and sending the packet if we have some reason to 
> believe it is constant for all packets, but that seems like a 
> dangerous assumption to make. We can also ignore other 
> constant offsets like the fact that the RTP timestamp in 
> audio packets corresponds to the first sample but we don't 
> send these packets until after we capture the last sample. 
> However, it would not be difficult at all to factor this into 
> the calculation and make this into a true offset. This could 
> make it useful for things like round-trip calculations 
> between two hosts if they were both synchronized to NTP. With 
> a "deviation compared to a static delay", we would not be 
> able to claim this.
> 
> For video encodings, the RTP timestamp in a packet represents 
> the sampling instant of the video image being encoded. If the 
> compression processing took zero time and the host sent all 
> of the packets in the video frame immediately, it would be 
> possible in this case for the offset values in all these 
> packets to be zero. However, a real system would probably 
> have some delay between sampling the frame and sending the 
> first packet, and this delay should really be reflected in 
> this offset value. Also, if there are any additional delays 
> before sending other packets associated with this frame, 
> these delays should be accumulated and included in the offset 
> values in each of the later packets, since all of the packets 
> in the frame have the same RTP timetstamp. This is true 
> regardless of whether the offset is a true offset or just a 
> deviation. So, making it a true offset for video doesn't 
> really increase the complexity much.
> 
> It would be good to capture some of these details in section 
> 6 in the latest version of the hdrext Internet Draft.
> 
> I've looked over the -03 version of this draft and most of my 
> original comments still apply. I'd still like to see 
> something which supports alignment of individual extensions 
> within the header extension block by allowing some form of 
> internal padding in addition to the padding at the end, and I 
> think the 'toffset' extension could use some examples. The 
> As for the point being raised in this mail thread expressing 
> concerns about using the RTP header extension instead of 
> something like a profile-specific fixed header extension, I 
> would also add my reservations about this. While this design 
> seems reasonable for RTP header extensions, I'm in no hurry 
> to encourage their use. I'd much prefer to see a new profile 
> defined which adds an appropriate fixed payload header for 
> anything that profile really needed.
> 
> >>> looked at the -03 version of the draft. The concept of 
> "offset" is 
> >>> not clear enough to me from the description provided in section 6.
> >>>
> >>> [1a] For each media type (audio,video & text), how common 
> is it that 
> >>> a sender does not transmit the RTP packets strictly according to 
> >>> their relative RTP timestamps?
> >>
> >>> [3] What is the purpose of knowing "effective" RTP 
> transmission time 
> >>> of the packet? Is it just for much accurate inter arrival jitter 
> >>> value reported in RTCP receiver reports?
> >>
> >>> [6] Should an RTP transmitter try its best to keep the value of 
> >>> offset as close as possible to 0?
> >>

---

From: Ron Frederick <[email-masked]>
Date: Wed, 19 Jul 2006 16:02:49 -0700

> may not accurately reflect the expected difference in the arrival time

If I understand what Magnus had in mind, the fixed offset would be the 
same for all RTP packets in the stream, so toffset2-toffset1 would 
result in the same value regardless of whether it was the true offset or 
some other offset that was off by the same fixed amount in both cases.

> frame consume multiple RTP packets.

For audio, I would generally agree that this extension will not add much 
value. Generally, audio packets are evenly spaced and relatively fixed 
in size, so there's not much motivation to "smooth" them on 
transmission. I don't have much experience with text streams being 
carried over RTP, but generally there I would expect that the bitrate 
would be low enough that smoothing would also not be a major factor, so 
I'd agree there as well that this wouldn't add much value. The main 
benefit will be for video.

Other than traffic smoothing, the only other reason I would expect 
offsets to vary is for sources where compression is being done in 
real-time and the processing time for the compression is varying. This 
could be reflected in the offset field so that this variable delay 
doesn't appear to be network jitter. However, even there, I would not 
expect this to be much of an issue for audio or text - again, it would 
mostly apply to video.

-- 
Ron Frederick
[email-masked]