Subject: RE: [AVT] Comments on draft-ietf-avt-rtp-hdrext-01.txt

From: "Allison, Art" <[email-masked]>
Date: Wed, 19 Jul 2006 15:17:31 -0400

Is determining when to render sound with respect to video left to the
implementer for the case where the audio frames per video frame is not
an integer? 

_____________
Art Allison
Director, Advanced Engineering
Science & Technology
National Association of Broadcasters
1771 N Street, NW
Washington, D.C. 20036
Phone: 202.429.5418
Fax: 202.777.4981
[email-masked]

The National Association of Broadcasters is a trade association that
advocates on behalf of more than 8,300 free, local radio and television
stations and also broadcast networks before Congress, the Federal
Communications Commission and the Courts.

-----Original Message-----
From: Ron Frederick [mailto:[email-masked]] 
Sent: Tuesday, July 18, 2006 6:07 PM
To: Dave Singer
Cc: Magnus Westerlund; Stephen Casner; AVT WG
Subject: Re: [AVT] Comments on draft-ietf-avt-rtp-hdrext-01.txt

Dave Singer wrote:
> Magnus has it, as usual.  The only thing I'd add is that definition of

> the absolute offset values is probably not needed.  Normally you'd 
> expect a packet with timestamp R2 to arrive after a packet with 
> timestamp R1 after an interval of R2-R1.  However, if the 
> transmissions were offset, for traffic smoothing etc., then you'd 
> expect them to arrive now at (R2+toffset2) - (R1+toffset1).

RTCP provides a mapping from RTP timestamps to an absolute NTP time, and
RFC 3551 specifically mentions what absolute time the RTP timestamp
should correspond to. For frame-based audio encodings for instance,
section 4.4 of RFC 3551 states:

     "The RTP timestamp reflects the instant at which the first sample
in
      the first frame was sampled, that is, the oldest information in
the
      packet."

So, if the transmission timestamp is computed by adding the RTP
timestamp in the packet and this offset value, the resulting value will
correspond to a specific absolute time, in addition to being usable in
the way you describe above to determine expected interarrival times for
things like jitter calculations.

Magnus said in his comments that the offset may represent a deviation
compared to a static delay between the sampling instant and the
transmission instant. If this is the case, though, it means that we can
no longer know the absolute transmission time. We can only know
something which may be an unknown fixed offset off of this time. The
advantage to this is that we can ignore things like computational
overhead in generating and sending the packet if we have some reason to
believe it is constant for all packets, but that seems like a dangerous
assumption to make. We can also ignore other constant offsets like the
fact that the RTP timestamp in audio packets corresponds to the first
sample but we don't send these packets until after we capture the last
sample. However, it would not be difficult at all to factor this into
the calculation and make this into a true offset. This could make it
useful for things like round-trip calculations between two hosts if they
were both synchronized to NTP. With a "deviation compared to a static
delay", we would not be able to claim this.

For video encodings, the RTP timestamp in a packet represents the
sampling instant of the video image being encoded. If the compression
processing took zero time and the host sent all of the packets in the
video frame immediately, it would be possible in this case for the
offset values in all these packets to be zero. However, a real system
would probably have some delay between sampling the frame and sending
the first packet, and this delay should really be reflected in this
offset value. Also, if there are any additional delays before sending
other packets associated with this frame, these delays should be
accumulated and included in the offset values in each of the later
packets, since all of the packets in the frame have the same RTP
timetstamp. This is true regardless of whether the offset is a true
offset or just a deviation. So, making it a true offset for video
doesn't really increase the complexity much.

It would be good to capture some of these details in section 6 in the
latest version of the hdrext Internet Draft.

I've looked over the -03 version of this draft and most of my original
comments still apply. I'd still like to see something which supports
alignment of individual extensions within the header extension block by
allowing some form of internal padding in addition to the padding at the
end, and I think the 'toffset' extension could use some examples. The
issue I raised with the BNF has been addressed already, though.

As for the point being raised in this mail thread expressing concerns
about using the RTP header extension instead of something like a
profile-specific fixed header extension, I would also add my
reservations about this. While this design seems reasonable for RTP
header extensions, I'm in no hurry to encourage their use. I'd much
prefer to see a new profile defined which adds an appropriate fixed
payload header for anything that profile really needed.

> At 18:20  -0400 13/07/06, Magnus Westerlund wrote:
>> Desineni, Harikishan wrote:
>>> I don't know what was there in -01 and -02 version of this draft. I 
>>> looked at the -03 version of the draft. The concept of "offset" is 
>>> not clear enough to me from the description provided in section 6.
>>>
>>> [1a] For each media type (audio,video & text), how common is it that

>>> a sender does not transmit the RTP packets strictly according to 
>>> their relative RTP timestamps?
>>
>> It is quite common, but is both media and application dependent. In 
>> streaming applications that use transmission scheduling, basically 
>> all packets will have some offset to produce a constant bit-rate 
>> stream from variable bit-rate encoded media.
>>
>> For video there are many examples where a single video frame will 
>> result in several packets. These packets can't be and really 
>> shouldn't be transmitted at the same time instance. Thus there will 
>> be some transmission times differences in these cases.
>>
>>> [1b]Is that not a bug in the implementation of the RTP transmitter?
>>
>> No, as it is commonly done on purpose and otherwise by necessity.
>>
>>> [1c] Does an ideal RTP transmitter always output offset value of 0?
>>>
>> I don't think so. Depending on in which aspect you define an ideal 
>> transmitter.
>>
>>>
>>> [2] When would the sender know that it is not transmitting the RTP 
>>> packets strictly according to their relative RTP timestamps?
>>> Is it an implementation issue?
>>
>> Yes, and application dependent. In many cases one can measure the 
>> deviations in transmission times compared to sampling time of the 
>> media content.
>>
>>>
>>> [3] What is the purpose of knowing "effective" RTP transmission time

>>> of the packet? Is it just for much accurate inter arrival jitter 
>>> value reported in RTCP receiver reports?
>>
>> It is beneficial in buffer handling and transmission rate adaptation.
>>
>>>
>>> "offset" is fixed 24-bit signed integer.
>>> [4] When would the offset be +ve and what does that mean? Delayed 
>>> transmission?
>>
>> To me the offset is actually the deviation compared to a static delay

>> between the sampling instant and the transmission instant. Thus the 
>> transmission may happen both slightly later and earlier than what the

>> fixed delay indicates.
>>
>>>
>>> [5] When would the offset be -ve and what does that mean? Too early 
>>> transmission? Can an early RTP transmission ever take place?
>>
>> Yes, see above.
>>
>>>
>>> [6] Should an RTP transmitter try its best to keep the value of 
>>> offset as close as possible to 0?
>>
>> No, as explained for certain application do benefit from moving the 
>> transmission time.
>>
>> Cheers
>>
>> Magnus Westerlund
>>
>> Multimedia Technologies, Ericsson Research EAB/TVA/A
>>
----------------------------------------------------------------------
>> Ericsson AB                | Phone +46 8 4048287
>> Torshamsgatan 23           | Fax   +46 8 7575550
>> S-164 80 Stockholm, Sweden | mailto: [email-masked]
>>
>> _______________________________________________
>> Audio/Video Transport Working Group
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/avt
--
Ron Frederick
[email-masked]

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 20 Jul 2006 19:40:26 +0200

Hi Art,

The problem is that the RTCP's TS to NTP mapping does not result in an 
absolute transmissions time as there are no requirement that the sender 
is NTP synchronized. Thus they may be correct absolute time but it might 
as well be off with some offset. You also need to consider cases where 
the TS in generated from some other information, rather than being based 
on a sampling clock. Thus putting in the absolute offset primarily 
consumes bits, but could off course be adjusted by changing the NTP.

In many cases a receiver also isn't aware of the absolute delay from the 
transmitter to the receiver. They are only interested in the change of 
the delay, rather then the absolute delay.

These are the reasons why I formulated it as I did. It will of course be 
possible to formulate the offset so that it is possible in cases when 
one knows this. However one needs to be careful so that it also isn't 
working in the quite common case that senders aren't NTP synchronized.

Cheers

Magnus Westerlund

---

From: Ron Frederick <[email-masked]>
Date: Thu, 20 Jul 2006 12:21:38 -0700

> working in the quite common case that senders aren't NTP synchronized.

 From a sender's point of view, there's no requirement to be NTP 
synchronized in order to produce an absolute offset between its "capture 
time" of the data (represented in the RTP timestamp) and its send time. 
Both of the timestamps are local to the same machine. It's true that a 
receiver would not be able to relate that send time to an absolute time 
on its local clock without some form of clock synchronization between 
the systems (NTP or otherwise), but allowing the sender to add some 
arbitrary offset to what it outputs in this field just doesn't seem to 
really add much value. It would still have to make sure that this offset 
was constant across all packets in the RTP stream, and the only way I 
can see of it doing that is to know what the absolute offset really was 
and then "normalize" it somehow by adding/subtracting some fixed 
constant. Why complicate things in that way, though?

The other advantage I see to an absolute offset is that it would allow a 
receiver to be able to meaningfully compare send times between packets 
arriving on different RTP streams from the same sender. This would work 
even in cases where there was no time synchronization between the sender 
and receiver at all. With the relative offset, this wouldn't be possible 
unless we could guarantee that the fixed offset was the same for all RTP 
streams, and I would not expect that to be the case.
-- 
Ron Frederick
[email-masked]

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 21 Jul 2006 10:24:41 +0200

Ron Frederick wrote:
>> absolute transmissions time as there are no requirement that the 
>> sender is NTP synchronized. Thus they may be correct absolute time but 
>> it might as well be off with some offset. You also need to consider 
>> cases where the TS in generated from some other information, rather 
>> than being based on a sampling clock. Thus putting in the absolute 
>> offset primarily consumes bits, but could off course be adjusted by 
>> changing the NTP.
>>
>> In many cases a receiver also isn't aware of the absolute delay from 
>> the transmitter to the receiver. They are only interested in the 
>> change of the delay, rather then the absolute delay.
>>
>> These are the reasons why I formulated it as I did. It will of course 
>> be possible to formulate the offset so that it is possible in cases 
>> when one knows this. However one needs to be careful so that it also 
>> isn't working in the quite common case that senders aren't NTP 
>> synchronized.
> 

Ron, I think we have slightly different views on how this is handled. 
However I do agree that it makes sense to define the offset in the 
following way.

offset = convert_to_RTP_TS(NTP_transmission_instant - NTP_sampling_instant)

Where NTP_sampling_instant will be the NTP value used in RTCP if the 
RTCP packet would contain the same RTP TS value. That will work 
independently of if the content is actually sampled or have a generated 
time line.

Cheers

Magnus Westerlund