Subject: RE: [AVT] Comments on draft-ietf-avt-rtp-hdrext-01.txt

From: Dave Singer <[email-masked]>
Date: Tue, 1 Aug 2006 15:35:43 -0700

I'm sorry, I was on vacation when this all erupted.  I've tried to 
read the entire thread, and respond below.

I will separate the transmission offset section into a separate 
draft.  It clearly needs separate handling.

At 21:38  -0700 12/07/06, Desineni, Harikishan wrote:
>I don't know what was there in -01 and -02 version of this draft. I
>looked at the -03 version of the draft. The concept of "offset" is not
>clear enough to me from the description provided in section 6.
>
>[1a] For each media type (audio,video & text), how common is it that a
>sender does not transmit the RTP packets strictly according to their
>relative RTP timestamps?

Three major cases:
(a) traffic smoothing -- in particular, video I -frames tend to be 
large and therefore represent a bandwidth spike that both needs 
spreading out, and may involve distorting the send time of adjacent 
frames as well
(b) I/B/P video, where the dependencies mean that frames have to be 
sent in decoding order but with presentation timestamps.  Then the 
timestamps are not even increasing with sequence number.
(c) re-transmissions.  They were clearly sent at a way different time 
than the original transmission.

>[3] What is the purpose of knowing "effective" RTP transmission time of
>the packet? Is it just for much accurate inter arrival jitter value
>reported in RTCP receiver reports?

That is one possible use.  There are other cases (e.g. when 
attempting to use any kind of adaptive client-server buffer 
management).  I proposed this because I had seen enough 
complaints/concerns over the years that there wasn't a standard place 
to inform the client about actual transmission times.

>"offset" is fixed 24-bit signed integer.
>[4] When would the offset be +ve and what does that mean? Delayed
>transmission?
>
>[5] When would the offset be -ve and what does that mean? Too early
>transmission? Can an early RTP transmission ever take place?
>
>[6] Should an RTP transmitter try its best to keep the value of offset
>as close as possible to 0?

My intention was that the offsets not have much meaning by 
themselves.  Classically a pair of RTP packets with timestamps R2 and 
R1 are transmitted with a time interval between them of (R2 - R1). 
The offsets O2 and O1 would be such that you would know that the 
original transmission interval was (R2 + O2) - (R1 + O1).

At 15:07  -0700 18/07/06, Ron Frederick wrote:
>
>RTCP provides a mapping from RTP timestamps to an absolute NTP time, 
>and RFC 3551 specifically mentions what absolute time the RTP 
>timestamp should correspond to. For frame-based audio encodings for 
>instance, section 4.4 of RFC 3551 states:
>
>     "The RTP timestamp reflects the instant at which the first sample in
>      the first frame was sampled, that is, the oldest information in the
>      packet."
>
>So, if the transmission timestamp is computed by adding the RTP 
>timestamp in the packet and this offset value, the resulting value 
>will correspond to a specific absolute time, in addition to being 
>usable in the way you describe above to determine expected 
>interarrival times for things like jitter calculations.

If we can find a good general definition of the absolute value of the 
offset (rather than the relative one I am suggesting), that would be 
great.  My concern about the way it is phrased in RTP is that it is 
actually rather limiting and limited.  Talking about the "sampling 
instant"
a) is a backwards-looking statement.  To me, it's better to express 
to the receiver what they need to know to do "the right thing" with 
what they get, rather than try to express something about how the 
stream was made (which is none of their business, in some sense).
b) talking about "sampling instants" assumes sampling, which doesn't 
actually happen for some payload types (e.g. MIDI streamed from a 
file) and even for other payload types, may never have happened (e.g. 
animations rendered to file).
c) it really isn't the sampling instant anyway;  if the material has 
been edited after capture and before streaming, then we really don't 
want to know "this was sampled at 2pm in hollywood" and then minutes 
later "this was sampled at 8pm on location in beverly hills".

In general, "sampling instant" focuses on the case of live streams to 
the detriment of understanding of other cases, in my opinion.

At 13:47  -0700 19/07/06, Desineni, Harikishan wrote:
>When R2 == R1 (Two RTP packets with same TS), the expected difference in
>arrival time would be (toffset2 - toffset1). It appears that this would
>be true only if each offset is "true offset" as defined by Ron. If the
>offset is just a variation from a fixed delay (as defined by Magnus), it
>may not accurately reflect the expected difference in the arrival time
>of these packets.

Minimally I think it has to be defined that the (R2 + O2) - (R1 + O1) 
equation holds, which, as you say, reduces to O2-O1 when R2==R1.  The 
debate is over whether we can define what (R2 + O2) means by itself.

At 12:21  -0700 20/07/06, Ron Frederick wrote:
>
>From a sender's point of view, there's no requirement to be NTP 
>synchronized in order to produce an absolute offset between its 
>"capture time" of the data (represented in the RTP timestamp) and 
>its send time. Both of the timestamps are local to the same machine. 
>It's true that a receiver would not be able to relate that send time 
>to an absolute time on its local clock without some form of clock 
>synchronization between the systems (NTP or otherwise), but allowing 
>the sender to add some arbitrary offset to what it outputs in this 
>field just doesn't seem to really add much value. It would still 
>have to make sure that this offset was constant across all packets 
>in the RTP stream, and the only way I can see of it doing that is to 
>know what the absolute offset really was and then "normalize" it 
>somehow by adding/subtracting some fixed constant. Why complicate 
>things in that way, though?
>
>The other advantage I see to an absolute offset is that it would 
>allow a receiver to be able to meaningfully compare send times 
>between packets arriving on different RTP streams from the same 
>sender. This would work even in cases where there was no time 
>synchronization between the sender and receiver at all. With the 
>relative offset, this wouldn't be possible unless we could guarantee 
>that the fixed offset was the same for all RTP streams, and I would 
>not expect that to be the case.

I'm taking Magnus's definition and so on, and I think it is this:

take an RTP stream that has a recent RTCP sender report relating R0 
to N0, and a packet sent after that with timestamp R1.  Nominally 
this is sent at (ignoring timescale conversion issues) N1 = (N0 + (R1 
- R0)).  If it was sent at a different time, Na, then the offset 
value is (Na - N1).

At 16:45  -0700 21/07/06, Desineni, Harikishan wrote:
>With this definition, the "offset" can never take a -ve value. The
>definition of offset field could be changed to 24-bit unsigned integer.

I don't see why this follows in all cases at all.

At 9:26  -0700 24/07/06, Ron Frederick wrote:
>Ideally, I'd prefer a design where a video-on-demand stream was 
>forced to look exactly the same as a live stream to make it easier 
>on receivers. However, we also need to consider what information a 
>sender has available, and a storage server may not have enough 
>information in the stored file to reconstruct a true picture of the 
>original sampling instant and encoding delays. So, unless we require 
>the sender to scan the entire file (which is impractical), we may 
>need to accept the fact that it will have to make an arbitrary guess 
>about the sampling instant that could cause some offsets to end up 
>being negative.

I'm OK with this.

At 13:39  -0700 28/07/06, Ron Frederick wrote:
>I suppose all of this is already determined, though, if you go with 
>the model that the streaming server is already computing some 
>mapping between its RTP timestamps and NTP time in RTCP. The offset 
>it reports would be the difference between that NTP time and the NTP 
>time of when it actually sent the packet, converted back to RTP 
>timestamp units. So, the question of what the "apparent capture 
>time" is was already determined when the streaming server picked 
>what NTP timestamp to use in RTCP. Has there been any guidance in 
>any of the specs for how a streaming server should choose this value 
>when sending stored content?

I am not aware of any guidance on picking NTP values for servers (if 
that is the question!).

>
>Other than traffic smoothing, what other strong factor could
>offset/deviate the tranmission of an RTP packet?
>
>It looks like this extension may not be of much use for audio/text media
>types. I'd like to know if there are any good use cases for audio/text
>media types. I don't think it is very common that a single speech/audio
>frame consume multiple RTP packets.

I agree that audio tends to be much less in need of smoothing, and 
(so far) does not have re-ordering issues.  However, re-transmission 
is used.
-- 
David Singer
Apple Computer/QuickTime

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Ron Frederick <[email-masked]>
Date: Tue, 01 Aug 2006 16:54:00 -0700

Dave Singer wrote:
> I'm sorry, I was on vacation when this all erupted.  I've tried to read 
> the entire thread, and respond below.
> 
> I will separate the transmission offset section into a separate draft.  
> It clearly needs separate handling.

That sounds good to me.

> (a) traffic smoothing -- in particular, video I -frames tend to be large 
> and therefore represent a bandwidth spike that both needs spreading out, 
> and may involve distorting the send time of adjacent frames as well
> (b) I/B/P video, where the dependencies mean that frames have to be sent 
> in decoding order but with presentation timestamps.  Then the timestamps 
> are not even increasing with sequence number.

This explanation is very useful, and it would be good to explicitly 
include it in the draft. Points (a) and (b) generally only apply to 
video and not audio or text, but one could imagine other payloads 
carried over RTP also being bursty in nature and needing the smoothing 
mentioned in (a). A shared whiteboard application comes to mind, for 
instance, where you might be pasting in a bitmap and need to spread it 
out over several packets. Point (c) could apply to any type of RTP 
traffic that was flexible enough to allow retransmissions.

> That is one possible use.  There are other cases (e.g. when attempting 
> to use any kind of adaptive client-server buffer management).  I 
> proposed this because I had seen enough complaints/concerns over the 
> years that there wasn't a standard place to inform the client about 
> actual transmission times.

Most buffer management schemes rely on setting an appropriate playout 
buffer, and that really can be completely driven from the presentation 
times. Can you give an example for how one might use transmission time 
in a scheme like that?

> My intention was that the offsets not have much meaning by themselves.  
> Classically a pair of RTP packets with timestamps R2 and R1 are 
> transmitted with a time interval between them of (R2 - R1). The offsets 
> O2 and O1 would be such that you would know that the original 
> transmission interval was (R2 + O2) - (R1 + O1).

I'm worried that this definition is too vague. Also, while you want this 
to hold between two RTP packets in a single stream, would you also 
expect it to hold between two RTP packets in different streams, such as 
the audio & video streams which are part of the same RTSP session? If 
we're going to require an invariant like this, we need to specify how 
widely it applies. Tying the definition to the NTP timestamp is one way 
to be very clear about how these values should be computed, and also has 
the benefit of provide meaningful relative transmission times even 
across different RTP streams.

> actually rather limiting and limited.  Talking about the "sampling instant"
> a) is a backwards-looking statement.  To me, it's better to express to 
> the receiver what they need to know to do "the right thing" with what 
> they get, rather than try to express something about how the stream was 
> made (which is none of their business, in some sense).
> actually happen for some payload types (e.g. MIDI streamed from a file) 
> and even for other payload types, may never have happened (e.g. 

I agree with all these points about sampling instants. My way of 
thinking about this for non-live traffic goes back to something I said 
earlier about ideally wanting to make even stored streams look to a 
receiver exactly like a live stream, but in practice that's often 
difficult to do. I'd prefer to come up with a definition that didn't 
rely on the notion of "sampling instant" at all.

That said, though, RFC 3550 does define the RTP timestamp field as 
follows: "The timestamp reflects the sampling instant of the first octet 
in the RTP data packet." It then goes on to discuss the way that these 
times are periodically paired with wall clock times in RTCP SR packets 
and how that can be used to do synchronization between different RTP 
streams. So, all RTP traffic must have a "virtual" sampling instant 
already chosen by the sender for each packet, and an association between 
that RTP timestamp and a wall clock time which is consistent across 
multiple RTP streams that need to be synchronized. In defining the RTP 
offset, we don't have to recreate any of this -- we can rely on the 
mapping which has already been established and have the offset only 
provide the additional information necessary to determine wall clock 
transmission time.

> debate is over whether we can define what (R2 + O2) means by itself.

I agree with this.

>> "capture time" of the data (represented in the RTP timestamp) and its 
>> send time. Both of the timestamps are local to the same machine. It's 
>> true that a receiver would not be able to relate that send time to an 
>> absolute time on its local clock without some form of clock 
>> field just doesn't seem to really add much value. It would still have 
>> to make sure that this offset was constant across all packets in the 
>> RTP stream, and the only way I can see of it doing that is to know 
>> what the absolute offset really was and then "normalize" it somehow by 
>> adding/subtracting some fixed constant. Why complicate things in that 
>> way, though?
>>
>> The other advantage I see to an absolute offset is that it would allow 
>> a receiver to be able to meaningfully compare send times between 
>> packets arriving on different RTP streams from the same sender. This 
>> would work even in cases where there was no time synchronization 
>> between the sender and receiver at all. With the relative offset, this 
>> wouldn't be possible unless we could guarantee that the fixed offset 
>> was the same for all RTP streams, and I would not expect that to be 
>> the case.
> 
> take an RTP stream that has a recent RTCP sender report relating R0 to 
> N0, and a packet sent after that with timestamp R1.  Nominally this is 
> sent at (ignoring timescale conversion issues) N1 = (N0 + (R1 - R0)).  
> If it was sent at a different time, Na, then the offset value is (Na - N1).

Yes. Doing this means that one can take (R1 + O1) and get an RTP 
timestamp that can be converted back to Na, which is a meaningful 
absolute wall clock time. This can be done for each individual packet, 
and your invariant that (R2 + O2) - (R1 + O1) should represent 
transmission time difference of the two packets holds. In fact, this 
invariant now holds even when comparing packets from two different RTP 
streams, as long as their mappings to NTP time were intended to be 
synchronized.

Exactly what the offsets turn out to be will depend on how the sender 
originally chose N0 (the reported "sampling instant" of the first RTP 
packet), which is beyond our control. However, as long as the sender has 
a way to measure the wall clock time when transmitting a packet, it is 
very clear what offset it should be reporting.

>> Ideally, I'd prefer a design where a video-on-demand stream was forced 
>> to look exactly the same as a live stream to make it easier on 
>> receivers. However, we also need to consider what information a sender 
>> has available, and a storage server may not have enough information in 
>> the stored file to reconstruct a true picture of the original sampling 
>> instant and encoding delays. So, unless we require the sender to scan 
>> the entire file (which is impractical), we may need to accept the fact 
>> that it will have to make an arbitrary guess about the sampling 
>> instant that could cause some offsets to end up being negative.
> 
> I'm OK with this.
> 
>> the model that the streaming server is already computing some mapping 
>> between its RTP timestamps and NTP time in RTCP. The offset it reports 
>> would be the difference between that NTP time and the NTP time of when 
>> it actually sent the packet, converted back to RTP timestamp units. 
>> So, the question of what the "apparent capture time" is was already 
>> determined when the streaming server picked what NTP timestamp to use 
>> in RTCP. Has there been any guidance in any of the specs for how a 
>> streaming server should choose this value when sending stored content?
> 

I'm not either, but fundamentally every RTP application out there ends 
up needing to answer this question (if it conforms to the spec and 
properly implements RTCP SRs). So, I think we can leverage that when we 
define RTP offsets.
-- 
Ron Frederick
[email-masked]