Subject: [MMUSIC] RTSP: Inconsistent RTP sequence number handling in C.3

From: Magnus Westerlund <[email-masked]>
Date: Thu, 05 Mar 2009 15:06:20 +0100

Hi,

There was raised that the RTP sequence number handling was inconsistent.
I have tried to clarify this but being more explicit about when there is
possible to have an exception to that next RTP packet is the next higher
sequence number modulo 65536. So please review this text.

C.3.  Handling Media Clock Time Jumps in the RTP Media Layer

   RTSP allows media clients to control selected, non-contiguous
   sections of media presentations, rendering those streams with an RTP
   media layer [RFC3550].  Two cases occur, the first is when a new PLAY
   request replaces an old ongoing request and the new request results
   in a jump in the media.  This should produce in the RTP layer a
   continuous media stream.  A client may also directly following a
   completed PLAY request perform a new PLAY request.  This will result
   in some gap in the media layer.  The below text will look into both
   cases.

   A PLAY request that replaces a ongoing request allows the media layer
   rendering the RTP stream without being affected by jumps in media
   clock time.  The RTP timestamps for the new media range is set so
   that they become continuous with the previous media range in the
   previous request.  The RTP sequence number for the first packet in
   the new range will be the next following the last packet in the
   previous range, i.e. monotonically increasing.  The goal is to allow
   the media rendering layer to work without interruption or
   reconfiguration across the jumps in media clock.  This should be
   possible in all cases of replaced PLAY requests for media that has
   random-access properties.  In this case care is needed to align
   frames or similar media dependent structures.

   In cases where jumps in media clock time are a result of RTSP
   signalling operations arriving after a completed PLAY operation, the
   request timing will result in that media becomes non-continuous.  The
   server becomes unable to send the media so that it arrive timely and
   still carry timestamps to make the media stream continuous.  In these
   cases the server will produce RTP streams where there are gaps in the
   RTP timeline for the media.  In such cases, if the media has frame
   structure, aligning the timestamp for the next frame with the
   previous structure reduces the burden to render this media.  The gap
   should represent the time the server hasn't been serving media, e.g.
   the time between the end of the media stream or a PAUSE request and
   the new PLAY request.  In these cases the RTP sequence number would
   normally be monotonically increasing across the gap.

   For RTSP sessions with media that lacks random access properties,
   like live streams, any media clock jump is commonly result of
   correspondingly long pause of delivery.  The RTP timestamp will have
   increased in direct proportion to the duration of the paused
   delivery.  Note also that in this case the RTP sequence number should
   be the next packet number.  If not, the RTCP packet loss reporting
   will indicate as loss all packets not received between the point of
   pausing and later resuming.  This may trigger congestion avoidance
   mechanisms.  An allowed exception from the above recommendation on
   monotonically increasing RTP sequence number is live media streams,
   likely being relayed.  In this case, when the client resumes
   delivery, it will get the media that is currently being delivered to
   the server itself.  For this type of basic delivery of live streams
   to multiple users over unicast, individual rewriting of RTP sequence
   numbers becomes quite a burden.  For solutions that anyway caches
   media, timeshifts, etc, the rewriting should be a minor issue.

   The goal when handling jumps in media clock time is that the provided
   stream is continuous without gaps in RTP timestamp or sequence
   number.  However, when delivery has been halted for some reason the
   RTP timestamp when resuming MUST represent the duration the delivery
   was halted.  RTP sequence number MUST generally be the next number,
   i.e. monotonically increasing modulo 65536.  For media resources with
   the properties Time-Progressing and Time-Duration=0.0 the server MAY
   create RTP media streams with RTP sequence number jumps in them due
   to client first halting delivery and later resuming it (PAUSE and
   then later PLAY).  However, servers utilizing this exception must
   take into consideration the resulting RTCP receiver reports that
   likely contains loss report for all the packets part of the
   discontinutity.  A client can not rely on that a server will align
   when resuming playing even if it is RECOMMENDED.  The RTP-Info header
   will provide information on how the server acts in each case.

      We cannot assume that the RTSP client can communicate with the RTP
      media agent, as the two may be independent processes.  If the RTP
      timestamp shows the same gap as the NPT, the media agent will
      assume that there is a pause in the presentation.  If the jump in
      NPT is large enough, the RTP timestamp may roll over and the media
      agent may believe later packets to be duplicates of packets just
      played out.  Having the RTP timestamp jump will also affect the
      RTCP measurements based on this.

   As an example, assume a RTP timestamp frequency of 8000 Hz, a
   packetization interval of 100 ms and an initial sequence number and
   timestamp of zero.

      C->S: PLAY rtsp://example.com/fizzle RTSP/2.0
        CSeq: 4
        Session: abcdefg
        Range: npt=10-15
        User-Agent: PhonyClient/1.2

      S->C: RTSP/2.0 200 OK
        CSeq: 4
        Session: abcdefg
        Range: npt=10-15
        RTP-Info: url="rtsp://example.com/fizzle/audiotrack"
                  ssrc=0D12F123:seq=0;rtptime=0

   The ensuing RTP data stream is depicted below:

      S -> C: RTP packet - seq = 0,  rtptime = 0,     NPT time = 10s
      S -> C: RTP packet - seq = 1,  rtptime = 800,   NPT time = 10.1s
       . . .
      S -> C: RTP packet - seq = 49, rtptime = 39200, NPT time = 14.9s

   Immediately after the end of the play range, the client follows up
   with a request to PLAY from a new NPT.

   C->S: PLAY rtsp://example.com/fizzle RTSP/2.0
         CSeq: 5
         Session: abcdefg
         Range: npt=18-20
         User-Agent: PhonyClient/1.2

   S->C: RTSP/2.0 200 OK
         CSeq: 5
         Session: abcdefg
         Range: npt=18-20
         RTP-Info: url="rtsp://example.com/fizzle/audiotrack"
                   ssrc=0D12F123:seq=50;rtptime=40100

   The ensuing RTP data stream is depicted below:

      S->C: RTP packet - seq = 50, rtptime = 40100, NPT time = 18s
      S->C: RTP packet - seq = 51, rtptime = 40900, NPT time = 18.1s
       . . .
      S->C: RTP packet - seq = 69, rtptime = 55300, NPT time = 19.9s

   In this example, first, NPT 10 through 15 is played, then the client
   request the server to skip ahead and play NPT 18 through 20.  The
   first segment is presented as RTP packets with sequence numbers 0
   through 49 and timestamp 0 through 39,200.  The second segment
   consists of RTP packets with sequence number 50 through 69, with
   timestamps 40,100 through 55,200.  While there is a gap in the NPT,
   there is no gap in the sequence number space of the RTP data stream.

   The RTP timestamp gap is present in the above example due to the time
   it takes to perform the second play request, in this case 12.5 ms
   (100/8000).

Cheers

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Jaehwan Kim" <[email-masked]>
Date: Thu, 12 Mar 2009 10:51:35 +0900

Magnus wrote:
>An allowed exception from the above recommendation on
In ideal world, when media server follows the basic rule, 
there is no difficulty to rewrite sequence number and timestamp at
relay server. 

When packet loss happened between media server and relay, 
relay server might be able to reflect it into the outgoing RTP stream 
to client and it looks not difficult. But I am not sure if it is right
because it is wrong from a statistics point of view. One stakeholder,
network department in one mobile operator, might want to see only
the loss rate between relay server and client. If true, it looks good
to use monotonic seq increment rule to skip loss packet. But still
there is time gap, eventually it is regarded as loss at decoder level.
Then, RTCP statistics reflects the actual situation, no loss to client.
Likewise, IMHO, it looks good for server to take into account only the
session between relay and client. 

It even supports more complicated situation making
one RTSP session from more than one media server such as in a RTSP TV
application. And if the rule is simple, the happiest guy would be
player.

Therefore, I prefer your old but simple and universal advice:
1) RTP layer is assumed to be independent from RTSP layer
2) RTP timestamps must reflect real time
3) RTP sequence number must be monotonically increased

Then, we might be able to minimize this section with just only 
a few examples.

>was halted. RTP sequence number MUST generally be the next number,
This and the following clauses failed to make me understood. IMHO,
a special care is needed to make an exception for a rule.
Is it really hard to avoid this exception?

>                  ssrc=0D12F123:seq=0;rtptime=0 
Missing semicolon between url and ssrc parameters.

>            ssrc=0D12F123:seq=50;rtptime=40100
I had to reconsider this example:
The gap between the previous seq 49 and seq 50 is very small
(around 112ms from the gap between rtptime 40100-39200). 
But still, packetization interval is 100 ms. Then, I guess 
server stopped RTP process when it received RTSP PLAY(cseq 5)
and it was less then 100ms from seq 49.

If not, this example is wrong because already seq=50
was sent before server makes RTP-Info. Then, RTP-Info should have
Seq=51 which helps player to skip previous RTP packet having seq=50.

Likewise, to show the actual situation player and server suffers,
I prefer using a practical example using seq skip at RTSP layer but
still rule 3) is maintained at RTP layer.

Regards,
Jaehwan Kim
Senior Development Manager / Xenon Platform Architect
Vidiator(r) - Mobilizing Interactive Media(tm)

7th Fl. AnnJay BLDG 718-2 
YeokSam-Dong, KangNam-Gu, 135-920, Seoul
 
d: +82.70.7012.2540
f:  +82.2.569.5348
m:+82.10.6225.0619
e:  [email-masked]
w: www.vidiator.com

The message represents the personal views and opinion of the individual
sender and under no circumstances represents those of HWL or its Group
Companies. The shareholders, directors and management of HWL and any of
its Group Companies accept no responsibility and accordingly shall have
no liability to any party whatsoever with respect to the contents of
this message. 
This message (including any attachments) is intended only for the use of
the addressee(s) named above. It may contain information that is
PRIVILEGED and CONFIDENTIAL and should not be read, copied or otherwise
used by any other person. 
If you are not the intended recipient, you are hereby notified that any
use, retention, disclosure, copying, printing, forwarding or
dissemination of this communication is strictly prohibited. If you have
received this communication in error, please erase all copies of the
message and its attachments and notify us immediately.

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 12 Mar 2009 16:51:27 +0100

Hi Jaehwan,

See inline.

> Is it really hard to avoid this exception?

First of all the rewriting you are talking about in cases of packet loss
is not always possible. It will depend on the RTP payload format if you
can remove the clients view on that the packet has been lost between
media sender source and client. So when it comes to rewriting I think
the safest is usually to represent the loss through to the client,
otherwise the relaying server needs to be media aware. That will
definitely not make your relaying easier.

On the question if rewriting always is a to hard to say about. For
multicast sessions it is impossible. But then there are other protocols
then RTSP that could be used to imply this. For unicasted I agree that
it is not technically infeasible. I think the people actually running
the system that needs to answer this.

> Missing semicolon between url and ssrc parameters.

No, the syntax specified for the RTP-Info header is:

RTP-Info         =  "RTP-Info" HCOLON [rtsp-info-spec
                       *(COMMA rtsp-info-spec)]
   rtsp-info-spec   =  stream-url 1*ssrc-parameter
   stream-url       =  "url" EQUAL DQ RTSP-REQ-Ref DQ
   ssrc-parameter   =  LWS "ssrc" EQUAL ssrc HCOLON
                       ri-parameter *(SEMI ri-parameter)
   ri-parameter     =  ("seq" EQUAL 1*DIGIT)
                    /  ("rtptime" EQUAL 1*DIGIT)
                    /  generic-param

Which means that there is only a linear white space between the url and
the SSRC value and its parameters. We have an unfortunate line break
here due to that the construct does not fit on a single line.

> and it was less then 100ms from seq 49.

There is actually a missing PLAY_NOTIFY message here. We should add it
after the last RTP packet (RTP seq=49). The server has finished the
playout of the first play and reverted to READY state.

After the server has completed the first PLAY request seq=4 the client
sends a new one. The delivery and processing of that results in that the
 server sends the next packet 100/8000 s after the end of the previous
media.

> Seq=51 which helps player to skip previous RTP packet having seq=50.

Please note, this example is not for the special case, but the usual
case simply adding the pause time in and send the next RTP Sequence
number. If does have a media level seek also, from 15 to 18 so it is for
an on demand stream.

An example for the exception (PLAY without range header) would rather
say that the next RTP packet to send would be:

RTP packet - seq = 51, rtptime = 40800, NPT time= 15.1

> still rule 3) is maintained at RTP layer.

What do you mean with seq skip at RTSP layer and rule 3?

Are you meaning a case where the incoming traffic has its RTP sequence
number rewritten to meet the strict sequence number rule.

Cheers

Magnus Westerlund

---

From: "Jaehwan Kim" <[email-masked]>
Date: Fri, 13 Mar 2009 12:19:12 +0900

Hi Magnus,
Thank you for your quick and kind answer.
See inline:

>here due to that the construct does not fit on a single line.
Indeed. BTW, I found one issue which was sent to you in a separate
e-mail.

>playout of the first play and reverted to READY state.
I guess no PLAY_NOTIFY because it is just transition to another PLAY
state.
"server stopped RTP process" meant server stopped the ongoing PLAY state
and prepare new PLAY state according to PLAY with different range.

>number rewritten to meet the strict sequence number rule.
Maybe I should correct my words:
"...example when the seq in RTP-Info has gap from the seq of the latest
RTP packet sent before PLAY method received, while rule 3), RTP sequence
number must be monotonically increased, is maintained at RTP layer."

I was happy with the following simple 3 rules you told us until
exception came out :)

Regards,
Jaehwan Kim

 
d: +82.70.7012.2540
f:  +82.2.569.5348
m:+82.10.6225.0619
e:  [email-masked]
w: www.vidiator.com

---

From: "Jaehwan Kim" <[email-masked]>
Date: Fri, 13 Mar 2009 12:37:56 +0900

Hi Magnus,
I found that the following seems not existing any more in new draft.

In RF2326, 4.2 Message Headers says
   See [H4.2]
In RFC2616(H4.2)
   Header fields can be
   extended over multiple lines by preceding each extra line with at
   least one SP or HT.

Although I found some description regarding multiple lines from
2326bis-20, its usage looks changed with limitation. According to the
new rule, we can't
use multiple lines for RTP-Info, for example, which is the most dominant
usecase in the current implementation.

In fact, the above rule is very common usage in case of long header
line, especially for the purpose of easy debugging. I am not sure if it
is just a mistake during editing or there is some background. Reference
to [H4.2] has existed until 2326bis-19. Anyway, please kindly provide
any background story if it has to be. (I failed to search the reason...)

Regards,
Jaehwan Kim

 
d: +82.70.7012.2540
f:  +82.2.569.5348
m:+82.10.6225.0619
e:  [email-masked]
w: www.vidiator.com

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 20 Mar 2009 10:59:33 -0700

> I guess no PLAY_NOTIFY because it is just transition to another PLAY

No, that was not intended, I intended for the play request to complete
and as soon as the client has detected the completness of the request it
issues another one.

> and prepare new PLAY state according to PLAY with different range.

Yes, that is intended, but it has transited to Ready state before the
second PLAY request happens.

> I was happy with the following simple 3 rules you told us until

Yes, and this is clearly a question to discuss. I do like feedback from
more people on this issue.

>>> 3) RTP sequence number must be monotonically increased

I agree that the client becomes simpler if the exception doesn't exist.
The question is if this really is realistic restriction. Unfortunately I
 expect that clients needs to handle this exception due to live stream
forwarding.

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 20 Mar 2009 11:06:04 -0700

Hi Jaehwan,

>    extended over multiple lines by preceding each extra line with at

This was intentional as we don't want any generic rules that allows
people to interpret spaces as allowing folding of lines, unless
explicitly allowed by the syntax for any header.

> usecase in the current implementation.

If you look at the current RTP-Info syntax:

you should look up what HCOLON, COMMA and EQUAL means:

   HCOLON          =  *( SP / HT ) ":" SWS
   EQUAL    =  SWS "=" SWS ; equal
   COMMA    =  SWS "," SWS ; comma
   SEMI     =  SWS ";" SWS ; semicolon
   COLON    =  SWS ":" SWS ; colon

They all uses the SWS contstruct that if you look at it actually allows
for line folding.

   LWS             =  [CRLF] 1*( SP / HT )
   SWS             =  [LWS] ; sep whitespace

So this has not been disallowed, only been made explicit in the syntax.

> any background story if it has to be. (I failed to search the reason...)

So not disagreement about allowing line folding at appropriate points.

Cheers

Magnus Westerlund