Subject: Re: [tsvwg] SCTP NDATA draft

From: Randy Stewart <[email-masked]>
Date: Sat, 1 Nov 2014 11:05:02 -0400

Mo:

Comments in-line below..

> On Oct 28, 2014, at 9:18 PM, Mo Zanaty (mzanaty) <[email-masked]> wrote:
> 
> Hi,
> While reviewing this draft for becoming mandatory in rtcweb data channels, I see two issues, which are independent of rtcweb.
> 
> 1. If negotiated, NDATA must be used for all messages. The NDATA format requires 8 more bytes than DATA (for MID+FSN). But these extra 8 bytes are only useful for fragmented messages. To avoid this overhead for small non-fragmented messages, can we relax / optimize this rule to be NDATA must be used for all *fragmented* messages, while non-fragmented messages continue to use DATA?
> 
> Proposed change to the next-to-last paragraph of section 1.1:
> 
> OLD:
>    The support of the N-DATA chunk is negotiated during the association
>    setup using the Supported Extensions Parameter as defined in
>    [RFC5061].  If N-DATA support has been negotiated for an association
>    N-DATA chunks are used for all user-messages and no DATA chunks.  It
>    should be noted, that an SCTP implementation needs to support the
>    coexistence of associations using DATA chunks and associations using
>    N-DATA chunks.
> 
> NEW:
>    The support of the N-DATA chunk is negotiated during the association
>    setup using the Supported Extensions Parameter as defined in
>    [RFC5061].  If N-DATA support has been negotiated for an association
>    N-DATA chunks are used for all fragmented user messages, and
>    DATA chunks are used for all non-fragmented user messages.  It
>    should be noted, that an SCTP implementation needs to support the
>    coexistence of associations using DATA chunks and associations using
>    N-DATA chunks.

I will let Michael comment on the above, I think he put that in place to be
consistent, but I will leave him to answer..

> 
> 2. Fragmented messages can’t be interleaved. This significantly dilutes the value of NDATA. Apps that truly care about head of line blocking can’t rely on NDATA at all. They must implement app layer fragmentation with messages well below common MTUs to avoid blocking, hence defeating NDATA. NDATA is arguably harmful for these apps, since it is 8 bytes closer to exceeding the MTU, forcing SCTP fragments and hence risking HOL blocking. To avoid this, can we relax this rule to allow interleaving fragmented messages, since MID is already part of NDATA?
> 
> Proposed change to the last paragraph of section 3.1:
> 
> OLD:
>    The sender MUST NOT have more than one ordered fragmented message
>    being produced in any one stream.  The sender MUST NOT have more than
>    one un-ordered fragmented message being produced in any one stream.
>    The sender MAY have one ordered and one unordered fragmented message
>    being produced within a single stream.  At any time multiple streams
>    MAY be producing an ordered or unordered fragmented message.
> 
> NEW:
>    The sender MAY have more than one ordered fragmented message
>    being produced in any one stream.  The sender MAY have more than
>    one un-ordered fragmented message being produced in any one stream.
>    The sender MAY have multiple ordered and unordered fragmented messages
>    being produced within a single stream.  At any time multiple streams
>    MAY be producing multiple ordered or unordered fragmented messages.

The above restriction’s were first put into place *before* a MID was added. I notice
that there is not a lot on the MID in the draft.. it needs to be specified as a monotonically 
incrementing number, that is incremented every time a fragmented message is sent on any stream. Since
you need a unique identifier for unordered to make it so you can have multiple in flight.

As to the ordered side, I don’t see that you can lift that restriction and here is why,
Consider this simple scenario:

BE = B
TSN = 1
SSN = 1
STR = 1
MID = 1
FSN = 0
<—

BE = B
TSN = 2
SSN = 2
STR = 1
MID = 2
FSN = 0
<—

BE = E
TSN = 3
SSN = 2
STR = 1
MID = 2
FSN = 1
<—

BE = E
TSN = 4
SSN = 1
STR = 1
MID = 1
FSN = 1
<—

Which is what could happen if you loosen the restriction upon ordered messages.

First of all from the user perspective, you gain *nothing*. By definition, an ordered
message will *not* allow STR=1, SSN=2 to be delivered *before* STR=1, SSN=1. So 
by allowing multiple to be assigned TSN’s you *gain* nothing from the user perspective.

Now what you do gain however is a deadlock scenario (which can also be used as a DOS 
attack). Now the above illustrates a simple set of 2 fragment messages, this is *not* really what
NDATA addresses, since where you see HOL is where you have 1Megabyte messages (or higher). An
application sending 3k messages is *not* going to see HOL and inside the same stream you
are enforcing HOL by definition.. But what happens if we *expand* SSN from its small
2800 bytes size, into a 500,000 byte message. Assuming a rwnd of 256,000 bytes you are deadlocked.
You will have filled the rwnd and have *no* way to deliver SSN=2 since you are waiting for
SSN=1 so you can *not* start the partial delivery API..

This can be used as a dead-lock or an attack vector. Get a bot to setup an association, send the
first of SSN=1, now fill the rwnd with SSN=2 and that association is hopelessly deadlocked and
cannot make forward progress.

This is why the restriction was placed on ordered messages.

For the unordered messages, we did add the same restriction. Why, well there was no MID, which
means that there was no way for the receiver to tell two fragmented messages apart. But there
is still a problem with an unordered message. When you start the partial delivery API, you now
may be seeing two unordered messages at the same time on the same stream. And in order to prevent
the deadlock described above you would *need* to start the PD-API in order to make forward progress
(imagine two 40Meg unordered messages being sent a the same time). Right now the Socket API has
no way to tell the difference.. i.e. it has no access to the MID.. so it can’t tell which
message is which.

We *could* possibly loosen the unordered restriction, but it will also require work on getting
the socket API to change.. which I don’t think Michael or I thought was worth it since the goal
of the draft was to get rid of the inter-stream HOL. Streams are cheap, so if you really wanted
to have 10 unordered 40Meg messages, you could easily put them in each one.. 

It would be quite a change to the Socket-API in order to do this, possibly effecting the
delivery mode.. but I suppose we could do it if the WG feels that its that critical.. I
personally don’t think it is worth the gain..

Remember, and maybe this needs clarification, the no “two” in flight at the same time
is really in respect to the TSN assignment.. i.e. if I send Message 1 and Message 2, I must
assign all the TSN’s of message 1 before starting to assign TSN’s for message 2. You
always have the possibility that TSN’s will arrive out of order.. 

No matter if the WG wants to remove the restriction on un-ordered and complicate the socket-API
I don’t see how one could remove the restriction on ordered messages.

Best wishes

R

> Regards,
> Mo

-----
Randall Stewart
[email-masked]

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 4 Nov 2014 21:05:10 +0000

Hi Randy,

Good points about filling rwnd and the API lacking MID.

The API issue seems like it could be fixed by just exposing MID in partial
deliveries. Is that very complex in the implementation? It seems rather
straightforward in the API/spec.

The rwnd problem already exists without any proposed changes, since rwnd
is for the entire association, right? For example, if you create 2
streams, send a huge message (>rwnd) on stream1 and a tiny message (<MTU)
on stream2, the default stream schedulers will block the tiny message on
stream2 until rwnd opens, right? You would have to create streams with
non-default schedulers like ³priority² or ³fair bandwidth² to avoid
blocking.

Unfortunately, WebRTC data channels have no API to set stream schedulers,
so they always get the default. More discussion of NDATA in the context of
WebRTC data channels here:
http://www.ietf.org/mail-archive/web/rtcweb/current/msg13351.html
(Sorry about scattering this thread between two lists, but tsvwg and
rtcweb are rather disjoint lists, so chairs may complain about cross
posting.)

As for malicious attacks, SCTP is not a secure protocol, so on-path
adversaries can easily flood, deadlock, or otherwise hose either end of
the association, independent of NDATA or any of the proposed changes.
Hence a major reason why WebRTC requires SCTP over DTLS.

Cheers,
Mo

On 11/1/14, 11:05 AM, Randy Stewart <[email-masked]> wrote:

Mo:

Comments in-line below..

> While reviewing this draft for becoming mandatory in rtcweb data
>channels, I see two issues, which are independent of rtcweb.
> 
> 1. If negotiated, NDATA must be used for all messages. The NDATA format
>requires 8 more bytes than DATA (for MID+FSN). But these extra 8 bytes
>are only useful for fragmented messages. To avoid this overhead for small
>non-fragmented messages, can we relax / optimize this rule to be NDATA
>must be used for all *fragmented* messages, while non-fragmented messages
>continue to use DATA?
> 

> 
> 2. Fragmented messages can¹t be interleaved. This significantly dilutes
>the value of NDATA. Apps that truly care about head of line blocking
>can¹t rely on NDATA at all. They must implement app layer fragmentation
>with messages well below common MTUs to avoid blocking, hence defeating
>NDATA. NDATA is arguably harmful for these apps, since it is 8 bytes
>closer to exceeding the MTU, forcing SCTP fragments and hence risking HOL
>blocking. To avoid this, can we relax this rule to allow interleaving
>fragmented messages, since MID is already part of NDATA?
> 

The above restriction¹s were first put into place *before* a MID was
added. I notice
that there is not a lot on the MID in the draft.. it needs to be specified
as a monotonically 
incrementing number, that is incremented every time a fragmented message
is sent on any stream. Since
you need a unique identifier for unordered to make it so you can have
multiple in flight.

As to the ordered side, I don¹t see that you can lift that restriction and
here is why,
Consider this simple scenario:

BE = B
TSN = 1
SSN = 1
STR = 1
MID = 1
FSN = 0
<‹

BE = B
TSN = 2
SSN = 2
STR = 1
MID = 2
FSN = 0
<‹

BE = E
TSN = 3
SSN = 2
STR = 1
MID = 2
FSN = 1
<‹

BE = E
TSN = 4
SSN = 1
STR = 1
MID = 1
FSN = 1
<‹

Which is what could happen if you loosen the restriction upon ordered
messages.

First of all from the user perspective, you gain *nothing*. By definition,
an ordered
message will *not* allow STR=1, SSN=2 to be delivered *before* STR=1,
SSN=1. So 
by allowing multiple to be assigned TSN¹s you *gain* nothing from the user
perspective.

Now what you do gain however is a deadlock scenario (which can also be
used as a DOS 
attack). Now the above illustrates a simple set of 2 fragment messages,
this is *not* really what
NDATA addresses, since where you see HOL is where you have 1Megabyte
messages (or higher). An
application sending 3k messages is *not* going to see HOL and inside the
same stream you
are enforcing HOL by definition.. But what happens if we *expand* SSN from
its small
2800 bytes size, into a 500,000 byte message. Assuming a rwnd of 256,000
bytes you are deadlocked.
You will have filled the rwnd and have *no* way to deliver SSN=2 since you
are waiting for
SSN=1 so you can *not* start the partial delivery API..

This can be used as a dead-lock or an attack vector. Get a bot to setup an
association, send the
first of SSN=1, now fill the rwnd with SSN=2 and that association is
hopelessly deadlocked and
cannot make forward progress.

For the unordered messages, we did add the same restriction. Why, well
there was no MID, which
means that there was no way for the receiver to tell two fragmented
messages apart. But there
is still a problem with an unordered message. When you start the partial
delivery API, you now
may be seeing two unordered messages at the same time on the same stream.
And in order to prevent
the deadlock described above you would *need* to start the PD-API in order
to make forward progress
(imagine two 40Meg unordered messages being sent a the same time). Right
now the Socket API has
no way to tell the difference.. i.e. it has no access to the MID.. so it
can¹t tell which
message is which.

We *could* possibly loosen the unordered restriction, but it will also
require work on getting
the socket API to change.. which I don¹t think Michael or I thought was
worth it since the goal
of the draft was to get rid of the inter-stream HOL. Streams are cheap, so
if you really wanted
to have 10 unordered 40Meg messages, you could easily put them in each
one.. 

It would be quite a change to the Socket-API in order to do this, possibly
effecting the
delivery mode.. but I suppose we could do it if the WG feels that its that
critical.. I
personally don¹t think it is worth the gain..

Remember, and maybe this needs clarification, the no ³two² in flight at
the same time
is really in respect to the TSN assignment.. i.e. if I send Message 1 and
Message 2, I must
assign all the TSN¹s of message 1 before starting to assign TSN¹s for
message 2. You
always have the possibility that TSN¹s will arrive out of order..

No matter if the WG wants to remove the restriction on un-ordered and
complicate the socket-API
I don¹t see how one could remove the restriction on ordered messages.

Best wishes

R

-----
Randall Stewart
[email-masked]

---

From: Randall Stewart <[email-masked]>
Date: Tue, 4 Nov 2014 16:17:31 -0500

> non-default schedulers like ³priority² or ³fair bandwidth² to avoid

It may be slow,  but you *can* make forward progress. I.e. you *can* deliver
messages from a stream say stream 1, even if stream 2 is hogging
up rwnd space. What your change breaks is  you create a situation where
you *can not* make forward progress at all. This is vastly different then
the current RFC or draft. You can *always* make forward progress, it may be slow
but you can.

> rtcweb are rather disjoint lists, so chairs may complain about cross

Of course that again is an API problem 

Just as in the un-ordered case for SCTP :-D

> Hence a major reason why WebRTC requires SCTP over DTLS.

The protocol is designed in general to *not* deadlock. Adding a deadlock to
the spec is something I could never agree with.

R

> I don¹t see how one could remove the restriction on ordered messages.

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Karen Elisabeth Egede Nielsen <[email-masked]>
Date: Tue, 4 Nov 2014 22:46:30 +0100

Hi,

>The above restriction’s were first put into place *before* a MID was
added. I
>notice that there is not a lot on the MID in the draft.. it needs to be
specified
>as a monotonically incrementing number, that is incremented every time a
>fragmented message is sent on any stream. Since you need a unique
>identifier for unordered to make it so you can have multiple in flight.
>
[Karen ]
I think that it is key to have the word "produced" be clarified.

It would be extremely inefficient if  this was intended to mean that only
the fragments
of one message (or one unordered and one ordered - meaning two messages)
may be in flight on
each stream.  This might be ok, when the use case scenario in mind is a
message (to be fragmented) which is larger than the RCV buffer,
but not in general.  E.g. not when there are messages  larger than PMTU
but << RCV buffer.

I think that it would be important to have this clarified as to explain
that what is meant is that only fragments of unordered messages may
overtake other message fragments or messages written into the stream in
respect to the order in which TSN numbers  are assigned and further that
the mutual sequential order in between sequentially written unordered
messages are maintained (as the order  is maintained for the messages
written for in-order delivery).

BR, Karen