Subject: [Detnet] Mirja Kühlewind's Discuss on draft-ietf-detnet-architecture-11: (with DISCUSS and COMMENT)

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 20 Feb 2019 05:05:51 -0800

Mirja Kühlewind has entered the following ballot position for
draft-ietf-detnet-architecture-11: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-detnet-architecture/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

Thanks for addressing the tsv-art review comments  (and big thanks to Michael
and David!) and all the work done so far! I think the document is in good shape
and I only have one minor comment that I would like to see addressed/more
explicitly spelled out. However, this should be done quickly with potentially
2-3 small changes in the draft. See below.

Given that DetNet traffic is often assumed to be not congestion controlled, it
is important that there is also some network function that makes sure the
source traffic stays within the requested bandwidth limit in-order to protect
non-Detnet traffic. This is to some extended discussed in section 3.3.2 but I
think it should be more clearly spelled out that this would require a rate
limiting function at each DetNet source/relay (tunnel ingress). Currently sec
3.3.2 says:

"Filters and policers should be used in a DetNet network to
   detect if DetNet packets are received on the wrong interface, or at
   the wrong time, or in too great a volume."

However, maybe this case of limiting non-congestion controlled traffic (in case
the source in not keeping to the limits on purpose/in order to cheat, because
it couldn't estimate the needed bandwidth requirement an better, or due to
timely fluctuations) could be explained more clearing and the respective
requirement to implement rate limiting could be state separately and more
strongly...?

One related comment on this sentence in Sec 3.1:

"As DetNet provides allocated resources (including provisioned capacity)
   to DetNet flows the use of transport layer congestion control
   [RFC2914] by App-flows is explicitly not required."

I guess congestion control should still be a requirement if the App-flow also
passes not-DetNet-aware segments of the path, e.g. maybe the first hop. Usually
use of congestion control for application limited flows is also not a problem
if sufficient bandwidth is available. Also note that, as I stated above, the
important part for not requiring congestion control is actually not only that
resources are allocated but also that rate limiting is in place to make sure
resources usage cannot be exceeded above the reserved allocation. Maybe this
sentence could also be further clarified in the draft...?

And then one more small comment that is also related. Sec 3.2.2.2 says:

"If packet replication and elimination is used over paths with
   resource allocation (Section 3.2.1), ..."

My assumption was that all DetNet traffic is send over pre-allocated
resources...? If that is not true that has implication on congestion control
and needs some additional considerations. Can you please confirm that and maybe
clarify in the draft! Thanks!

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

I agree with Alexey and Benjamin that this document should be informational.
Informational documents can also have IETF consensus, so that cannot be the
reason to go for PS. However, this document does not specify a protocol or any
requirements that are mandatory to implement for interoperability and therefore
should not be PS.

---

From: Lou Berger <[email-masked]>
Date: Wed, 20 Feb 2019 09:46:29 -0500

Hi Mirja,

I'm responding as doc shepherd (and chair). Thank you for your 
comments.  Please see below for specific responses.

> https://datatracker.ietf.org/doc/draft-ietf-detnet-architecture/

The input from Michael and David certainly led to a much improved 
document.  If you can help the authors with some specific text 
suggestions that would be most helpful.

> requirement to implement rate limiting could be state separately and more

Section 3.2.1.1 days:

    Ensuring adequate buffering requires, in turn, that the source, and
    every DetNet node along the path to the destination (or nearly every
    node, see Section 4.3.3) be careful to regulate its output to not
    exceed the data rate for any DetNet flow, except for brief periods
    when making up for interfering traffic.  Any packet sent ahead of its
    time potentially adds to the number of buffers required by the next
    hop DetNet node and may thus exceed the resources allocated for a
    particular DetNet flow.

Section 3.3.1 also says:

    Starvation of non-DetNet traffic must be avoided, e.g., by traffic
    policing functions (e.g., [RFC2475]).  Thus, the net effect of the
    presence of DetNet flows in a network on the non-DetNet flows is
    primarily a reduction in the available bandwidth.

Is this not sufficient?  If not, can you suggest some additional language?

> passes not-DetNet-aware segments of the path, e.g. maybe the first hop.

This point is addressed in section 3.3.2

    Note that the sending of App-flows that do not use transport layer
    congestion control per [RFC2914] into a network that is not
    provisioned to handle such DetNet traffic has to be treated as a
    fault and prevented.

> Usually

Is the above sufficient?  If not, can you suggest some additional language?

> clarify in the draft! Thanks!

The intent was to cover sending over non-pre-allocated paths as a fault 
case, and cover it in section 3.3.2.

In addition to the previously quoted text, the section also says:

    PRF generated DetNet member flows also need to
    be treated as not using transport layer congestion control even if
    the original App-flow supports transport layer congestion control
    because PREOF can remove congestion indications at the PEF and
    thereby hide such indications (e.g., drops, ECN markings, increased
    latency) from end systems.

Does this address your concern?

I defer to the IESG and AD on this...

Thank you!

Lou

> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 20 Feb 2019 17:06:24 +0100

Hi Lou,

I’m happy to see that my concerns seem to be cover. So maybe this is mostly a point about wording. Please see below.

> Am 20.02.2019 um 15:46 schrieb Lou Berger <[email-masked]>:
> 
> Hi Mirja,
> 
> I'm responding as doc shepherd (and chair). Thank you for your comments.  Please see below for specific responses.
> 
> The input from Michael and David certainly led to a much improved document.  If you can help the authors with some specific text suggestions that would be most helpful.
> 
> 

My question is mostly if you only need to protect _complete_ starvation or any over-use by DetNet flows and how you would do that. If the answer is to rate limit the DetNet flows at ingress to the allocated bandwidth, that would completely address my concern. However note that buffer dimensioning and rate limiting is not the same. Rate limiting is usually a scheduling question, e.g. using a token bucket scheme or a fixed non-work conserving scheduler. Therefore it would be nice to add one additional sentence in section 3.2 that spells that out directly. e.g. something like:

„In order to protect non-DetNet traffic from potentially misbehaving DetNet-Traffic sources, rate limiting, e.g. using a token bucket scheme, at each tunnel ingress must be applied.“ 

I’m sure you can adapt the wording to make it more DetNet-like language and maybe you also want to add another sentence explaining the problem, please feel free to do so but that’s more or less what I’m looking for.

> Is the above sufficient?  If not, can you suggest some additional language?

Maybe we can just extend the above statement a bit in order to be crystal clear about this, e.g.:

"As DetNet flow are assumed to be rate-limited and DetNet is designed to provide sufficient allocated resources (including provisioned capacity), the use of transport layer congestion control [RFC2914] for end-to-end DetNet flows is not required, however, if resources are allocated appropriately, use of congestion control should not impact transmission negatively.“

> The intent was to cover sending over non-pre-allocated paths as a fault case, and cover it in section 3.3.2.

If that is the case, maybe we can just removed the half-sentence I cited above?

Thanks!
Mirja

---

From: Lou Berger <[email-masked]>
Date: Wed, 20 Feb 2019 11:24:21 -0500

Hi Mirja,

     See below.

> I’m sure you can adapt the wording to make it more DetNet-like language and maybe you also want to add another sentence explaining the problem, please feel free to do so but that’s more or less wha…

I do agree that we seem to be having a terminology issue here - and that 
we are on the same page with what should happen.  The terminology I've 
used/seen in past RFCs has usually used the terms 'shaping' and 
'policing' - and these are what we've been using within the WG.  For 
example, both of these terms are used in RFC2475 but 'rate limiting' is 
not. Would it work for you for the draft to use/adapt the rest of your 
language, but keep 'by traffic policing functions (e.g., [RFC2475])' 
rather than 'rate limiting, e.g. using a token bucket scheme,' ?

> "As DetNet flow are assumed to be rate-limited and DetNet is designed to provide sufficient allocated resources (including provisioned capacity), the use of transport layer congestion control [RFC2…
Great!  (I'll leave it to the authors to do the final wording adaptation)
> If that is the case, maybe we can just removed the half-sentence I cited above?

WFM!

Thank you!

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 20 Feb 2019 19:36:32 +0100

Hi Lou,

see below.

> Am 20.02.2019 um 17:24 schrieb Lou Berger <[email-masked]>:
> 
> Hi Mirja,
> 
>     See below.
> 
> I do agree that we seem to be having a terminology issue here - and that we are on the same page with what should happen.  The terminology I've used/seen in past RFCs has usually used the terms 'shaping' and 'policing' - and these are what we've been using within the WG.  For example, both of these terms are used in RFC2475 but 'rate limiting' is not. Would it work for you for the draft to use/adapt the rest of your language, but keep 'by traffic policing functions (e.g., [RFC2475])' rather than 'rate limiting, e.g. using a token bucket scheme,' ?
> 
Indeed there is a wording issue here, but there is one important difference here. Policing as defined in [RFC2475] is the action of dropping to comply to a policy. While shaping is delaying to confirm to a traffic profile. A token bucket would do both… however, the important bit is not what you do but what your goal/policy/profile is, and the goal is to keep the traffic (on average) below a certain rate. This bit should be spelled out more explicitly.

Mirja

---

From: Lou Berger <[email-masked]>
Date: Wed, 20 Feb 2019 14:49:35 -0500

Hi Mirja,

> Indeed there is a wording issue here, but there is one important difference here. Policing as defined in [RFC2475] is the action of dropping to comply to a policy. While shaping is delaying to conf…

I completely agree that both shaping and policing play a role in 
supporting DetNet services (independent of the specific mechanism 
selected for implementation) and think adding 'shaping' here - and 
perhaps other places that policing is mentioned is a worthwhile change.

Thanks!

Lou

---

From: János Farkas <[email-masked]>
Date: Tue, 26 Feb 2019 13:45:21 +0100

Hi Mirja,

Thank you for your review!

Please see in-line changes suggested to address your comments.

On 2/20/2019 5:24 PM, Lou Berger wrote:
>> I’m happy to see that my concerns seem to be cover. So maybe this is 
>> mostly a point about wording. Please see below.
>>
>>>> Please refer to 
>>>> https://www.ietf.org/iesg/statement/discuss-criteria.html
>> My question is mostly if you only need to protect _complete_ 
>> starvation or any over-use by DetNet flows and how you would do that. 
>> If the answer is to rate limit the DetNet flows at ingress to the 
>> allocated bandwidth, that would completely address my concern. 
>> However note that buffer dimensioning and rate limiting is not the 
>> same. Rate limiting is usually a scheduling question, e.g. using a 
>> token bucket scheme or a fixed non-work conserving scheduler. 
>> Therefore it would be nice to add one additional sentence in section 
>> 3.2 that spells that out directly. e.g. something like:
>>
>> „In order to protect non-DetNet traffic from potentially misbehaving 
>> DetNet-Traffic sources, rate limiting, e.g. using a token bucket 
>> scheme, at each tunnel ingress must be applied.“
>
>> I’m sure you can adapt the wording to make it more DetNet-like 
>> language and maybe you also want to add another sentence explaining 
>> the problem, please feel free to do so but that’s more or less what 
>> I’m looking for.
>
> I do agree that we seem to be having a terminology issue here - and 
> that we are on the same page with what should happen.  The terminology 
> I've used/seen in past RFCs has usually used the terms 'shaping' and 
> example, both of these terms are used in RFC2475 but 'rate limiting' 
> is not. Would it work for you for the draft to use/adapt the rest of 
> your language, but keep 'by traffic policing functions (e.g., 
> [RFC2475])' rather than 'rate limiting, e.g. using a token bucket 
> scheme,' ?

We could extend the 2nd paragraph in 3.2.1.1 
(https://tools.ietf.org/html/draft-ietf-detnet-architecture-11#section-3.2.1.1) 
to capture this comment:

OLD:
             Ensuring adequate buffering requires, in turn, that the 
source, and every DetNet
             node along the path to the destination (or nearly every 
node, see
             <xref target="Incomplete"/>) be careful to regulate its 
output to not exceed the
             data rate for any DetNet flow, except for brief periods 
when making up for
             interfering traffic.  Any packet sent ahead of its time 
potentially adds to the
             number of buffers required by the next hop DetNet node and 
may thus exceed the resources
             allocated for a particular DetNet flow.

NEW:

             Ensuring adequate buffering requires, in turn, that the 
source, and every DetNet
             node along the path to the destination (or nearly every 
node, see
             <xref target="Incomplete"/>) be careful to regulate its 
output to not exceed the
             data rate for any DetNet flow, except for brief periods 
when making up for
             interfering traffic.  Any packet sent ahead of its time 
potentially adds to the
             number of buffers required by the next hop DetNet node and 
may thus exceed the resources
             allocated for a particular DetNet flow. Furthermore, rate 
limiting, e.g., using traffic
             policing and shaping functions, e.g., <xref 
target="RFC2475"/>, at the ingress of the
             DetNet domain must be applied. This is needed for meeting 
the requirements of DetNet
             flows as well as for protecting non-DetNet traffic from 
potentially misbehaving DetNet
             traffic sources.

Furthermore, replace the two occurrences of "policing" with "policing 
and shaping":

OLD:
    Starvation of non-DetNet traffic must be avoided, e.g., by traffic
    policing functions (e.g., [RFC2475]).

NEW:
    Starvation of non-DetNet traffic must be avoided, e.g., by traffic
    policing and shaping functions (e.g., [RFC2475]).

OLD:
    Examples of such techniques include traffic policing
    functions (e.g., [RFC2475]) and separating flows into per-flow rate-
    limited queues.

NEW:
    Examples of such techniques include traffic policing and shaping
    functions (e.g., [RFC2475]) and separating flows into per-flow rate-
    limited queues.

>> "As DetNet flow are assumed to be rate-limited and DetNet is designed 
>> to provide sufficient allocated resources (including provisioned 
>> capacity), the use of transport layer congestion control [RFC2914] 
>> for end-to-end DetNet flows is not required, however, if resources 
>> are allocated appropriately, use of congestion control should not 
>> impact transmission negatively.“

"App-flow" seems to be better than "end-to-end DetNet flow". The change 
with that would be:

OLD:
    As
    DetNet provides allocated resources (including provisioned capacity)
    to DetNet flows the use of transport layer congestion control
    [RFC2914] by App-flows is explicitly not required.

NEW:
             As
             DetNet flows are assumed to be rate-limited and DetNet is 
designed
             to provide sufficient allocated resources (including 
provisioned
             capacity), the use of transport layer congestion control 
[RFC2914]
             for App-flows is not required; however, if resources are 
allocated
             appropriately, use of congestion control should not impact
             transmission negatively.

>>> The intent was to cover sending over non-pre-allocated paths as a 
>>> fault case, and cover it in section 3.3.2.

The sentence after removing the text as you suggest:

OLD:
    If packet replication and elimination is used over paths with
    resource allocation (Section 3.2.1), and member flows that take
    different-length paths through the network are combined, a merge
    point may require extra buffering to equalize the delays over the
    different paths.

NEW:
    If member flows that take different-length paths through the
    network are combined, a merge point may require extra buffering
    to equalize the delays over the different paths.

Thank you!
Janos