Subject: [Detnet] Magnus Westerlund's Discuss on draft-ietf-detnet-mpls-11: (with DISCUSS)

From: Magnus Westerlund via Datatracker <[email-masked]>
Date: Tue, 08 Sep 2020 08:09:21 -0700

Magnus Westerlund has entered the following ballot position for
draft-ietf-detnet-mpls-11: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-detnet-mpls/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I like to thank the TSV-ART reviewer for helping me consider one aspect of the
issue I see needing some discussion for this document.

This relates to Section 4.2.2.2. and 4.2.2.3.

So both of these section discuss the use of the sequence number for removing
packet duplicates and handling reorder. As the text discusses there can be a
configured limit for how deep the buffer and state are for performing these
operations. We all know that the implementation of this will have a practical
limit in both buffer space for reordering as well as state for tracking which
sequence numbers that have been forwarded. I think that should be more clearly
expressed in the document that these practical limits exists. Thus, the
implementations will have tracking and determination of what are new packets
(increasing sequence number within a window higher than previous largest seen.
And consider sequence number form currently highest seen and a bit backwards as
older packets. Thus how this is implemented will impact how this acts in cases
of disruptions of the packet flow. Thus, I wonder if there is actually need to
be  a bit more specific in how classification should be done. Especially if the
wrap-around of the sequence number space approaches a small multiple of round
trip times for the path which is likely for the 16-bit space.

Then  sections fails to discuss how the duplication removal, the reordering
buffering and bound latency interacts and affet each other.  So if the latency
is bounded then the reordering has an hard time limit for the maximum delay. If
there is a boundary for reordering then there are no point in de-duplicating
packets that will not be forwarded due to the reordering. And even if there are
no bounded latency the reordering buffer size will still impact the depth of
de-duplication. These practical limits will also be limitations on the
guarantees that can be provided.

Thus, from my perspective there is need for more text on the requirements of
the implementation of these functions and their interactions of creating
limitations.

Another point on 4.2.2.2:

When configured, the
   implementation MUST track the sequence number contained in received
   d-CWs and MUST ensure that duplicate (replicated) instances of a
   particular sequence number are discarded.

That second MUST I think is possible to meet given that one discard all packets
outside of the current window where one have information if a packet sequence
number have been forwarded or not. Given that a very late packet beyond the
amount of state for the flow likely anyway have little utility that is likely
the right choice. However, I think it needs to be made explicit that this is
okay.

In Section 4.2.2.3:

 When configured, the
   implementation MUST track the sequence number contained in received
   d-CWs and MUST ensure that packets are processed in the order
   indicated in the received d-CW sequence number field, which may not
   be in the order the packets are received.

I think this part needs to be explicit that packets that are to fare out of
order for the implementation to handle will/shall be dropped.

   Note that an implementation MAY wish to constrain the maximum number
   of out of order packets that can be processed, on platform-wide or
   per flow basis.  Some implementations MAY support the provisioning of
   this number on either a platform-wide or per flow basis.  The number
   of out of order packets that can be processed also impacts the
   latency of a flow.

If there exists a latency requirement then that will interact with this when it
comes to reordering. In fact a significant issue here is that if the packet
flow is not periodic at a steady pace the maximum latency in the reordering
buffers based on packet sequence numbers can not be ensured. Instead some form
of time limit needs to exist also. If that time limit is only local then there
exists a risk that over multiple reordering buffers if multiple independent
service labels are used the jitter and latency becomes cumulative. If the goal
is to avoid this then the individual packets would need to carry a time stamp
to ensure that from ingress of the service label path until the egress a
maximum latency is added.

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 8 Sep 2020 17:12:57 +0100

> On 8 Sep 2020, at 16:09, Magnus Westerlund via Datatracker <[email-masked]> wrote:
> 
> In Section 4.2.2.3:
> 
> When configured, the

It seems to me that this is getting into the realms of in-time and on-time delivery which is outside the scope  of this work. Indeed it is quite a significant increase in the technology to address that which needs to be thought through very carefully rather than added on the fly.

At this stage we should accept that the design mitigates the loss characteristics of the network and leave the more advanced work such as in network latency management to future work.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 8 Sep 2020 17:21:39 +0100

> the implementation of these functions and their interactions of creating

The text seems to line up well with the work that we did on pseudowire sequence numbers and we have had no reports from implementors or operators that we need to increase the text and provide implementation guidance.

I am always worried about overprescribing the operation of such systems since it is easy to rule out legitimate implementation optimisations, or inadvertently prevent the use of the technology in some way that we have not yet considered but which would be valid.

I would argue that the buffer sizing, constraints and management is more properly a discussion between the vendor and the operator.

On latency we are not in this design providing precision latency management, just best effort mitigation of the natural latency of the network. This may change if we introduce queue management or other services, but I do not think we are there yet, and suspect that once we get to latency management we will need more metadata in the packet.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 8 Sep 2020 17:35:36 +0100

> the right choice. However, I think it needs to be made explicit that this is

Yes we need to make sure that when we are tracking sequence numbers, packets outside the window are discarded.

I am not sure how common the re-order option will be. In latency centric applications I suspect that re-order will not happen. I am also concerned about how a high performance implementation would do a re-order, so I suspect that this will only be applied to relatively slow flows that are processed by a classical pure software implementation. In turn I am not sure if we will ever see that in an MPLS network. So whilst the design has capability, I am not sure how profitable it is to drill down into the detail with some operational experience. I don’t think we need to do a lot of hand holding, since those that understand their actual requirement will understand how to do this, or soon work it out, and I so not see any interoperability issues per se with the text as it is.

- Stewart

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 8 Sep 2020 21:12:38 +0200

Thanks Magnus, *:

Related to your comments, i would like to raise a concern about the initial
sentence in the spec:

...DetNet provides zero congestion loss and bounded latency and jitter.

To me, this is overselling what DetNet actually "provides" or that
uniquely distinguishes DetNet from other solutions. It sounds as if DetNet
provides a novel solution whereas in reality it just allows to adopt
existing or new solutions.

With the definitions DetNet has done today, any IP or MPLS network where
end-to-end flows can be identified as e.g.: an IP 5-tuple or an LSP
identifier and that manages to figure out how to implement or operationalize
one of the solutions for bounded latency such as a PHB in support of rfc2212.

Aka: one could equally write:

...DetNet like any other IP/MPLS network with per-flow forwarding provides
zero congestion loss and bounded latency and jitter.

Which would be equally true and equally misleading.

So, here is proposed IMHO more technically correct text to replace the
IMHO misleading "marketing" sentence segment:

...DetNet MPLS sets up point-to-point LSPs end-to-end across DetNet domains.

Because of this, DetNet MPLS can integrate with pre-existing and/or future Per-Hop-Behavior
(PHB) (such one derived from RFC2212) that can provide per-flow (e.g.: LSP) bounded latency,
bounded jitter and no congestion loss, as long as such a PHB does not require additional
network packet header information beside the flow/LSP identification.

Cheers
    Toerless

On Tue, Sep 08, 2020 at 08:09:21AM -0700, Magnus Westerlund via Datatracker wrote:
> _______________________________________________
> detnet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/detnet

-- 
---
[email-masked]

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 9 Sep 2020 07:27:35 +0000

On Tue, 2020-09-08 at 17:12 +0100, Stewart Bryant wrote:
> 
> 
> > On 8 Sep 2020, at 16:09, Magnus Westerlund via Datatracker <[email-masked]
> > > wrote:
> > 
> > In Section 4.2.2.3:
> > 
> > When configured, the
> It seems to me that this is getting into the realms of in-time and on-time
> delivery which is outside the scope  of this work. Indeed it is quite a
> significant increase in the technology to address that which needs to be
> thought through very carefully rather than added on the fly.

I think a potential solution here without specifying things to much is to at
least make clear that there are no support for the timeliness aspect in this
version. As currently specified if you are using reordering buffer you have no
way of providing a bounded delay unless you have knowledge of the minimal packet
transmission rate and that following some particular schedule so that you can
calculate what worst case latency may be incurred. You likely have statistical
better boundaries, but to my understanding there are no mechanism in this
document that can ensure bounded delay on packets that are not lost. 

> 
> At this stage we should accept that the design mitigates the loss
> characteristics of the network and leave the more advanced work such as in
> network latency management to future work.

I have no problem with this, I just think the document needs to come straight
with its actual capabilities. 

Cheers

Magnus Westerlund 

----------------------------------------------------------------------
Networks, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 9 Sep 2020 07:42:02 +0000

On Tue, 2020-09-08 at 17:21 +0100, Stewart Bryant wrote:
> The text seems to line up well with the work that we did on pseudowire
> sequence numbers and we have had no reports from implementors or operators
> that we need to increase the text and provide implementation guidance.
> 
> I am always worried about overprescribing the operation of such systems since
> it is easy to rule out legitimate implementation optimisations, or
> inadvertently prevent the use of the technology in some way that we have not
> yet considered but which would be valid.

So, I think any issues would only really show up when your close to the
boundaries of how many packets are in flight. And with current technolghy and
link rates the 28-bit sequence number supports rates for DETNET flows into the
terrabit range if I calcualted correctly unless the path is very long (>>100
ms). The 16-bit field has much more restriction and could be pushed into
wrapping on the time scales packets has some chance of later showing up. Likely
the application are not pushing that amount of bits, but I want to bet on that.
This might be used in particla physic or simiarl field which produces stupendous
amount of data that needs to be processed in order. 

> 
> I would argue that the buffer sizing, constraints and management is more
> properly a discussion between the vendor and the operator.

The issue I see here from an interoperability stand point is if the
configuration option for example buffer depth needs to be set differently due to
how the buffering and processing is done for different vendor. That is not great
for interoperbility or ease of management for the operator. 

> 
> On latency we are not in this design providing precision latency management,
> just best effort mitigation of the natural latency of the network. This may
> change if we introduce queue management or other services, but I do not think
> we are there yet, and suspect that once we get to latency management we will
> need more metadata in the packet.

I think this comes back to the previous email. The document is not properly
expressing the limitation in what actual properties it can ensure. And when it
comes to latency the theoretical boundaries appears to be much worse than what
the practical ones likely are for the wast majority. 
 
Cheers

Magnus Westerlund

---

From: Balázs Varga A <[email-masked]>
Date: Wed, 9 Sep 2020 13:50:34 +0000

Hi Toerless,

Many thanks for the comments. One remark:
- I disagree with your statement "DetNet like any other IP/MPLS network with per-flow forwarding provides"
Just as an example, PREOF functions are not available in current MPLS networks. 

Thanks
Bala'zs

-----Original Message-----
From: Toerless Eckert <[email-masked]> 
Sent: Tuesday, September 8, 2020 9:13 PM
To: Magnus Westerlund <[email-masked]>
Cc: The IESG <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Detnet] Magnus Westerlund's Discuss on draft-ietf-detnet-mpls-11: (with DISCUSS)

Thanks Magnus, *:

Related to your comments, i would like to raise a concern about the initial sentence in the spec:

To me, this is overselling what DetNet actually "provides" or that uniquely distinguishes DetNet from other solutions. It sounds as if DetNet provides a novel solution whereas in reality it just allows to adopt existing or new solutions.

With the definitions DetNet has done today, any IP or MPLS network where end-to-end flows can be identified as e.g.: an IP 5-tuple or an LSP identifier and that manages to figure out how to implement or operationalize one of the solutions for bounded latency such as a PHB in support of rfc2212.

...DetNet like any other IP/MPLS network with per-flow forwarding provides zero congestion loss and bounded latency and jitter.

So, here is proposed IMHO more technically correct text to replace the IMHO misleading "marketing" sentence segment:

Because of this, DetNet MPLS can integrate with pre-existing and/or future Per-Hop-Behavior
(PHB) (such one derived from RFC2212) that can provide per-flow (e.g.: LSP) bounded latency, bounded jitter and no congestion loss, as long as such a PHB does not require additional network packet header information beside the flow/LSP identification.

Cheers
    Toerless

On Tue, Sep 08, 2020 at 08:09:21AM -0700, Magnus Westerlund via Datatracker wrote:
> email addresses included in the To and CC lines. (Feel free to cut 
> this introductory paragraph, however.)
> 
> 
> Please refer to 
> https://www.ietf.org/iesg/statement/discuss-criteria.html
> I like to thank the TSV-ART reviewer for helping me consider one 
> aspect of the issue I see needing some discussion for this document.
> 
> So both of these section discuss the use of the sequence number for 
> removing packet duplicates and handling reorder. As the text discusses 
> there can be a configured limit for how deep the buffer and state are 
> for performing these operations. We all know that the implementation 
> of this will have a practical limit in both buffer space for 
> reordering as well as state for tracking which sequence numbers that 
> have been forwarded. I think that should be more clearly expressed in 
> the document that these practical limits exists. Thus, the 
> implementations will have tracking and determination of what are new packets (increasing sequence number within a window higher than previous largest seen.
> And consider sequence number form currently highest seen and a bit 
> backwards as older packets. Thus how this is implemented will impact 
> how this acts in cases of disruptions of the packet flow. Thus, I 
> wonder if there is actually need to be  a bit more specific in how 
> classification should be done. Especially if the wrap-around of the 
> sequence number space approaches a small multiple of round trip times for the path which is likely for the 16-bit space.
> 
> Then  sections fails to discuss how the duplication removal, the 
> reordering buffering and bound latency interacts and affet each other.  
> So if the latency is bounded then the reordering has an hard time 
> limit for the maximum delay. If there is a boundary for reordering 
> then there are no point in de-duplicating packets that will not be 
> forwarded due to the reordering. And even if there are no bounded 
> latency the reordering buffer size will still impact the depth of 
> de-duplication. These practical limits will also be limitations on the guarantees that can be provided.
> 
> Thus, from my perspective there is need for more text on the 
> requirements of the implementation of these functions and their 
> interactions of creating limitations.
> 
> That second MUST I think is possible to meet given that one discard 
> all packets outside of the current window where one have information 
> if a packet sequence number have been forwarded or not. Given that a 
> very late packet beyond the amount of state for the flow likely anyway 
> have little utility that is likely the right choice. However, I think 
> it needs to be made explicit that this is okay.
> 
> In Section 4.2.2.3:
> 
>  When configured, the
> I think this part needs to be explicit that packets that are to fare 
> out of order for the implementation to handle will/shall be dropped.
> 
> If there exists a latency requirement then that will interact with 
> this when it comes to reordering. In fact a significant issue here is 
> that if the packet flow is not periodic at a steady pace the maximum 
> latency in the reordering buffers based on packet sequence numbers can 
> not be ensured. Instead some form of time limit needs to exist also. 
> If that time limit is only local then there exists a risk that over 
> multiple reordering buffers if multiple independent service labels are 
> used the jitter and latency becomes cumulative. If the goal is to 
> avoid this then the individual packets would need to carry a time 
> stamp to ensure that from ingress of the service label path until the egress a maximum latency is added.
> 
> 
> 
> 
> 

--
---
[email-masked]

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 9 Sep 2020 14:26:00 +0000

Hi,

Have thought about the discuss a bit more and think I have come to kernel of the
issue related to bounded latency in this context.

So the POF must be able to buffer for a certain period to be able to reorder.
The Detnet very prominently says it can provide bounded latency and I think the
only shortcoming is the consideration of the POF and how take that into account.

So https://datatracker.ietf.org/doc/draft-ietf-detnet-data-plane-framework/:

3.5.2.2.  Path Differential Delay

   In the preceding example, proper working of duplicate elimination and
   reordering of packets are dependent on the number of out-of-order
   packets that can be buffered and the delay difference of arriving
   packets.  DetNet uses flow-specific requirements (e.g., maximum
   number of out-of-order packets, maximum latency of the flow) for
   configuration of POF-related buffers.  If the differential delay
   between paths is excessively large or there is excessive mis-ordering
   of the packets, then packets may be dropped instead of being
   reordered.  Likewise, PEF uses the sequence number to identify
   duplicate packets, and large differential delays combined with high
   numbers of packets may exceed the ability of the PEF to work
   properly.

I think this makes it clear that there are a requirement on the POF also for
MPLS to handle these aspects. So the main question from my perspective for the
WG to consider I think is if there is a solution where the POF can be configured
with a maximum latency incurred by the POF or if there is need for additional
meta data (timestamp) in the MPLS encapsulation to be able to provide this
function. 

Assuming that links and forwarding and other packet buffers can be bounded then
the reordering should be possible to be simply configured with a maximum latency
in reordering so that packet that arrives are released within that time frame to
achieve bound latency. However, I am not certain that I have considered all
aspects of inter packet arrival and its impact also on the actual reordering
window. 

Thus, resolving this aspect of the discuss may be as simple as mandating a
configuration of the maximum latency incurred by the POF so that the latency of
the MPLS network in dealing with a particular S-Label can be bounded. 

Cheers

Magnus Westerlund

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 9 Sep 2020 17:20:49 +0200

> Just as an example, PREOF functions are not available in current MPLS networks. 

PREOF is not subject of the sentence part in question. My concern is only about:

... DetNet provides zero congestion loss and bounded latency and jitter

Of course, now you mention it: The MPLS forwarding plane of this spec does
support PEROF, but the sentence only talks about "DetNet", for which at
large in my assesment this is not true (no current PREOF for IPv4/IPv6 AFAIK).

Aka: also for the part of PREOF its better to re-scope the sentence to talk only
the MPLS forwarding plane of this document instead of (unnecessarily?) make
claims about DetNet at large.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Lou Berger <[email-masked]>
Date: Wed, 9 Sep 2020 11:39:20 -0400

The  doc currently reads (asterisks indicate the sentence under discussion):

    1.  Introduction

    Deterministic Networking (DetNet) is a service that can be offered by
    a network to DetNet flows.  *DetNet provides these flows with
    extremely low packet loss rates and assured maximum end-to-end
    delivery latency.*  General background and concepts of DetNet can be
    found in [RFC8655].

The sentence in question was copied from the draft version of RFC8655, 
which now reads slightly differently:

    ... which provides a capability for the delivery of
    data flows with extremely low packet loss rates and bounded end-to-
    end delivery latency.

I suggest either (a) updating the draft to match the RFC text or (b) 
dropping it altogether and let the reference to RFC8655 stand alone.

Lou

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 9 Sep 2020 16:59:02 +0100

I think it has to be dropped, because as the work stands I cannot see that we have a way of bounding the latency.

I know that we are talking about MPLS, but of course we need to look at all of the data plane drafts in this respect.

- Stewart

> On 9 Sep 2020, at 16:39, Lou Berger <[email-masked]> wrote:
> 
> The sentence in question was copied from the draft version of RFC8655, which now reads slightly differently:
> 
> I suggest either (a) updating the draft to match the RFC text or (b) dropping it altogether and let the reference to RFC8655 stand alone.
> 
> Lou
>

---

From: Loa Andersson <[email-masked]>
Date: Thu, 10 Sep 2020 00:09:40 +0800

All,

While I won't cry if this is dropped, I'm a little bit more than a
bit concerned that every time we try to state what we can expect of
a DetNet service we seem to stumble on the finish line.

If it works for MPLS and is updated to match RFC 8655, I'd say we leave
it in.

/Loa

-- 

Loa Andersson                        email: [email-masked]
Senior MPLS Expert                          [email-masked]
Bronze Dragon Consulting             phone: +46 739 81 21 64

---

From: Jeff Tantsura <[email-masked]>
Date: Wed, 9 Sep 2020 19:10:45 +0200

MPLS data plane formally operates on a FEC not a flow (while a flow can be a FEC).

Regards,
Jeff

> On Sep 9, 2020, at 17:21, Toerless Eckert <[email-masked]> wrote:
>

---

From: Balázs Varga A <[email-masked]>
Date: Wed, 9 Sep 2020 17:24:27 +0000

Hi All,
Many thanks for the suggestions.
I will update the draft to match rfc8655.
Cheers
Bala'zs

-----Original Message-----
From: Loa Andersson <[email-masked]> 
Sent: Wednesday, September 9, 2020 6:10 PM
To: Stewart Bryant <[email-masked]>; Lou Berger <[email-masked]>
Cc: Magnus Westerlund <[email-masked]>; [email-masked]; Toerless Eckert <[email-masked]>; Balázs Varga A <[email-masked]>; The IESG <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Detnet] Magnus Westerlund's Discuss on draft-ietf-detnet-mpls-11: (with DISCUSS)

All,

While I won't cry if this is dropped, I'm a little bit more than a bit concerned that every time we try to state what we can expect of a DetNet service we seem to stumble on the finish line.

If it works for MPLS and is updated to match RFC 8655, I'd say we leave it in.

/Loa

On 09/09/2020 23:59, Stewart Bryant wrote:
>>> Of course, now you mention it: The MPLS forwarding plane of this 
>>> spec does support PEROF, but the sentence only talks about "DetNet", 
>>> for which at large in my assesment this is not true (no current PREOF for IPv4/IPv6 AFAIK).
>>>
>>> Aka: also for the part of PREOF its better to re-scope the sentence 
>>> to talk only the MPLS forwarding plane of this document instead of 
>>> (unnecessarily?) make claims about DetNet at large.
>>>
>>> Cheers
>>>      Toerless
>>>
>>>> Thanks
>>>> Bala'zs
>>>>
>>>> Cc: The IESG <[email-masked]>; [email-masked]; [email-masked]; 
>>>> [email-masked]; [email-masked]
>>>> Subject: Re: [Detnet] Magnus Westerlund's Discuss on 
>>>> draft-ietf-detnet-mpls-11: (with DISCUSS)
>>>>
>>>> Thanks Magnus, *:
>>>>
>>>>> When responding, please keep the subject line intact and reply to 
>>>>> all email addresses included in the To and CC lines. (Feel free to 
>>>>> cut this introductory paragraph, however.)
>>>>>
>>>>>
>>>>> Please refer to
>>>>> So both of these section discuss the use of the sequence number 
>>>>> for removing packet duplicates and handling reorder. As the text 
>>>>> discusses there can be a configured limit for how deep the buffer 
>>>>> and state are for performing these operations. We all know that 
>>>>> the implementation of this will have a practical limit in both 
>>>>> buffer space for reordering as well as state for tracking which 
>>>>> sequence numbers that have been forwarded. I think that should be 
>>>>> more clearly expressed in the document that these practical limits 
>>>>> exists. Thus, the implementations will have tracking and determination of what are new packets (increasing sequence number within a window higher than previous largest seen.
>>>>> backwards as older packets. Thus how this is implemented will 
>>>>> impact how this acts in cases of disruptions of the packet flow. 
>>>>> Thus, I wonder if there is actually need to be  a bit more 
>>>>> specific in how classification should be done. Especially if the 
>>>>> wrap-around of the sequence number space approaches a small multiple of round trip times for the path which is likely for the 16-bit space.
>>>>>
>>>>> That second MUST I think is possible to meet given that one 
>>>>> discard all packets outside of the current window where one have 
>>>>> information if a packet sequence number have been forwarded or 
>>>>> not. Given that a very late packet beyond the amount of state for 
>>>>> the flow likely anyway have little utility that is likely the 
>>>>> right choice. However, I think it needs to be made explicit that this is okay.
>>>>>
>>>>> In Section 4.2.2.3:
>>>>>
>>>>>   When configured, the
>>>>> I think this part needs to be explicit that packets that are to 
>>>>> fare out of order for the implementation to handle will/shall be dropped.
>>>>>
>>>>> this when it comes to reordering. In fact a significant issue here 
>>>>> is that if the packet flow is not periodic at a steady pace the 
>>>>> maximum latency in the reordering buffers based on packet sequence 
>>>>> numbers can not be ensured. Instead some form of time limit needs to exist also.
>>>>> If that time limit is only local then there exists a risk that 
>>>>> over multiple reordering buffers if multiple independent service 
>>>>> labels are used the jitter and latency becomes cumulative. If the 
>>>>> goal is to avoid this then the individual packets would need to 
>>>>> carry a time stamp to ensure that from ingress of the service label path until the egress a maximum latency is added.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>

--

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 9 Sep 2020 20:05:03 +0200

RFC8655 makes claims that can still not be substantiated with actual specification
work from DetNet: There is no PREOF for IPv4/IPv4. AFAIK, there is no IETF PHB
specification for any of the possible bounded latency options that _could_ be inherited
from e.g.: TSN.  The only bounded latency RFC2212 is not even referenced in either
RFC8655 nor this subject draft. 

If this draft intends to repeat unsubstantiated marketing claims then it should be fair
game to criticize them. Especially because its completely irrelevant what "DetNet"
at large may or may not do. The relevant facts are those that apply to the content
of the document.

"DetNet provides" should be reserved to actual work from DetNet. I do not see any 
spec work from DetNet for bounded latency / no congestion loss. See above. If 
"DetNet can be used in conjunction with <third-party/TSN-IntServ-work>", thats fine,
but thats not what is written. I think that is false advertisement.

A lot of people who d not understand the technology are going to read the RFCs
and think DetNet is magically solving problems that it is not. Because of these
sentences. 

I proposed better text that hopefully technically correct describes what
the MPLS data plane achieves. I have seen no comment about that proposed text.
I think technical accuracy and truth in advertisement in not too much to ask.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 9 Sep 2020 20:12:49 +0200

>    delivery latency.*  General background and concepts of DetNet can be

Suggest:

     Deterministic Networking (DetNet) is a service that can be offered by
     a network to DetNet flows.  The DetNet MPLS Data Plane for DetNet uses point-to-point
     LSP to transport DetNet Flows. The DetNet MPLS Data Plane DetNet flows with
     extremely low packet loss rates because of PREOF (see below). The DetNet
     MPLS Data Plane can be combined with pre-existing or future Per-Hop-Behavior (PHB)
     to provide bounded latency and zero congestion loss. 

The bounded jitter is somewhat questionable to claim for the DetNet MPLS
Data Plane from this document: I don't know if for example IntServ/GS or
TSN/ATS would claim to provide bounded jitter: Their jitter is between physcial
propagation latency and maximum guaranteed latency... Thats kinda "free" jitter
without any additional work. Mechanisms that provide further constraints on jitter
such as some of the TSN options or the proposed cyclic queuing (Large Scale
Deterministic Networking) would require additional packet header fields to work,
which are not provided by this DetNet MPLS Data Plane.
.. maybe i am overlooking something here.

> dropping it altogether and let the reference to RFC8655 stand alone.

The architecture is aspirational and to a good extend vague and makes claims
not substantiated by spec today. The bar to repeat such claims in an actual
protocol spec like subject one should be somewhat higher.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Uma Chunduri <[email-masked]>
Date: Wed, 9 Sep 2020 12:15:48 -0700

Mostly agree with the suggestion below. One comment in-lie.

--
Uma C.

>      to provide bounded latency and zero congestion loss.

[Uma]: The last sentence should be dropped. As that is not possible for
MPLS dataplane (not many TC bits left) without further enhancements in the
packet fields (CW or otherwise).

> > > > - I disagree with your statement "DetNet like any other IP/MPLS
> network with per-flow forwarding provides"
> > > > With the definitions DetNet has done today, any IP or MPLS network
> where end-to-end flows can be identified as e.g.: an IP 5-tuple or an LSP
> identifier and that manages to figure out how to implement or
> operationalize one of the solutions for bounded latency such as a PHB in
> support of rfc2212.
> > > >
> > > > ...DetNet like any other IP/MPLS network with per-flow forwarding
> provides zero congestion loss and bounded latency and jitter.
> > > >
> > > > So, here is proposed IMHO more technically correct text to replace
> the IMHO misleading "marketing" sentence segment:
> > > >
> > > > (PHB) (such one derived from RFC2212) that can provide per-flow
> (e.g.: LSP) bounded latency, bounded jitter and no congestion loss, as long
> as such a PHB does not require additional network packet header information
> beside the flow/LSP identification.
> > > >
> > > > Cheers
> > > >      Toerless
> > > >
> all
> > > > > implementations will have tracking and determination of what are
> new packets (increasing sequence number within a window higher than
> previous largest seen.
> impact
> > > > > sequence number space approaches a small multiple of round trip
> times for the path which is likely for the 16-bit space.
> > > > >
> fare
> is

---

From: Janos Farkas <[email-masked]>
Date: Wed, 9 Sep 2020 20:34:28 +0000

Hi Magnus,

I think the parameter you are looking for is there in section  4.2.2.3. Packet Ordering Function Processing (https://tools.ietf.org/id/draft-ietf-detnet-mpls-11.html#pof-requirements), just encoded a different way; in particular in the number of packets:
“Note that an implementation MAY wish to constrain the maximum number of out of order packets that can be processed, on platform-wide or per flow basis. Some implementations MAY support the provisioning of this number on either a platform-wide or per flow basis. The number of out of order packets that can be processed also impacts the latency of a flow.”

Ultimately, the latency requirements of flows have to be met. The maximum packet size of a DetNet flow is known, see item c) MaxPayloadSize in https://tools.ietf.org/html/draft-ietf-detnet-flow-information-model-10#section-5.5 and max-payload-size in https://tools.ietf.org/html/draft-ietf-detnet-yang-07. 
So, maximum POF buffer size to be applied for a flow can be determined; can be converted to time and taken into account during latency engineering of a DetNet network; thus bounded latency can be provided.

I fully agree with you that this needs to be taken into account. Together with what the set of documents that specify DetNet provide, I think it is fine what we have in the MPLS data plane draft. When it comes to configuration - as you pointed out - I think the job we need to do is to pay attention to do it right in the YANG draft, which is still under development.

Regards,
Janos

-----Original Message-----
From: Magnus Westerlund <[email-masked]> 
Sent: Wednesday, September 9, 2020 4:26 PM
To: [email-masked]; magnus.westerlund=[email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Magnus Westerlund's Discuss on draft-ietf-detnet-mpls-11: (with DISCUSS)

Hi,

Have thought about the discuss a bit more and think I have come to kernel of the issue related to bounded latency in this context.

So the POF must be able to buffer for a certain period to be able to reorder.
The Detnet very prominently says it can provide bounded latency and I think the only shortcoming is the consideration of the POF and how take that into account.

I think this makes it clear that there are a requirement on the POF also for MPLS to handle these aspects. So the main question from my perspective for the WG to consider I think is if there is a solution where the POF can be configured with a maximum latency incurred by the POF or if there is need for additional meta data (timestamp) in the MPLS encapsulation to be able to provide this function. 

Assuming that links and forwarding and other packet buffers can be bounded then the reordering should be possible to be simply configured with a maximum latency in reordering so that packet that arrives are released within that time frame to achieve bound latency. However, I am not certain that I have considered all aspects of inter packet arrival and its impact also on the actual reordering window. 

Thus, resolving this aspect of the discuss may be as simple as mandating a configuration of the maximum latency incurred by the POF so that the latency of the MPLS network in dealing with a particular S-Label can be bounded. 

Cheers

Magnus Westerlund

---

From: Loa Andersson <[email-masked]>
Date: Thu, 10 Sep 2020 11:14:10 +0800

Toerless,

>       to provide bounded latency and zero congestion loss.

Maybe, but I find it a bit hard to parse.

"The DetNet MPLS Data Plane for DetNet uses point-to-point LSP to
transport DetNet Flows."

Isn't there at least one "DetNet to many in that sentence?

"The DetNet MPLS Data Plane DetNet flows..."

Agreed that the MPLS Data Plane for DetNet supports DetNet flows, but
does the data palne itself flow?

"...combined with pre-existing or future Per-Hop-Behavior (PHB)..."

First, is inconceivable that we define a PHB that does not contribute
to bounded latency?

Second, is PHB's the only thing that contribute to bounded latency?

/Loa

--

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 10 Sep 2020 08:19:20 +0000

On Wed, 2020-09-09 at 20:34 +0000, Janos Farkas wrote:
> Hi Magnus,
> 
> I think the parameter you are looking for is there in section  4.2.2.3. Packet
> Ordering Function Processing (
> https://tools.ietf.org/id/draft-ietf-detnet-mpls-11.html#pof-requirements),
> just encoded a different way; in particular in the number of packets:
> “Note that an implementation MAY wish to constrain the maximum number of out
> of order packets that can be processed, on platform-wide or per flow basis.
> Some implementations MAY support the provisioning of this number on either a
> platform-wide or per flow basis. The number of out of order packets that can
> be processed also impacts the latency of a flow.”
> 
> Ultimately, the latency requirements of flows have to be met. The maximum
> packet size of a DetNet flow is known, see item c) MaxPayloadSize in 
> https://tools.ietf.org/html/draft-ietf-detnet-flow-information-model-10#section-5.5
>  and max-payload-size in 
> https://tools.ietf.org/html/draft-ietf-detnet-yang-07. 
> So, maximum POF buffer size to be applied for a flow can be determined; can be
> converted to time and taken into account during latency engineering of a
> DetNet network; thus bounded latency can be provided.

So given that the buffer is specified in either bytes or simply packets to be
buffered will result in that the POF buffering time becomes packet flow
dependent and not bounded in time. So if you make the calculation for a DETNET
flow thinking it will send 500 packets per second evenly spaced. Then the a
buffer of 5 packets would represent an upper limit 1/100th of second. If then
the flow sends only 100 packet per second then suddendly the 5-packet buffering
would represent 1/20th of a second. Thus defining it in packets or size doesn't
work, the upper buffering time needs to be defined in time to provide a bounded
latency. 

> 
> I fully agree with you that this needs to be taken into account. Together with
> what the set of documents that specify DetNet provide, I think it is fine what
> we have in the MPLS data plane draft. When it comes to configuration - as you
> pointed out - I think the job we need to do is to pay attention to do it right
> in the YANG draft, which is still under development.

From my perspective I think this document needs to correctly document the
requirement to bound in time the POF and having a configuration. The actual
detail of the configuration is fine to leave to another document. 

Cheers

Magnus Westerlund

---

From: Janos Farkas <[email-masked]>
Date: Thu, 10 Sep 2020 13:26:10 +0000

Hi Magnus,

I think we have the very same goal: make DetNet work; avoid any bugs.

I may be wrong, but as far as I know, multiple methods could be used for buffer dimensioning, e.g.: bytes, number of packets, time; and if a good job is done, then one can be converted to another.

I think we can make DetNet work if the POF buffer size is given in number of packets if we do a good job with the traffic specification.
The flow information model draft (https://tools.ietf.org/html/draft-ietf-detnet-flow-information-model-10#section-5.5) provides the following:

   TrafficSpecification has the following attributes:

   a.  Interval: the period of time in which the traffic specification
       is specified.

   b.  MaxPacketsPerInterval: the maximum number of packets that the
       Ingress will transmit in one Interval.

   c.  MaxPayloadSize: the maximum payload size that the Ingress will
       transmit.

Thus, my interpretation of your point is that the TrafficSpecification is not good enough to make the conversion between time and number of packets.

My preference is to leave number of packets for POF int the MPLS data plane draft and make it sure that the TrafficSpecification provides all the parameters needed for engineering to convert between number of packets and time.

Please let us know if we have overlooked anything with regards to the TrafficSpecification and such conversion is not possible. If there is any issue with the TrafficSpecification, then I think we should fix it.

Regards,
Janos

-----Original Message-----
From: Magnus Westerlund <[email-masked]> 
Sent: Thursday, September 10, 2020 10:19 AM
To: Janos Farkas <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Magnus Westerlund's Discuss on draft-ietf-detnet-mpls-11: (with DISCUSS)

On Wed, 2020-09-09 at 20:34 +0000, Janos Farkas wrote:
> Hi Magnus,
> 
> I think the parameter you are looking for is there in section  
> 4.2.2.3. Packet Ordering Function Processing ( 
> https://tools.ietf.org/id/draft-ietf-detnet-mpls-11.html#pof-requireme
> nts), just encoded a different way; in particular in the number of 
> packets:
> “Note that an implementation MAY wish to constrain the maximum number 
> of out of order packets that can be processed, on platform-wide or per flow basis.
> Some implementations MAY support the provisioning of this number on 
> either a platform-wide or per flow basis. The number of out of order 
> packets that can be processed also impacts the latency of a flow.”
> 
> Ultimately, the latency requirements of flows have to be met. The 
> maximum packet size of a DetNet flow is known, see item c) 
> MaxPayloadSize in
> So, maximum POF buffer size to be applied for a flow can be 
> determined; can be converted to time and taken into account during 
> latency engineering of a DetNet network; thus bounded latency can be provided.

So given that the buffer is specified in either bytes or simply packets to be buffered will result in that the POF buffering time becomes packet flow dependent and not bounded in time. So if you make the calculation for a DETNET flow thinking it will send 500 packets per second evenly spaced. Then the a buffer of 5 packets would represent an upper limit 1/100th of second. If then the flow sends only 100 packet per second then suddendly the 5-packet buffering would represent 1/20th of a second. Thus defining it in packets or size doesn't work, the upper buffering time needs to be defined in time to provide a bounded latency. 

> 
> I fully agree with you that this needs to be taken into account. 
> Together with what the set of documents that specify DetNet provide, I 
> think it is fine what we have in the MPLS data plane draft. When it 
> comes to configuration - as you pointed out - I think the job we need 
> to do is to pay attention to do it right in the YANG draft, which is still under development.

From my perspective I think this document needs to correctly document the requirement to bound in time the POF and having a configuration. The actual detail of the configuration is fine to leave to another document. 

Cheers

Magnus Westerlund

---

From: Lou Berger <[email-masked]>
Date: Thu, 10 Sep 2020 10:17:57 -0400

Magnus,

I agree with Janos here, I think this should be addressed in the traffic 
specification.  This said, I think it completely reasonable to add a 
statement in this document to that effect so this interaction isn't 
missed by the implementer.

How about adding the following to the end of the last paragraph in 
section 4.2.2.3:

    The latency impact on the system resources needed to support a
    specific DetNet flow will need to be evaluated by the controller plane
    based on that flow's traffic specification.  An example traffic 
specification
    that can be used with MPLS with Traffic Engineering (MPLS-TE) can be
    found in [RFC2212].

Does this address your concern?

Lou

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 10 Sep 2020 15:31:31 +0000

On Thu, 2020-09-10 at 13:26 +0000, Janos Farkas wrote:
> I may be wrong, but as far as I know, multiple methods could be used for
> buffer dimensioning, e.g.: bytes, number of packets, time; and if a good job
> is done, then one can be converted to another.
> 
> I think we can make DetNet work if the POF buffer size is given in number of
> packets if we do a good job with the traffic specification.
> The flow information model draft (
> ) provides the following:
> 
> Thus, my interpretation of your point is that the TrafficSpecification is not
> good enough to make the conversion between time and number of packets.

Exactly, this specification is good when one like ensure capability can handle
the traffic by specifying the maxmimum kind of resource the flow will consume.
But it doesn't work for calculate a bounded delay as by lowering the number of
packets per second actually sent, the actual latency increases in the POF if it
is configured only on number of packets. To bound latency based on number of
packets, in that case you have to enforce that the flow have a minimal packet
interval. 

> 
> My preference is to leave number of packets for POF int the MPLS data plane
> draft and make it sure that the TrafficSpecification provides all the
> parameters needed for engineering to convert between number of packets and
> time.

So the number of packets is a relevant aspect, but as the dataplane draft states
it is the combination of time and packets that bounds the POF function. 

I have no opinion if the TrafficSpecification is the right place to put this
configuration boundary. It is really not a traffic flow property, it is more a
configuration of the network from my perspecitive. And given a bound latency
requirement from the Detnet flow the POF buffer depth may be different depending
on what path through the MPLS network one uses. 

> 
> Please let us know if we have overlooked anything with regards to the
> TrafficSpecification and such conversion is not possible. If there is any
> issue with the TrafficSpecification, then I think we should fix it.
> 

To be clear i don't want to tell the WG how to do its work. I have noted what I
consider a serious short comming, one that I think needs to be solved. I am
happy to discuss it and the proposed way forward. 

Cheers

Magnus Westerlund

---

From: Lou Berger <[email-masked]>
Date: Thu, 10 Sep 2020 13:15:27 -0400

Magnus,

I agree that minimum sized frames are needed as well.  Please see the 
text I proposed in my last message and note that the reference traffic 
specification (RFC2212) includes a minimum policed unit (m), and the 
maximum   datagram size (M).

Lou

---

From: Balázs Varga A <[email-masked]>
Date: Mon, 14 Sep 2020 13:38:52 +0000

Hi,

DetNet networks need strong control of network resources. The MPLS 
data plane draft specifies the DetNet specific encapsulation and the 
role of DetNet functions (e.g., PREOF for the DetNet service 
sub-layer). There can be multiple implementation variants of PEF and 
POF functions, based on various algorithms, however, they are 
out-of-scope of in this document.
  
In order to address the raised concerns the following change is 
proposed. Text to be added to the end of “4.2.2.3.  Packet Ordering 
Function Processing”:
--- NEW TEXT
       DetNet uses flow-specific requirements (e.g., maximum
       number of out-of-order packets, maximum latency of the flow) for
       configuration of POF-related buffers.  POF implementation details
       are out-of-scope in this document. POF configuration parameters
       – determined by the controller plane – are implementation specific.
       In case a PEF is combined with POF, then their configurations need 
       to be coordinated to ensure the combined functionality.

Cheers
Bala'zs

-----Original Message-----
From: Lou Berger <[email-masked]> 
Sent: Thursday, September 10, 2020 7:15 PM
To: Magnus Westerlund <[email-masked]>; Janos Farkas <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Magnus Westerlund's Discuss on draft-ietf-detnet-mpls-11: (with DISCUSS)

Magnus,

I agree that minimum sized frames are needed as well.  Please see the text I proposed in my last message and note that the reference traffic specification (RFC2212) includes a minimum policed unit (m), and the maximum   datagram size (M).

Lou

On 9/10/2020 11:31 AM, Magnus Westerlund wrote:
>> I may be wrong, but as far as I know, multiple methods could be used 
>> for buffer dimensioning, e.g.: bytes, number of packets, time; and if 
>> a good job is done, then one can be converted to another.
>>
>> I think we can make DetNet work if the POF buffer size is given in 
>> number of packets if we do a good job with the traffic specification.
>> Thus, my interpretation of your point is that the 
>> TrafficSpecification is not good enough to make the conversion between time and number of packets.
> Exactly, this specification is good when one like ensure capability 
> can handle the traffic by specifying the maxmimum kind of resource the flow will consume.
> But it doesn't work for calculate a bounded delay as by lowering the 
> number of packets per second actually sent, the actual latency 
> increases in the POF if it is configured only on number of packets. To 
> bound latency based on number of packets, in that case you have to 
> enforce that the flow have a minimal packet interval.
>
>> My preference is to leave number of packets for POF int the MPLS data 
>> plane draft and make it sure that the TrafficSpecification provides 
>> all the parameters needed for engineering to convert between number 
>> of packets and time.
> So the number of packets is a relevant aspect, but as the dataplane 
> draft states it is the combination of time and packets that bounds the POF function.
>
> I have no opinion if the TrafficSpecification is the right place to 
> put this configuration boundary. It is really not a traffic flow 
> property, it is more a configuration of the network from my 
> perspecitive. And given a bound latency requirement from the Detnet 
> flow the POF buffer depth may be different depending on what path through the MPLS network one uses.
>
>> TrafficSpecification and such conversion is not possible. If there is 
>> any issue with the TrafficSpecification, then I think we should fix it.
>>
> To be clear i don't want to tell the WG how to do its work. I have 
> noted what I consider a serious short comming, one that I think needs 
> to be solved. I am happy to discuss it and the proposed way forward.
>
>
> Cheers
>
> Magnus Westerlund
>
>

---

From: Balázs Varga A <[email-masked]>
Date: Tue, 15 Sep 2020 08:44:25 +0000

Loa missed to send to the workgroup list as well, so forwarded now ...

-----Original Message-----
From: Loa Andersson <[email-masked]> 
Sent: Tuesday, September 15, 2020 6:54 AM
To: Balázs Varga A <[email-masked]>
Subject: Re: [Detnet] Magnus Westerlund's Discuss on draft-ietf-detnet-mpls-11: (with DISCUSS)

Balázs,

I can live with this, but have two snmall-ish comments.

>         to be coordinated to ensure the combined functionality.

  "DetNet uses flow-specific requirements..."

For me "requirements are something we documented and use to verify if an implementation is up to what we intended.

I think the sentence could be " "DetNet uses flow-specific parameterss..."

We also have

"POF configuration parameters – ... – are implementation specific."

I would say no, they can be more specific that that, they are service (or maybe flow) specific, at leas in theory a single implementation might support more than just one set of flow parameters.

/Loa

--

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 17 Sep 2020 14:29:00 +0000

Hi,

Okay, will clear my discuss based on the proposed changes. This is based on that
you will want these functions as black boxes that are implementation dependent.
However, the new text do inidcate the bare minimal requirements on the control
plane to achieve the service. So I think the working group will have some work
to do to figure out what knobs or information elements you need in the control
plane to make this work so that bound delay also when the traffic flow operates
at packet rates and bandwidths below the maximum according to any traffic
specification. 

Please consider if you need to adjust the abstract and introduction to correctly
represent what is provided. 

Cheers

Magnus

On Mon, 2020-09-14 at 13:38 +0000, Balázs Varga A wrote:
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
> ; [email-masked]; [email-masked]
> I agree that minimum sized frames are needed as well.  Please see the text I
> proposed in my last message and note that the reference traffic specification
> (RFC2212) includes a minimum policed unit (m), and the maximum   datagram size
> (M).
> 
> Lou
> 
-- 
Cheers

Magnus Westerlund 

----------------------------------------------------------------------
Networks, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Mobile +46 73 0949079
Torshamnsgatan 23           |
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------