Subject: Re: [Detnet] Questions about the Networks in DetNet charter

From: Gyan Mishra <[email-masked]>
Date: Sun, 5 Jul 2020 22:35:54 -0400

Detnet WG

I did not hear back anything on this question I had so was just wondering
on the correlation between Detnet and non ECMP type flows.

Also it does state in the charter but how was the WG name Detnet history?

Kind Regards

Gyan

On Sun, Jun 28, 2020 at 6:51 PM Gyan Mishra <[email-masked]> wrote:

>
> Deterministic networking by definition means the inbound and outbound
> paths use the same set of nodes and links for the flow.
>
> So generally that means no ECMP per packet or per flow as IP ECMP is
> traditionally an XOR source/destination hash and so is deemed non
> deterministic predictable path taken by a flow.
>
> Thanks
>
> Gyan
>
> On Sun, Jun 28, 2020 at 6:17 PM Gyan Mishra <[email-masked]> wrote:
>
>>
>> My last paragraph related to my question fixed.
>>
>> I noticed in the charter or RFC 8578 does not mention non ECMP routing as
>> deemed deterministic routing used in Detnet.
>>
>> Thanks
>>
>> Gyan
>>
>> On Sun, Jun 28, 2020 at 6:01 PM Gyan Mishra <[email-masked]>
>> wrote:
>>
>>>
>>> I had a generic question related to the Detnet WG charter.
>>>
>>> So I understand the focus of latency and jitter time sensitive
>>> applications such as voice and video over any single administrative domain
>>> which is covers am any carriers network including RAN 4G/5G.  Am good with
>>> all that.
>>>
>>> To me the term “deterministic” from a network routing perspective
>>> historically precluded ECMP flow based load balancing or for example vxlan
>>> source port entropy IP ECMP load balancing.   Since IP ECMP load balancing
>>> is all flow based technically UDP voice and video flows are not impacted as
>>> IP ECMP is prevalent in most all providers and enterprises.  In the past,
>>> IP per packet load existed and that reeked havoc with voice and video UDP
>>> RTP based flows out of order packets.  Since then most vendors due to
>>> issues with per packet load balancing over multiple paths have been
>>> eliminated.
>>> In my mind anything other then per packet load balanced flows is
>>> deterministic.
>>>
>>> Deterministic routing I would think means in the simplest sense a non
>>> load balanced single path with no ECMP along the entire path.  Another way
>>> to look at is that you can see the flow along any hop in the path if you
>>> did a packet capture where non deterministic path for a flow that would not
>>> be possible.
>>>
>>> I noticed in the charter it does mention ECMP Verdi’s not or in RFC
>>> 8578.
>>>
>>> What does the deterministic mean in the context of the WG framework as
>>> it does not seem related to deterministic routing.
>>>
>>>
>>> Kind Regards
>>>
>>> Gyan
>>>
>>> On Sun, Jun 28, 2020 at 5:35 PM Janos Farkas <Janos.Farkas=
>>> [email-masked]> wrote:
>>>
>>>> Hi Quan.
>>>>
>>>>
>>>>
>>>> This was my previous response.
>>>>
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Janos
>>>>
>>>>
>>>>
>>>> *From:* Janos Farkas
>>>> *Sent:* Monday, June 22, 2020 10:54 AM
>>>> *To:* [email-masked]
>>>> *Cc:* [email-masked]; [email-masked];
>>>> [email-masked]
>>>> *Subject:* RE: Questions about the Networks in DetNet charter
>>>>
>>>>
>>>>
>>>> Hi Quan.
>>>>
>>>>
>>>>
>>>> The main purpose of the cited sentence is to limit the scope of the
>>>> work to make it reasonable, i.e., not trying to boil the ocean. Actually,
>>>> the main message is that DetNet is not for the big I Internet, but for
>>>> smaller networks than that.
>>>>
>>>>
>>>>
>>>> Networks like mobile backhaul are definitely in scope of DetNet. It is
>>>> actually explicitly there in the DetNet Use Cases
>>>> https://tools.ietf.org/html/rfc8578#section-6.2.2.
>>>>
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Janos
>>>>
>>>>
>>>>
>>>> *From:* [email-masked] <[email-masked]>
>>>> *Sent:* Monday, June 22, 2020 4:29 AM
>>>> *To:* [email-masked]; Janos Farkas <[email-masked]>;
>>>> [email-masked]
>>>> *Cc:* [email-masked]
>>>> *Subject:* Questions about the Networks in DetNet charter
>>>>
>>>>
>>>>
>>>> Dear Chairs,
>>>>
>>>>
>>>>
>>>> I noticed that in DetNet Charter, it mentions that the networks which
>>>> WG foucs on as following shown.
>>>>
>>>>
>>>>
>>>> "The Working Group will initially focus on solutions for networks that
>>>> are under a single administrative control or within a closed group of
>>>> administrative control; these include not only campus-wide networks but
>>>> also can include private WANs. The DetNet WG will not spend energy on
>>>> solutions for large groups of domains such as the Internet."
>>>>
>>>>
>>>>
>>>> Could you please clarify that the WAN  such as Metropolitan area
>>>> network and Mobile backhaul network is included in DetNet use case or not?
>>>>
>>>> And does the DetNet WG only focus on the small networks similier with
>>>> TSN?
>>>>
>>>>
>>>>
>>>> Your reply is important and appreciated.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Quan
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> detnet mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/detnet
>>>>
>>> --
>>>
>>> <http://www.verizon.com/>
>>>
>>> *Gyan Mishra*
>>>
>>> *Network Solutions A**rchitect *
>>>
>>>
>>>
>>> *M 301 502-134713101 Columbia Pike *Silver Spring, MD
>>>
>>> --
>>
>> <http://www.verizon.com/>
>>
>> *Gyan Mishra*
>>
>> *Network Solutions A**rchitect *
>>
>>
>>
>> *M 301 502-134713101 Columbia Pike *Silver Spring, MD
>>
>> --
>
> <http://www.verizon.com/>
>
> *Gyan Mishra*
>
> *Network Solutions A**rchitect *
>
>
>
> *M 301 502-134713101 Columbia Pike *Silver Spring, MD
>
> --

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*M 301 502-134713101 Columbia Pike *Silver Spring, MD

---

From: Lou Berger <[email-masked]>
Date: Mon, 6 Jul 2020 15:20:38 -0400

On 7/5/2020 10:35 PM, Gyan Mishra wrote:
> Detnet WG
>
> I did not hear back anything on this question I had so was just 
> wondering on the correlation between Detnet and non ECMP type flows.

Hi,

multiple documents recommend against use of ECMP, e.g., 
draft-ietf-detnet-data-plane-framework:

          3.5.1.5
          <https://tools.ietf.org/html/draft-ietf-detnet-data-plane-framework-06#section-3.5.1.5>.
          Load sharing

    Use of packet-by-packet load sharing of the same DetNet flow over
    multiple paths is not recommended except for the cases listed above
    where PREOF is utilized to improve protection of traffic and maintain
    order.  Packet-by-packet load sharing, e.g., via ECMP or UCMP,
    impacts ordering and possibly jitter.

also, see https://www.google.com/search?q=draft-ietf-detnet+ecmp

> Also it does state in the charter but how was the WG name Detnet history?

I'm not sure what the question is here. 
https://datatracker.ietf.org/wg/detnet has the history and record for 
past meetings and activities.

Lou

>
> Kind Regards
>
> Gyan
>
> On Sun, Jun 28, 2020 at 6:51 PM Gyan Mishra <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     Deterministic networking by definition means the inbound and
>     outbound paths use the same set of nodes and links for the flow.
>
>     So generally that means no ECMP per packet or per flow as IP ECMP
>     is traditionally an XOR source/destination hash and so is deemed
>     non deterministic predictable path taken by a flow.
>
>     Thanks
>
>     Gyan
>
>     On Sun, Jun 28, 2020 at 6:17 PM Gyan Mishra <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>
>         I noticed in the charter or RFC 8578 does not mention non ECMP
>         routing as deemed deterministic routing used in Detnet.
>
>         Thanks
>
>         Gyan
>
>         On Sun, Jun 28, 2020 at 6:01 PM Gyan Mishra
>         <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>             So I understand the focus of latency and jitter time
>             sensitive applications such as voice and video over any
>             single administrative domain which is covers am any
>             carriers network including RAN 4G/5G.  Am good with all that.
>
>             To me the term “deterministic” from a network routing
>             perspective historically precluded ECMP flow based load
>             balancing or for example vxlan source port entropy IP ECMP
>             load balancing.   Since IP ECMP load balancing is all flow
>             based technically UDP voice and video flows are not
>             impacted as IP ECMP is prevalent in most all providers and
>             enterprises.  In the past, IP per packet load existed and
>             that reeked havoc with voice and video UDP RTP based flows
>             out of order packets.  Since then most vendors due to
>             issues with per packet load balancing over multiple paths
>             have been eliminated.
>             In my mind anything other then per packet load balanced
>             flows is deterministic.
>
>             Deterministic routing I would think means in the simplest
>             sense a non load balanced single path with no ECMP along
>             the entire path.  Another way to look at is that you can
>             see the flow along any hop in the path if you did a packet
>             capture where non deterministic path for a flow that would
>             not be possible.
>
>             I noticed in the charter it does mention ECMP Verdi’s not
>             or in RFC 8578.
>
>             What does the deterministic mean in the context of the WG
>             framework as it does not seem related to deterministic
>             routing.
>
>
>             Kind Regards
>
>             Gyan
>
>             On Sun, Jun 28, 2020 at 5:35 PM Janos Farkas
>             <Janos.Farkas=[email-masked]
>             <mailto:40ericsson.com@dmarc..ietf.org>> wrote:
>
>                 Hi Quan.
>
>                 *To:* [email-masked]
>                 <mailto:[email-masked]>
>                 *Cc:* [email-masked]
>                 <mailto:[email-masked]>; [email-masked]
>                 <mailto:[email-masked]>; [email-masked]
>                 <mailto:[email-masked]>
>                 The main purpose of the cited sentence is to limit the
>                 scope of the work to make it reasonable, i.e., not
>                 trying to boil the ocean. Actually, the main message
>                 is that DetNet is not for the big I Internet, but for
>                 Networks like mobile backhaul are definitely in scope
>                 of DetNet. It is actually explicitly there in the
>                 DetNet Use Cases
>                 *From:* [email-masked]
>                 <mailto:[email-masked]> <[email-masked]
>                 <mailto:[email-masked]>>
>                 *To:* [email-masked]
>                 <mailto:[email-masked]>; Janos Farkas
>                 <[email-masked]
>                 <mailto:[email-masked]>>; [email-masked]
>                 <mailto:[email-masked]>
>                 <mailto:[email-masked]>
>                 I noticed that in DetNet Charter, it mentions that the
>                 networks which WG foucs on as following shown.
>
>                 "The Working Group will initially focus on solutions
>                 for networks that are under a single administrative
>                 control or within a closed group of administrative
>                 control; these include not only campus-wide networks
>                 but also can include private WANs. The DetNet WG will
>                 not spend energy on solutions for large groups of
>                 domains such as the Internet."
>
>                 Could you please clarify that the WAN  such
>                 as Metropolitan area network and Mobile backhaul
>                 network is included in DetNet use case or not?
>
>                 And does the DetNet WG only focus on the small
>                 networks similier with TSN?
>
>                 [email-masked] <mailto:[email-masked]>
>             /Network Solutions A//rchitect /
>
>             /M 301 502-1347
>             13101 Columbia Pike
>             /Silver Spring, MD
>
>
>         -- 
>
>         /Network Solutions A//rchitect /
>
>         /M 301 502-1347
>         13101 Columbia Pike
>         /Silver Spring, MD
>
>
>     -- 
>
>     /Network Solutions A//rchitect /
>
>     /M 301 502-1347
>     13101 Columbia Pike
>     /Silver Spring, MD
>
>
> -- 
>
> /Network Solutions A//rchitect /
>
> /M 301 502-1347
> 13101 Columbia Pike
> /Silver Spring, MD
>
>
>

---

From: "Grossman, Ethan A." <[email-masked]>
Date: Tue, 7 Jul 2020 02:36:28 +0000

Hi Gyan,
    > Also it does state in the charter but how was the WG name Detnet history?
Lou wrote: 

If you are asking about the history of how the DetNet workgroup came to be, awhile ago I wrote up a brief summary from my perspective, which I can paraphrase here: 
---------------------------------------
In 2013 I became my employer's Technical Representative to the Avnu Alliance, an industry group dedicated to propagating the AVB technology. At that time Avnu had a “Pro AV” (Professional Audio and Video) subgroup that I joined, and also around that time the AES67 standard was introduced, which was the start of the broadcast industry’s interest in using packet based IP/Ethernet networks to replace dedicated point-to-point HW technologies for audio and video in broadcast production. It became clear to us in Avnu that we needed to figure out how to access the AVB services from the IP layer. 
 
The broadcast industry did a formal “gap analysis” about what was needed to transition the whole broadcast industry to packet-based technologies, and we at Avnu submitted a response pitching AVB as a useful technology. They basically replied that a LAN technology didn’t solve many of their connectivity use cases, they needed an IP based solution, regardless of whether the performance was guaranteed or not. 
 
Then Cisco sent Norm Finn to join Avnu Alliance, and he realized that we (i.e. Avnu) had to go to the IETF to address this. So he organized a BOF (at IETF 93 in Prague in 2015) in which he issued a call for use cases for deterministic IP networking, and various people presented the core use cases that eventually became the core of RFC8578 (DetNet Use Cases), including the Pro Audio and Video use case that I co-authored. I didn’t attend that BOF, but I shortly thereafter became editor of the DetNet Use Cases draft, and dropped my participation in Avnu to devote my time to DetNet.
--------------------------------------

Hope that helps. 
Ethan.

-----Original Message-----
From: Lou Berger <[email-masked]> 
Sent: Monday, July 6, 2020 12:21 PM
To: Gyan Mishra <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [Detnet] Questions about the Networks in DetNet charter

Hi,

          3.5.1.5

Lou

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 7 Jul 2020 00:04:37 -0400

Lou

Thank you for responding to my questions.

That is exactly what I was looking for.

3.5.1.5 <https://tools.ietf.org/html/draft-ietf-detnet-data-plane-framework-06#section-3.5.1.5>.
Load sharing

>From a network load balancing perspective most operators, both service
providers and enterprises have voice and video as well as data that rides
the common converged network for years now, and so “source / destination”
per flow based hardware switching load balancing  is the standard for
almost every vendor.  With Flow based load balancing all packets that are
part of the same source / destination flow are hashed to the same path via
the vendors ECMP hash algorithm which is usually an source / destination
XOR.  So flow based load balancing flows since all packets traverse the
same path are not subject to out of order packets or latency and jitter.
However, QOS end to end is a must for both unicast and multicast voice and
video as all traffic is RTP/UDP based to prevent queueing and head of line
blocking with small packets queuing up behind large packets.

In the past,  older routers has the option to do per packet load balancing
process switched, but due to issues with processing per packet in software
platforms as well as latency and jitter and out of order packets most if
not all vendors have moved away from per packet load balancing to hardware
based platforms flow based ECMP.

So my point was as to the name of the WG “Detnet” as deterministic voice
and video latency and jitter sensitive flows would get same path source /
destination address which has existed for 25+ years now by most vendors
hardware switched  flow based ECMP and all voice and video plays nicely and
is works well.

The concept of per packet load balancing  is really graconian in my mind
and non existent.

Kind regards

Gyan

On Mon, Jul 6, 2020 at 3:20 PM Lou Berger <[email-masked]> wrote:

>           <
> https://tools.ietf.org/html/draft-ietf-detnet-data-plane-framework-06#section-3.5.1.5
> >.
>           Load sharing
>
>
>
-- 

*Gyan Mishra*

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 7 Jul 2020 00:08:23 -0400

Thank you Ethan for the history of Detnet!!

Kind Regards

Gyan

On Mon, Jul 6, 2020 at 10:36 PM Grossman, Ethan A. <[email-masked]> wrote:

> If you are asking about the history of how the DetNet workgroup came to
> be, awhile ago I wrote up a brief summary from my perspective, which I can
> paraphrase here:
> In 2013 I became my employer's Technical Representative to the Avnu
> Alliance, an industry group dedicated to propagating the AVB technology. At
> that time Avnu had a “Pro AV” (Professional Audio and Video) subgroup that
> I joined, and also around that time the AES67 standard was introduced,
> which was the start of the broadcast industry’s interest in using packet
> based IP/Ethernet networks to replace dedicated point-to-point HW
> technologies for audio and video in broadcast production. It became clear
> to us in Avnu that we needed to figure out how to access the AVB services
> from the IP layer.
>
> The broadcast industry did a formal “gap analysis” about what was needed
> to transition the whole broadcast industry to packet-based technologies,
> and we at Avnu submitted a response pitching AVB as a useful technology.
> They basically replied that a LAN technology didn’t solve many of their
> connectivity use cases, they needed an IP based solution, regardless of
> whether the performance was guaranteed or not.
>
> Then Cisco sent Norm Finn to join Avnu Alliance, and he realized that we
> (i.e. Avnu) had to go to the IETF to address this. So he organized a BOF
> (at IETF 93 in Prague in 2015) in which he issued a call for use cases for
> deterministic IP networking, and various people presented the core use
> cases that eventually became the core of RFC8578 (DetNet Use Cases),
> including the Pro Audio and Video use case that I co-authored. I didn’t
> attend that BOF, but I shortly thereafter became editor of the DetNet Use
> Cases draft, and dropped my participation in Avnu to devote my time to
> DetNet.
-- 

*Gyan Mishra*