Subject: [Raw] FW: New Version Notification for draft-pthubert-raw-architecture-07.txt

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 4 Jun 2021 13:49:44 +0000

Many thanks for your comments Greg!

I added subsections to section 4.3 of draft-pthubert-raw-architecture 07; can you please have a look?

Coincidently the new text affects the (todo) section 4.3 of draft-ietf-raw-oam-support.

Fabrice and all, it would be nice that we sync on this. We might also exchange pieces of text between the 2 drafts.

Keep safe;

Pascal

-----Original Message-----
From: [email-masked] <[email-masked]> 
Sent: vendredi 4 juin 2021 15:38
To: Georgios Z. Papadopoulos <[email-masked]>; Georgios Papadopoulos <[email-masked]>; Lou Berger <[email-masked]>; Pascal Thubert (pthubert) <[email-masked]>; Rex Buddenberg <[email-masked]>
Subject: New Version Notification for draft-pthubert-raw-architecture-07.txt

A new version of I-D, draft-pthubert-raw-architecture-07.txt
has been successfully submitted by Pascal Thubert and posted to the IETF repository.

Name:		draft-pthubert-raw-architecture
Revision:	07
Title:		Reliable and Available Wireless Architecture/Framework
Document date:	2021-06-04
Group:		Individual Submission
Pages:		35
URL:            https://www.ietf.org/archive/id/draft-pthubert-raw-architecture-07.txt
Status:         https://datatracker.ietf.org/doc/draft-pthubert-raw-architecture/
Html:           https://www.ietf.org/archive/id/draft-pthubert-raw-architecture-07.html
Htmlized:       https://datatracker.ietf.org/doc/html/draft-pthubert-raw-architecture
Diff:           https://www.ietf.org/rfcdiff?url2=draft-pthubert-raw-architecture-07

Abstract:
   Reliable and Available Wireless (RAW) provides for high reliability
   and availability for IP connectivity over a wireless medium.  The
   wireless medium presents significant challenges to achieve
   deterministic properties such as low packet error rate, bounded
   consecutive losses, and bounded latency.  This document defines the
   RAW Architecture.  It builds on the DetNet Architecture and discusses
   specific challenges and technology considerations needed to deliver
   DetNet service utilizing scheduled wireless segments and other media,
   e.g., frequency/time-sharing physical media resources with stochastic
   traffic.

The IETF Secretariat

---

From: Fabrice Theoleyre <[email-masked]>
Date: Fri, 4 Jun 2021 19:22:57 +0200

Dear Pascal, 

Thanks for your comments.

I expect to have some time at the end of the next week to read both documents and to react :-)

Cheers,
Fabrice

> Le 4 juin 2021 à 15:49, Pascal Thubert (pthubert) <[email-masked]> a écrit :
>

---

From: Fabrice Theoleyre <[email-masked]>
Date: Mon, 7 Jun 2021 17:19:15 +0200

Hi Pascal,

I just read your draft. 

For the ingress / egress end-systems in section 4.3, shouldn’t you use the term MEP? 
Besides, I have the feeling that MEP should be the hosts at the border of the wireless part: in that way, you focus uniquely on the wireless links to collect all the measurements, etc.
It would be easier to manage. 

What’s your opinion?

Fabrice

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 7 Jun 2021 15:26:34 +0000

Good point, Fabrice

I'll add text saying that ingress and egress serve as Maintenance Entity Group End Points.

Note that the role of ingress covers more stuff like track selection, encapsulation and PSE.

Keep safe!

Pascal

> From: Fabrice Theoleyre <[email-masked]>
> Sent: lundi 7 juin 2021 17:19
> To: Pascal Thubert (pthubert) <[email-masked]>
> Cc: Greg Mirsky <[email-masked]>; [email-masked];
> [email-masked]; [email-masked]; Carlos J. Bernardos
> <[email-masked]>
> Subject: Re: New Version Notification for draft-pthubert-raw-architecture-
> 07.txt
> 
> Hi Pascal,
> 
> I just read your draft.
> 
> Besides, I have the feeling that MEP should be the hosts at the border of the
> wireless part: in that way, you focus uniquely on the wireless links to
> collect all the measurements, etc.
> > Fabrice and all, it would be nice that we sync on this. We might also
> exchange pieces of text between the 2 drafts.
> >
> > Keep safe;
> >
> > Pascal
> >
> > To: Georgios Z. Papadopoulos <[email-masked]>;
> Georgios Papadopoulos <[email-masked]>; Lou Berger
> <[email-masked]>; Pascal Thubert (pthubert) <[email-masked]>; Rex
> Buddenberg <[email-masked]>

---

From: <[email-masked]>
Date: Thu, 17 Jun 2021 03:39:18 +0800 (CST)

Hi Pascal,

apologies for a delay to respond. I've read the draft and believe it is a great document that is the absolute must for anyone who, as I am, is not intimately familiar with wireless media and the work on TISCH in the 6tisch WG. thank you for making to read it.

I've added several comments in the attached copy (along with some editorial knit picking). Please feel free to use the copy for the continued discussion or extract comments into an email.

Best regards,

Greg Mirsky

Sr. Standardization Expert
预研标准部/有线研究院/有线产品经营部 Standard Preresearch Dept./Wireline Product R&D Institute/Wireline Product Operation Division

E: [email-masked] 
www.zte.com.cn

Original Mail

Sender: PascalThubert(pthubert)
To: Fabrice Theoleyre;
CC: [email-masked];[email-masked];gregory mirsky10211915;Carlos J. Bernardos;[email-masked];
Date: 2021/06/07 08:27
Subject: Re: [Raw] New Version Notification for draft-pthubert-raw-architecture-07.txt

Good point, Fabrice

 
Keep safe!
 
Pascal

-- 
RAW mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/raw

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 17 Jun 2021 14:15:24 +0000

Hello Greg 😊

Many thanks for your great comments. Being inlined in a word document, they helps me a lot as author of the personal submission, but will not be seen by many people in the list and maybe lost in archival.
That’s a bit sad, there are many very interesting discussion topics in your comments. I’ll come back for discussion threads on individual topics. In the meantime I’m applying the most basic changes to the latest github version.

Again, many thanks!

Pascal

From: RAW <[email-masked]> On Behalf Of [email-masked]
Sent: mercredi 16 juin 2021 21:39
To: pthubert=[email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Raw] New Version Notification for draft-pthubert-raw-architecture-07.txt

Hi Pascal,

Best regards,

Greg Mirsky

E: [email-masked]<mailto:[email-masked]>
www.zte.com.cn<http://www.zte.com.cn/>
Original Mail
Sender: PascalThubert(pthubert)
To: Fabrice Theoleyre;
CC: [email-masked];[email-masked];gregory<mailto:[email-masked];[email-masked];gregory> mirsky10211915;Carlos J. Bernardos;[email-masked];
Date: 2021/06/07 08:27
Subject: Re: [Raw] New Version Notification for draft-pthubert-raw-architecture-07.txt
Good point, Fabrice

Keep safe!

Pascal

> From: Fabrice Theoleyre <[email-masked]<mailto:[email-masked]>>
> To: Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>>
> Cc: Greg Mirsky <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Carlos J. Bernardos
> <[email-masked]<mailto:[email-masked]>>
> > Le 4 juin 2021 à 15:49, Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>> a
> écrit :
> >
> > From: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
> > To: Georgios Z. Papadopoulos <[email-masked]<mailto:[email-masked]>>;
> Georgios Papadopoulos <[email-masked]<mailto:[email-masked]>>; Lou Berger
> <[email-masked]<mailto:[email-masked]>>; Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>>; Rex
> Buddenberg <[email-masked]<mailto:[email-masked]>>

--
RAW mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/raw

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 23 Jun 2021 16:47:16 +0000

Hello again Greg

> It might be that here we talk about the reliability. I see the availability as reflection of the reliability of the path over a period of time. In fact, media with the constant bit-rate, e.g., SONET/SDH, had availability defined and measured for decades. I’ve attempted to extend that model into a packet networks in draft-mirsky-ippm-epm<https://datatracker.ietf.org/doc/draft-mirsky-ippm-epm/>.

True, fixed.

> Is it that the affected process is not the transmission, i.e., the process of transmitting, but the reception of a packet?

I’d agree if transmitting == sending or emitting; which is often the way we (ab)use the word. Here it is understood as the act of passing from one to another.

> Not sure if 5G URLLC is a distinct technology different from, for example, massive Machine Type Communication (mMTC) using 5G

Arguably it’s all OFDMA but we want to focus on URLLC as it is the angle that seems to echo the DetNet requirements.

“

[BFD] detect faults in the path between an Ingress and an Egress

      forwarding engines, but is unaware of the complexity of a path

      with replication, and expects bidirectionality.  BFD considers

      delivery as success whereas with RAW the bounded latency can be as

      important as the delivery itself.

“

> Not necessarily. It appears that BFD Demand mode with unsolicited notifications may be more suitable then the Asynchronous BFD mode. The use of the Demand mode in MPLS is analyzed in draft-mirsky-bfd-demand<https://datatracker.ietf.org/doc/draft-mirsky-bfd-mpls-demand/>. It is possible to extend the scope the draft to an IP network.

The bottom line is that link up down is not key. The key is bounded latency. I changed the text to:
“
   *  [BFD] detect faults in the path between an Ingress and an Egress
      forwarding engines, but is unaware of the complexity of a path
      with replication, and expects bidirectionality.  BFD asynchronous
      mode considers delivery as success whereas with DetNet and RAW,
      the bounded latency can be as important as the delivery itself,
      and delivering too late is actually a failure.  BFD Demand mode
      with unsolicited notifications may be more suitable then the
      Asynchronous BFD mode.  The use of the Demand mode in MPLS is
      analyzed in [I-D.mirsky-bfd-mpls-demand] and similar
      considerations could apply to IP as well.
“

I need to read your draft to understand more. What can we do to address bounded latency conformance?
This was the point above, and it’s a MAC issue so pure PHY-based demand BFD cannot tell.

> The Error Performance Measurement<https://datatracker.ietf.org/doc/draft-mirsky-ippm-epm/> draft, referenced earlier, uses PM and FM OAM mechanisms to determine availability/unavailability of a path according to predefined SLA.

I added that information in the “DetNet OAM” section.

“

   The RAW OAM operation in the Network Plane observes either a full
   Track or subTracks that are being used at this time.

“
> Would there be a value in also monitoring available but not currently used subTracks using active OAM methods? It seems that information about unused links could be used by PSE in making packet forwarding decision.

Certainly! Service-Layer Assurance should be able to test each individual PREOF operation (in vs out). This might mean injecting a packet in the middle of the Track and extracting it a few hops later.

Replaced with

   RAW In-situ OAM operation in the Network Plane may observe either a
   full Track or subTracks that are being used at this time.  Additional
   out-of-band RAW OAM may observe all including unused segments to
   prepare for a rerouting decision.  Finally, the RAW Service Layer
   Assurance may observe the individual PAREO operation of a relay node
   to ensure that it is conforming; this might require injecting an OAM
   packet at an upstream point inside the Track and extracting that
   packet at another point downstream before it reaches the egress.

“

                                                     Within that

   framework, OAM messages follow the same forward path as the data

   packets and gather information about their individual treatment at

   each hop.  When the destination receives an OAM message, it gets a

   view on the full path or at least of a segment of the path from the

   source of the flow.

   In-situ OAM (IOAM) adds telemetry information about the experience of
   one packet within the packet itself [I-D.ietf-ippm-ioam-data],

“

> Note that such HbH mode is costly on transit nodes. It can be expected that e2e OAM that monitors the entire path from ingress to egress would be broadly used.

May be different for RAW. Here we need to switch between subTracks.

> HbH telemetry could be used, for example, to optimize subTrack. If the task is to monitor a particular subTrack, then e2e PM and FM OAM could be what an operator needs.

You lost me. I thought you were advocating for e2e. Please let me know if you need more text than the one I already added (see diffs)

> IOAM and analogous on-path telemetry methods are capable of facilitating collection of useful telemetry information that characterizes the state of a system as experienced by the packet. But because of statistical character of a packet network, these methods may not be used to monitor the continuity of a path (Track) or proper connectivity of the Track (no leaking packets across Tracks).

If you do not mind I’ll add that quote to the text.

“

   Classical OAM typically measures information at the transmitter,
   e.g., residence time in the node or transmit queue size.

“

> A residence time (RT) can be defined as time period between the reception of a packet starts and the transmission of the packet begins. If we use that, the RT is more useful for a transit node, not ingress or egress. Also, RT is for an ordered pair of ingress and egress interfaces of the given node. A special technique can be used in a path engineered environment where a test probe can be directed over an arbitrary path.

I added text including terminology based on the above. Many thanks! All this makes you  a contributor. I added your name if you do not mind?

“

   it is important that RAW OAM captures the state of the

   medium across an adjacency over multiple transmission and over a

   recent period of time, whether the transmitted packets belong to this

“
> That is a very important requirement for PM OAM in RAW (I think that the same should be true for DetNet too).

“

                                This makes an approach like HTS more

   suitable as it can trigger the capture of multiple information over

   a short period of time.  On the other hand, the PSE needs a

   continuous measurement stream where a single trigger is followed by a

   periodic follow up capture.

“
> If the goal is to allow PSE to evaluate the availability of a Track, then such function might be located at the egress of the Track (see the EPM draft). Then the function would notify forwarding controlling function of the PSE once conditions on the Track degraded below pre-defined threshold.

Yes, that is the intent when the PSE source routes the subTrack to be used.

“

   out-of-band OAM packets must circulate in the exact same
   fashion as the flows that they observe.

“
> Out-of-band packets can be used to collect measurement data, counters. SNMP query is one of examples. Out-of-band cannot be used for monitoring a data flow as packets outside the flow are scheduled, processed differently. Active OAM (per definition in RFC 7799) methods use specially constructed test packets. It is important, if not critical, that these test packets are in-band with the monitored data flow. Some environments, e.g., ECMP, present challenges to keeping test packets in-band with data packets.

Please see https://www.ietf.org/archive/id/draft-pthubert-detnet-ipv6-hbh-04.html; the out of band OAm can be tagged just like the DetNet flows and be subject to the same forwarding plane behavior. RAW is actually a use case for that. And the expectation includes alerting if the bounded latency is not obtained. Since it is DetNet I had to use path instead of Track but we are talking about the same thing.

Do I miss something?

Again, many thanks!

I pushed 08 with this, please see https://www.ietf.org/rfcdiff?url2=draft-pthubert-raw-architecture-08.txt

You all keep safe,

Pascal

From: Pascal Thubert (pthubert)
Sent: jeudi 17 juin 2021 16:15
To: [email-masked]; pthubert=[email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: RE: [Raw] New Version Notification for draft-pthubert-raw-architecture-07.txt

Hello Greg 😊

Again, many thanks!

Pascal

From: RAW <[email-masked]<mailto:[email-masked]>> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: mercredi 16 juin 2021 21:39
To: pthubert=[email-masked]<mailto:pthubert=[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Raw] New Version Notification for draft-pthubert-raw-architecture-07.txt

Hi Pascal,

Best regards,

Greg Mirsky

Keep safe!

Pascal

---

From: <[email-masked]>
Date: Fri, 25 Jun 2021 04:10:42 +0800 (CST)

Hi Pascal,
I greatly appreciate your kind consideration of my comments and thoughtful updates. Please find my notes in-lined and tagged by GIM>>.

Regards,
Greg Mirsky
------------------Original Mail------------------
Sender: PascalThubert(pthubert)
To: gregory mirsky10211915;pthubert=[email-masked];
CC: [email-masked];[email-masked];[email-masked];[email-masked];[email-masked];
Date: 2021/06/23 09:51
Subject: RE: [Raw]  New Version Notification for draft-pthubert-raw-architecture-07.txt
v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:url(#default#VML);} .shape {behavior:url(#default#VML);}" _ue_custom_node_="true">
Hello again Greg
> It might be that here we talk about the reliability. I see the availability as reflection of the reliability of the path over a period of time. In fact, media with the constant  bit-rate, e.g., SONET/SDH, had availability defined and measured for decades. I’ve attempted to extend that model into a packet networks in draft-mirsky-ippm-epm.
True, fixed.
> Not necessarily. It appears that BFD Demand mode with unsolicited notifications may be more suitable then the Asynchronous BFD mode. The  use of the Demand mode in MPLS is analyzed in  draft-mirsky-bfd-demand. It is possible to extend the scope the draft to an IP network.
GIM>> Thak you for adding the reference. I think that the draft on the Error Performance Measurement (EPM) adds SLO conformance criterea to well-understood path continuity. I believe that it is beneficial to an operator if we can reflect the state of a network in fewer, preferably a single, metric. Note, that in EPM we don't specify the type of the metric being used as SLO. It could be, for example, mean packet delay or 99 percentile. The goal is to combine SLOs with the state of the path. As you've noted, exceeding bounded delay is equivalent to losing the packet. On the other hand, losing a packet can be seen as infinite delay.

> The Error Performance Measurement draft, referenced earlier, uses PM and FM OAM mechanisms to determine availability/unavailability of a path according to predefined SLA.
GIM>> I am afraid that "Additional out-of-band RAW OAM" might confuse some readers. True, using active OAM methods to monitor redundant path that are not exrecised by the particular flow may be seen, relative to the specific RAW flow, as out-of-band. But, on the other hand, that flow of OAM packets must experience the same network treatment as if the RAW flow was traversing redndant path. I propose s/out-of-band/active/. What do you think?
GIM>> My pleasure.
GIM>> ;) I'll be honored, thank you.
“
GIM>> An active OAM, i.e., specially constructed test packet, is considered in-band with the monitored data flow when it traverses the same set of links and interfaces receiving the same QoS treatment as the monitored flow. We can refer to an active OAM being out-of-band if, for example, this path through the network is topologically the same as of data, but QoS treatment is different (e.g., lower CoS). But using active OAM in that manner might produce measurements that cannot be attributed to the data flow we intend to monitor. The out-of-band OAM can be used not to measure performance but to collect measurement results and the telemetry information in general. In that case, even the topological equivalency can be relaxed to visiting the set of nodes. That is what, in effect, is proposed in the HTS and iOAM Direct Export.
GIM>> I've centered on the topic of OAM and will read the entire document on the weekend.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 25 Jun 2021 08:07:54 +0000

Hello Greg:

Many thanks for your great insights. You're certainly welcome to contribute more text.

> GIM>> Thak you for adding the reference. I think that the draft on the
> Error Performance Measurement (EPM) adds SLO conformance criterea to well-
> understood path continuity. I believe that it is beneficial to an operator
> if we can reflect the state of a network in fewer, preferably a single,
> metric. Note, that in EPM we don't specify the type of the metric being
> used as SLO. It could be, for example, mean packet delay or 99 percentile.
> The goal is to combine SLOs with the state of the path. As you've noted,
> exceeding bounded delay is equivalent to losing the packet. On the other
> hand, losing a packet can be seen as infinite delay.

PT> That's the way RPL works actually. We expose many metrics (physical or statistical) but then there's a common "objective function" that is out of scope for the (generic) routing protocol and that digests those metrics (plus whatever logic / policy) to create a single abstract metrics, the rank. The rank is what the DV operation uses.

PT> In RAW, we need an abstract sense of the quality of the "rest of the way through that next hop" in terms of chances to deliver the packet within latency budget. Other constraints will bind different next hops together like their propension to introduce too many packets out of order when used together. This is also something that we need to know for the ordering function, though it is probably more difficult to evaluate and secondary, at least for low rates. 

>> "RAW In-situ OAM operation in the Network Plane may observe either a full
>> Track or subTracks that are being used at this time.  Additional out-of-
>> band RAW OAM may observe all including unused segments to prepare for a
>> rerouting decision.  Finally, the RAW Service Layer Assurance may observe
>> the individual PAREO operation of a relay node to ensure that it is
>> conforming; this might require injecting an OAM packet at an upstream
>> point inside the Track and extracting that packet at another point
>> downstream before it reaches the egress."

> GIM>> I am afraid that "Additional out-of-band RAW OAM" might confuse some
> readers. True, using active OAM methods to monitor redundant path that are
> not exrecised by the particular flow may be seen, relative to the specific
> RAW flow, as out-of-band. But, on the other hand, that flow of OAM packets
> must experience the same network treatment as if the RAW flow was
> traversing redndant path. I propose s/out-of-band/active/. What do you
> think?
> “

PT> All good. But I 'd appreciate your help on providing definitions in the terminology section to ensure that the reader is on the same page for the distinction between things like in situ, out-of-band and active. Based on your text below I produced this:

"

   AOM:  Operations, administration and management or operations,
      administration and maintenance are the processes, activities,
      tools, and standards involved with operating, administering,
      managing and maintaining any system, a Track in the context of
      RAW.

   Active OAM:  An active OAM is specially constructed test packet
      crafted to explore a particular Track, subTrack, or Segment of a
      Track.

   In-Band OAM:  An active OAM is considered in-band in the monitored
      Track when it traverses the same set of links and interfaces
      receiving the same QoS and PAREO treatment as the data flows that
      are injected in the Track.

   Out-of-Band OAM:  An active OAM is out-of-band if its datapath is
      topologically the same as of that of the data flow, but the QoS
      or PAREO treatment is different (e.g., lower CoS).

   Local Assurance OAM:  An active OAM is a local assurance OAM when it
      is covers the forwarding or service layer operation over a node, a
      segment, or a subTrack of the Track.  It is injected in the
      datapath and extracted from the datapath around the particular
      function or subnetwork (e.g., a relay providing a service layer
      replication point) that is being tested.

"

>> Please see  https://www.ietf.org/archive/id/draft-pthubert-detnet-ipv6-
>> hbh-04.html; the out of band OAm can be tagged just like the DetNet flows
>> and be subject to the same forwarding plane behavior. RAW is actually a
>> use case for that. And the expectation includes  alerting if the bounded
>> latency is not obtained. Since it is DetNet I had to use path instead of
>> Track but we are talking about the same thing.

> GIM>> An active OAM, i.e., specially constructed test packet, is
> considered in-band with the monitored data flow when it traverses the same
> set of links and interfaces receiving the same QoS treatment as the
> monitored flow. We can refer to an active OAM being out-of-band if, for
> example, this path through the network is topologically the same as of
> data, but QoS treatment is different (e.g., lower CoS). But using active
> OAM in that manner might produce measurements that cannot be attributed to
> the data flow we intend to monitor. The out-of-band OAM can be used not to
> measure performance but to collect measurement results and the telemetry
> information in general. In that case, even the topological equivalency can
> be relaxed to visiting the set of nodes. That is what, in effect, is
> proposed in the HTS and iOAM Direct Export.

Many thanks, I took the liberty to use your text for the terminology proposal above.

Please let me know what you think.

Many thanks!

Pascal