Subject: [mpls] Mail regarding draft-zzhang-tsvwg-generic-transport-functions

From: Stewart Bryant <[email-masked]>
Date: Tue, 10 Nov 2020 14:57:14 +0000

Please can I draw the attention of the authors to https://tools.ietf.org/html/rfc4623 <https://tools.ietf.org/html/rfc4623>

This standards track RFC  specifies how you can sent a fragmented Ethernet frame over a PW in an MPLS network and would seem applicable to the problem that you address in your draft.

BR 

Stewart

---

From: "Andrew G. Malis" <[email-masked]>
Date: Tue, 10 Nov 2020 10:33:21 -0500

Indeed, this is an already-solved problem.

Cheers,
Andy

On Tue, Nov 10, 2020 at 9:57 AM Stewart Bryant <[email-masked]>
wrote:

> Please can I draw the attention of the authors to
> https://tools.ietf.org/html/rfc4623
>
> This standards track RFC  specifies how you can sent a fragmented Ethernet
> frame over a PW in an MPLS network and would seem applicable to the problem
> that you address in your draft.
>
> BR
>
> Stewart
> _______________________________________________
> Pals mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pals
>

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Tue, 10 Nov 2020 23:13:28 +0000

Hi Andy, Stewart,

If I understand it correctly, RFC4623 is specifically for PWs (p2p) and cannot be used for EVPN/VPLS.

The reason is that the sequence number in the control word is specific to the PW and the fragmentation/reassembly is performed in the context of the PW. In case of EVPN/VPLS, an egress PE could receive fragments from different ingress PEs and reassembly must be done in the right context.

In addition, RFC 7432 (EVPN) specifically calls out that control word is either not used or only with all-0:

   - If a network uses deep packet inspection for its ECMP, then the
     "Preferred PW MPLS Control Word" [RFC4385<https://tools.ietf.org/html/rfc4385>] SHOULD be used with the
     value 0 (e.g., a 4-octet field with a value of zero) when sending
     EVPN-encapsulated packets over an MP2P LSP.

   - If a network uses entropy labels [RFC6790<https://tools.ietf.org/html/rfc6790>], then the control word
     SHOULD NOT be used when sending EVPN-encapsulated packets over an
     MP2P LSP.

   - When sending EVPN-encapsulated packets over a P2MP LSP or P2P LSP,
     then the control word SHOULD NOT be used.

This draft-zzhang allows the context to be determined from the extended “identification” field or from the outer header. In addition, it is “generic” such that it can be used for any situations where fragmentation is needed at any layer for any solution.

Thanks.

Jeffrey

From: Andrew G. Malis <[email-masked]>
Sent: Tuesday, November 10, 2020 10:33 AM
To: Stewart Bryant <[email-masked]>
Cc: [email-masked]; mpls <[email-masked]>; [email-masked]
Subject: Re: [Pals] Mail regarding draft-zzhang-tsvwg-generic-transport-functions

[External Email. Be cautious of content]

Cheers,
Andy

On Tue, Nov 10, 2020 at 9:57 AM Stewart Bryant <[email-masked]<mailto:[email-masked]>> wrote:
Please can I draw the attention of the authors to https://tools.ietf.org/html/rfc4623<https://urldefense.com/v3/__https:/tools.ietf.org/html/rfc4623__;!!NEt6yMaO-gk!WnDW7J-i0YOpcV5sF1KDAfZnAaxHY5z-pG0oiSXKdSXdg9o9pRoRlJeq1ENEdsSd$>

BR

Stewart
_______________________________________________
Pals mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/pals<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/pals__;!!NEt6yMaO-gk!WnDW7J-i0YOpcV5sF1KDAfZnAaxHY5z-pG0oiSXKdSXdg9o9pRoRlJeq1KRQDg8M$>

Juniper Business Use Only

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 11 Nov 2020 12:59:18 +0000

First some comments on the draft itself.

SB> The draft needs to discuss RFC4623 and explain when this draft would be used in place of RFC4623

   Some functionalities (e.g. fragmentation/reassembly and Encapsulating
   Security Payload) provided by IPv6 can be viewed as independent of
   IPv6 or even IP entirely.  
SB> Specifically IPv6 does not support network fragmentation because it provides PMTUD and devolves the problem to host.
This document proposes to provide those
   functionalities at different layers (e.g., MPLS, BIER or even
   Ethernet) independent of IP.

SB> I think we need to understand why after all these years this is now needed.
SB> Although we have this in PW, I am not sure it is widely used.

1.  Introduction

   Consider an operator providing Ethernet services such as pseudowires,
SB> It is already provided for PWs. Now you can build a case for why you need something better, but you must build the case.

   VPLS or EVPN.  The Ethernet frames that a Provider Edge (PE) device
   receives from a Customer Edge (CE) device may have a larger size than
   the PE-PE path MTU (pMTU) in the provider network.  
SB> I think you need to build a more complete case that the following because correctly implemented IPv6 will not send a frame that is too large because PMTUD and host fragmentation or transport fragmantation will prevent that.  IPv4 supports fragmentation.
SB> So that begs the question of what these packets are?

   This could be
   because

   1.  the provider network is built upon virtual connections (e.g.
       pseudowires) provided by another infrastructure provider, or

   2.  the customer network uses jumbo frames while the provider network
       does not, or

   3.  the provider-side overhead for transporting customers packets
       across the network pushes past the pMTU.

   In any case, the provider simply cannot require its customers to
   change their MTU.
SB> Don’t they change it automatically for most protocols?

   To get those large frames across the provider network, currently the
   only workaround is to encapsulate the frames in IP (with or without
   GRE) and then fragment the IP packets.  
SB> Again I think we need a bigger discussion of what those packets are.

   Even if MPLS is used for
   service delimiting, IP is used for transporation (MPLS over IP/GRE).
   This may not be desirable in certain deployment scenarios, where MPLS
   is the preferred transport or IP encapsulation overhead is deemed
   excessive.

SB> Again, I think we need more detail to justify this.

Zhang, et al.              Expires May 5, 2021                  [Page 2]

Internet-Draft         Generic Transport Functions         November 2020

   IPv6 fragmentation and reassembly are based on the IPv6 Fragmentation
   header below [RFC8200]:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Next Header  |   Reserved    |      Fragment Offset    |Res|M|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                    Figure 1: IPv6 Fragmentation Header

   This document proposes reusing this header in non-IP contexts, since
   the fragmentation/reassembly function is actually independent of IPv6
   except the following aspects:

   o  The fragment header is identified as such by the "previous"
      header.

   o  The "Next Header" value is from the "Internet Protocol Numbers"
      registry.

   o  The "Identification" value is unique in the (source, destination)
      context provided by the IPv6 header
SB> Which of course MPLS does not provide.

   The "Identification" field, in conjunction with the IPv6 source and
   destination identifies fragments of the original packet, for the
   purpose of reassembly.

   Therefore, the fragmentation/reassembly function can be applied at
   other layers as long as a) the fragment header is identified as such;
   and b) the context for packet identification is provided.  Examples
   of such layers include MPLS, BIER, and Ethernet (if IEEE determines
   it is so desired).
SB> Presumable we will liaise this to IEEE?

   For the layers where the IETF is concerned, the "Next Header" value
   will still be from the "Internet Protocol Numbers" registry when the
   function is applied at non-IP layers.
SB> IPv6 has next headers because Frag is an option and that is the way IPv6 works. It is unclear why a transport network fragmantation method as described here would need it.

   For the same consideration, the IP Encapsulating Security Payload
   (ESP) [RFC4303] could also be applied at other layers if ESP is
   desired there.  For example, if for whatever reason the Ethernet
   service provider wants to provide ESP between its PEs, it could do so
   without requiring IP encapsulation if ESP is applied at non-IP
   layers.
SB> I think we should get to the mechanics before discussing the options.

   The possibility of applying some other IP functions (e.g.
   Authentication Header [RFC4302]) is for further study.

Zhang, et al.              Expires May 5, 2021                  [Page 3]

Internet-Draft         Generic Transport Functions         November 2020

2.  Specifications

2.1.  Generic Fragmentation Header

   For generic fragmentation/reassembly functionality independent of IP,
   the following Generic Fragmentation Header (GFH) is defined:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Next Header  | Header Length |      Fragment Offset    |R|S|M|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   |                           (variable)                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                  Figure 2: Generic Fragmentation Header

   The "Next Header", "Fragment Offset" and "M" flag bit fields are as
   in the IPv6 Fragmentation Header.
SB> I am really not sure why a transport network frag needs to worry about NH. It should not care about what it is carrying. Its job is to simply glue the packet back together and let something that understands the glued together packet interpret it.
SB> IN any case NH could, at least in theory, alias IP and cause ECMP problems unless this sits behind a CW, which I don’t think we have discussed yet.

   Header Length:  the number of octets of the entire header.

   R: The "R" flag bit is reserved.  It MUST be 0 on transmitting and
      ignored on receiving.

   Identification:  at least 4-octet long.
SB> Four seems dangerously short given that IPv4 uses Eight. Also we need some discussion about how the ID is generated such that a misdelivered packet cannot get fragmented.
SB> BTW since later in your discussion you talk about multi-path delivery, you need to discuss reassembly timeout. That is not a problem with a PW since it is single path.

   S: If the "S" flag bit is clear, the context for the Identification
      field is provided by the outer header, and only the source-
      identifying information in the outer header is used.  If the "S"
      flag bit is set, the variable Identification field encodes both
      source-identifying information (e.g. the IP address of the node
      adding the GFH) and an identification number unique within that
      source.
SB> We need to be clearer that the ID must be unique per flow for the lifetime of any packet from that flow in the network, including “stuck” packets.

   The outer header MUST identify that a Generic Fragmentation Header
   follows and MAY carry source-identifying information.

   If the outer header is BIER, a TBD value for the "proto" field in the
   BIER header identifies that a GFH follows.  If the "S" flag bit is
   clear, the "BFIR-id" field in the BIER header provides the context
   for the "Identification" field.
SB> It is not yet clear why we need the protocol type at the MPLS layer.

   If the outer header is MPLS, the "S" flag bit MAY be clear if the the
   label preceeding the GFH identifies the sending BFR in addition to
   indicating that a GFH follows (see Section 2.2).

> On 10 Nov 2020, at 23:13, Jeffrey (Zhaohui) Zhang <[email-masked]> wrote:
> 
> Hi Andy, Stewart,
>  

It was certainly intended for use in P2P PWs, I am not sure anyone has thought about its application to EVPN/VPLS.

>  
> The reason is that the sequence number in the control word is specific to the PW and the fragmentation/reassembly is performed in the context of the PW.

That is correct.

> In case of EVPN/VPLS, an egress PE could receive fragments from different ingress PEs and reassembly must be done in the right context. 

We need to be quite clear here, you mean that it could be concurrently reassembling fragments from different ingress PEs to the same egress PE on the same EVPN label. Yes, that is a problem that needs to be addressed and there are a number of ways that it could be addressed. We need t think about whether the FO method is better than the BE method. 

Now the questions is where we start from scratch or provide that identity to RFC4623, and how we provide that identity. One method is SFL, another is to provide identity of the type that you describe as an extended CW. Whatever happens you are going to need a CW to desire that you defeat ECMP inspection of the payload in legacy routers.

>      "Preferred PW MPLS Control Word" [RFC4385 <https://tools.ietf.org/html/rfc4385>] SHOULD be used with the

So it has a CW.

What method does it use to do OAM, presumable GAL at BOS?

>  
>    - If a network uses entropy labels [RFC6790 <https://tools.ietf.org/html/rfc6790>], then the control word

That will only work if you have no legacy nodes on the path. A legacy node is entitled to ignore the EL and do DPI ECMP.

>      then the control word SHOULD NOT be used.

Given the f/b we got in PALS from the operator community about ECMP on PWs without CW, I wonder what is happening on these networks?

>  
> This draft-zzhang allows the context to be determined from the extended “identification” field or from the outer header.

So you have added Identification, and I agree that with unsolicited packets as opposed to the P2P PW case it is needed.

I am not sure about the rest of the design, and in particular I am not sure how you deal with reassembly lockup. That was a problem that could occur in the case of PWs because the reassembly buffer could be considered part of the PW which was something we provisioned. If we had a corruption in the network, then it would all work itself out. With an arbitrary fan in and no preprovisioning, I am not sure what the error behaviour will be. This is something that you certainly need to discuss in the text.

> In addition, it is “generic” such that it can be used for any situations where fragmentation is needed at any layer for any solution.

I am not convinced that the situations are real. I think there needs to be some more context on the deployments. Should be just fine.

BTW I note that this is a TSVWG draft. I am not sure it belongs there. I would have though that it belonged in the PALS/MPLS WGs or at least somewhere in RTG which is where EVPN was developed.

Hopefully the chairs have allowed enough time to discuss next Friday.

Best regards

Stewart

>  
> Thanks.
>  
> Jeffrey
>  
> From: Andrew G. Malis <[email-masked] <mailto:[email-masked]>> 
> To: Stewart Bryant <[email-masked] <mailto:[email-masked]>>
> Cc: [email-masked] <mailto:[email-masked]>; mpls <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> On Tue, Nov 10, 2020 at 9:57 AM Stewart Bryant <[email-masked] <mailto:[email-masked]>> wrote:
> Please can I draw the attention of the authors to https://tools.ietf.org/html/rfc4623 <https://urldefense.com/v3/__https:/tools.ietf.org/html/rfc4623__;!!NEt6yMaO-gk!WnDW7J-i0YOpcV5sF1KDAfZnAaxHY5z-pG0oiSXKdSXdg9o9pRoRlJeq1ENEdsSd$>
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/pals <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/pals__;!!NEt6yMaO-gk!WnDW7J-i0YOpcV5sF1KDAfZnAaxHY5z-pG0oiSXKdSXdg9o9pRoRlJeq1KRQDg8M$>

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 11 Nov 2020 13:11:15 +0000

Oh there is a deeper problem

Unless you guarantee that you have defeated ECMP, which I don’t think the design does since first nibble is not 0000, I think the different fragments can get ECMPed differently. If that happens then (unlike the PW case) you cannot be sure that the fragments will arrive on the same LC, and that means that you need to support cross line card reassembly. If that happens your performance falls through the floor.

Of course you could argue that it will mostly work, just like people argued that not having the CW would work well enough in PW, and it did mostly work, until it didn’t and we had a disorderly :) queue of operators at the door of PALS WG.

Now what happens in the VPN cases in the existing design is that just like PW you will be seeing occasional Ethernet reordering, which mostly does not matter because what is being carried is IP, but as we found out in PALS it sometimes does matter an it is very difficult for the operators to diagnose, and then it does matter. However what you have is potentially much worse because you will accumulate fragments on different line cards, or have a very complex implementation problem to reassemble at speed.

- Stewart

---

From: "Andrew G. Malis" <[email-masked]>
Date: Wed, 11 Nov 2020 08:17:24 -0500

Just to add a bit to Stewart's replies, I assume that you've read RFC 8469,
which was the result of the "disorderly queue of operators at the door of
PALS".

Also, regarding VPLS, VPLS is just a full mesh of P2P Ethernet PWs, and
while RFC 4762 is silent on the CW, in practice many (most?) VPLS
implementations use the CW.

Cheers,
Andy

On Wed, Nov 11, 2020 at 8:11 AM Stewart Bryant <[email-masked]>
wrote:

> Unless you guarantee that you have defeated ECMP, which I don’t think the
> design does since first nibble is not 0000, I think the different fragments
> can get ECMPed differently. If that happens then (unlike the PW case) you
> cannot be sure that the fragments will arrive on the same LC, and that
> means that you need to support cross line card reassembly. If that happens
> your performance falls through the floor.
>
> Of course you could argue that it will mostly work, just like people
> argued that not having the CW would work well enough in PW, and it did
> mostly work, until it didn’t and we had a disorderly :) queue of operators
> at the door of PALS WG.
>
> Now what happens in the VPN cases in the existing design is that just like
> PW you will be seeing occasional Ethernet reordering, which mostly does not
> matter because what is being carried is IP, but as we found out in PALS it
> sometimes does matter an it is very difficult for the operators to
> diagnose, and then it does matter. However what you have is potentially
> much worse because you will accumulate fragments on different line cards,
> or have a very complex implementation problem to reassemble at speed.
>
> - Stewart
>
>

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Wed, 11 Nov 2020 16:12:44 +0000

Hi Stewart,

Please see zzh> below.

From: Stewart Bryant <[email-masked]>
Sent: Wednesday, November 11, 2020 7:59 AM
To: Jeffrey (Zhaohui) Zhang <[email-masked]>
Cc: Stewart Bryant <[email-masked]>; Andrew G. Malis <[email-masked]>; [email-masked]; mpls <[email-masked]>; [email-masked]
Subject: Re: [Pals] Mail regarding draft-zzhang-tsvwg-generic-transport-functions

Zzh> Agreed. What prompted to this draft-zzhang is the EVPN use case and I added the PW scenario just because I was not aware of RFC4623.

Zzh> Understand.

Zzh> In the EVPN case, the IPv6 encapsulation (followed by fragmentation/reassembly) would be done by the PEs so the PEs are considered “hosts” for this purpose.

Zzh> I was told that there are customers who have to resort to adding IP/GRE encapsulation just to get around this problem. Either people are not aware of RFC4623 or they just resorted to IP/GRE encapsulation because RFC4623 may not be supported by vendors.

1.  Introduction

Zzh> As explained above, I can remove PWs (or explain RFC4623 relevance) and focus on EVPN. But if it is done for EVPN, then it can also be used for PWs (and any scenario).

Zzh> Please see bellow.

   This could be

   because

       does not, or

   change their MTU.

Zzh> If you’re an operator with smaller MTUs and your EVPN customer uses jumbo frames, you may not be able to ask the customer to move away from jumbo frames?

Zzh> To the EVPN operator, the packets would be ethernet. Even if the ethernet payload is IP, It may be impossible for the operator to ask its EVPN customer to use smaller MTU.

   Even if MPLS is used for

   excessive.

   header below [RFC8200]:

   |                         Identification                        |

      header.

      registry.

Zzh> There is actually a way (defined in this draft).

   purpose of reassembly.

   it is so desired).

Zzh> Not worried about that for now – if we get IETF work agreed upon first, we can casually check if they’re interested, but that’s not a priority at all.

Zzh> Consider that EVPN-MPLS use case: an outer label will identify a GFH follows; the GFH header’s “next header” will indicate MPLS because after reassembly the beginning of the reassembled payload is an MPLS label that identifies the EVPN bridge domain (similar to PW label).

   layers.

Zzh> The priority now is fragmentation. The paragraph just points out that ESP could be done similarly. The details are deferred, and it may be determined that ESP may not be done generically.

2.  Specifications

   |                         Identification                        |

   |                           (variable)                          |

Zzh> Perhaps “transport” is not the right word; but please see above for explanation why NH is needed.

      ignored on receiving.

Zzh> It is “at least” 4-octet long. The real length is derived from “Header Length”. If the outer header provides context, then 4-octet should be enough (just like in IPv6 case).

Zzh> It is really no different from IPv4/v6 reassembly – just pulled out to a shim layer.

      source.

Zzh> It is really no different from IPv4/v6 reassembly – just pulled out to a shim layer.

Zzh> Do you mean why the GFH may need to indicate NH is MPLS? That was explained earlier.

On 10 Nov 2020, at 23:13, Jeffrey (Zhaohui) Zhang <[email-masked]<mailto:[email-masked]>> wrote:

Hi Andy, Stewart,

Zzh> This email thread may conclude that it can’t be used for EVPN/VPLS 😊

That is correct.

Zzh> I thought the generic fragmentation/reassembly way in this draft will address that (just like how IP addresses it) – reassembly only considers <source, fragment identification> and the VPN demultiplexing comes *after* that.
Zzh> What does FO/BE mean here? Pardon my ignorance 😊

Zzh> The generic fragmentation/reassembly, whether applied to EVPN/VPLS or PW, is done independently – fragmentation after EVPN/VPLS/PW and reassembly before EVPN/VPLS/PW processing. Whether CW is used is orthogonal.

   - If a network uses deep packet inspection for its ECMP, then the
     "Preferred PW MPLS Control Word" [RFC4385<https://urldefense.com/v3/__https:/tools.ietf.org/html/rfc4385__;!!NEt6yMaO-gk!VuVUW3ykBENH4X3_cX68Wk4a2xzDLhqMBD4rmN4Dvo7mgs21RyTJGlPeSYeHfSrM$>] SHOULD be used with the
     value 0 (e.g., a 4-octet field with a value of zero) when sending
     EVPN-encapsulated packets over an MP2P LSP.

So it has a CW.

zzh> Apparently it only uses all-0 to prevent ECMP mishap. Nothing else.

   - If a network uses entropy labels [RFC6790<https://urldefense.com/v3/__https:/tools.ietf.org/html/rfc6790__;!!NEt6yMaO-gk!VuVUW3ykBENH4X3_cX68Wk4a2xzDLhqMBD4rmN4Dvo7mgs21RyTJGlPeSX0ugpyE$>], then the control word
     SHOULD NOT be used when sending EVPN-encapsulated packets over an
     MP2P LSP.

zzh> It is really the same as IP reassembly. An IP host doing reassembly could run into the same problems. Of course, a PE would be more challenged.

Zzh> As of now, fragmentation for EVPN is the most real use case; but the design is generic so that it can be used other scenarios if they come up.

Zzh> The use cases for now is in EVPN/MPLS/BIER but the mechanism probably belong to TSVWG, but I am fine with any WG that can take up the work. I did schedule presentation in BESS/MPLS/BIER but I missed PALS.
Zzh> Thanks!
Zzh> Jeffrey

Best regards

Stewart

Thanks.

Jeffrey

From: Andrew G. Malis <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, November 10, 2020 10:33 AM
To: Stewart Bryant <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; mpls <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Pals] Mail regarding draft-zzhang-tsvwg-generic-transport-functions

Cheers,
Andy

BR

Stewart

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Wed, 11 Nov 2020 16:32:32 +0000

Hi Andy, Stewart,

Please see zzh> below.

From: Andrew G. Malis <[email-masked]>
Sent: Wednesday, November 11, 2020 8:17 AM
To: Stewart Bryant <[email-masked]>
Cc: Jeffrey (Zhaohui) Zhang <[email-masked]>; [email-masked]; mpls <[email-masked]>; [email-masked]
Subject: Re: [Pals] Mail regarding draft-zzhang-tsvwg-generic-transport-functions

Just to add a bit to Stewart's replies, I assume that you've read RFC 8469, which was the result of the "disorderly queue of operators at the door of PALS".

Zzh> Actually I need to read on that yet.

Also, regarding VPLS, VPLS is just a full mesh of P2P Ethernet PWs, and while RFC 4762 is silent on the CW, in practice many (most?) VPLS implementations use the CW.

Zzh> OK if VPLS relies on data plane mac learning for packets received from the core then indeed they’re a full mesh of PWs and that’s different from EVPN. I’ll need to dig deeper into that. Still, if we solve the EVPN problem using the generic fragmentation method, the solution works for VPLS/PW as well.

Cheers,
Andy

On Wed, Nov 11, 2020 at 8:11 AM Stewart Bryant <[email-masked]<mailto:[email-masked]>> wrote:
Oh there is a deeper problem

Unless you guarantee that you have defeated ECMP, which I don’t think the design does since first nibble is not 0000, I think the different fragments can get ECMPed differently. If that happens then (unlike the PW case) you cannot be sure that the fragments will arrive on the same LC, and that means that you need to support cross line card reassembly. If that happens your performance falls through the floor.
Zzh> I see what you mean. Perhaps we can separate the one-octet “Next Header” from the existing IP “Next Header” space so that the first nibble is always 0000? That gives us 16 types of “Next Header” – or we only need to make sure the nibble is not 4 or 6 – that gives us more types.
Zzh> Thanks.
Zzh> Jeffrey

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 12 Nov 2020 12:16:18 +0000

In my view, the discussion on this draft demonstrate that it is not yet ready for detailed discussion by these working groups next week.

- Stewart

> On 11 Nov 2020, at 16:32, Jeffrey (Zhaohui) Zhang <[email-masked]> wrote:
> 
> Hi Andy, Stewart,
>  
> Please see zzh> below.
>  
> Cc: Jeffrey (Zhaohui) Zhang <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>; mpls <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> On Wed, Nov 11, 2020 at 8:11 AM Stewart Bryant <[email-masked] <mailto:[email-masked]>> wrote:

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Thu, 12 Nov 2020 21:37:30 +0000

Hi Stewart,

I hope you did not mean that the draft should not be presented. Whether detailed discussions can happen after the presentation in the session itself depends on many factors (we quite often run out of time in the sessions), and the presentation itself quite often is a good way to stir up wide discussions.

Thanks to your review and comment, I think we’re having good discussions on mailing lists on the draft. I hope I have answered your questions and addressed your main concerns, and I hope more people chime in, before and after the presentations in relevant WGs.

Thanks.
Jeffrey

From: Stewart Bryant <[email-masked]>
Sent: Thursday, November 12, 2020 7:16 AM
To: Jeffrey (Zhaohui) Zhang <[email-masked]>
Cc: Stewart Bryant <[email-masked]>; Andrew G. Malis <[email-masked]>; [email-masked]; mpls <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [Pals] Mail regarding draft-zzhang-tsvwg-generic-transport-functions

- Stewart

On 11 Nov 2020, at 16:32, Jeffrey (Zhaohui) Zhang <[email-masked]<mailto:[email-masked]>> wrote:

Hi Andy, Stewart,

Please see zzh> below.

Cc: Jeffrey (Zhaohui) Zhang <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; mpls <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>

Cheers,
Andy

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 16 Nov 2020 09:34:35 +0000

I think that it would be more productive to address the known issues first, particularly the issue of ECMP and the need to change the packet structure to make it ECMP safe, and then the WG discussion will be more focused. 

- Stewart

> On 12 Nov 2020, at 21:37, Jeffrey (Zhaohui) Zhang <[email-masked]> wrote:
> 
> Hi Stewart,
>  
> From: Stewart Bryant <[email-masked] <mailto:[email-masked]>> 
> To: Jeffrey (Zhaohui) Zhang <[email-masked] <mailto:[email-masked]>>
> Cc: Stewart Bryant <[email-masked] <mailto:[email-masked]>>; Andrew G. Malis <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>; mpls <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>
> On 11 Nov 2020, at 16:32, Jeffrey (Zhaohui) Zhang <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi Andy, Stewart,
>  
> Please see zzh> below.
>

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Mon, 16 Nov 2020 12:57:08 +0000

Hi Stewart,

In my previous email, I acknowledged that ECMP issue you brought up and proposed a solution.
In fact, we can make the GFH to start with nibble 0000.

Thanks.
Jeffrey

Sent: Monday, November 16, 2020 4:35 AM

- Stewart

On 12 Nov 2020, at 21:37, Jeffrey (Zhaohui) Zhang <[email-masked]<mailto:[email-masked]>> wrote:

Hi Stewart,

Thanks.
Jeffrey

From: Stewart Bryant <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, November 12, 2020 7:16 AM
To: Jeffrey (Zhaohui) Zhang <[email-masked]<mailto:[email-masked]>>
Cc: Stewart Bryant <[email-masked]<mailto:[email-masked]>>; Andrew G. Malis <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; mpls <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Pals] Mail regarding draft-zzhang-tsvwg-generic-transport-functions

- Stewart

Hi Andy, Stewart,

Please see zzh> below.

Cheers,
Andy

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 19 Nov 2020 06:21:01 +0000

> Zzh> I see what you mean. Perhaps we can separate the one-octet “Next Header” from the existing IP “Next Header” space so that the first nibble is always 0000? That gives us 16 types of “Next Heade…

You should use 0000.

This is the IP version number space. 4 and 6 are taken as IP. BIER sits on 3, PWs sit on 0 and 1 (for OAM), and of course there is a chance that INT might like invent another IP at some time in the future (a general note and nothing to do with NewIP).

Now what I am not sure about is why you need a next header as this is not needed elsewhere in MPLS PWs. If you do, then I would suggest that 16 is likely too few.

The standard MPLS method is to trigger processing from the label, so if you want to do other processing then why not allocate a new label so you can vector straight to the right packet handler.

What I think you need is the existing PW frag design with the identification field appended. Then to advertise those propertied with the label.

- Stewart

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Thu, 19 Nov 2020 17:21:24 +0000

Stewart,

Please see zzh2> below.

Sent: Thursday, November 19, 2020 1:21 AM

You should use 0000.

Zzh> Agreed.

Now what I am not sure about is why you need a next header as this is not needed elsewhere in MPLS PWs.

Zzh> PW fragmentation is in the context of PWs, and the egress PE does not need to care what the payload it is.
Zzh> With *generic* fragmentation, the egress PE does need to care what the reassembled payload is to continue processing, that’s why it needs a “next header” field.

If you do, then I would suggest that 16 is likely too few.

Zzh> Because of that, what was presented in BIER/BESS is that the “Hdr Len” field, not the “Next Header” field gets squeezed. The Next Header field is still 8-bit. The “Hdr Len” is 4-bit but in the unit of 8-octets, so it should be enough (and we could make it 16-octet units if necessary).

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0 0 0 0|  Next Header  |Hdr Len|      Fragment Offset    |R|S|M|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Identification                        |
   |                           (variable)                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Zzh> There is a GFH label in front of the GFH, and that tells the packet handler that a GFH follows (and optionally tell who did the fragmentation – the reassembling node can advertise different GFH labels for different fragmenting nodes).
Zzh> It’s what happens after the reassembly that requires the “Next Header” in the GFH. Consider the EVPN-MPLS example, the fragments received by the egress PE have the following headers: <label stack1, GFH, label stack2>, where the label stack1 is <transport labels, GFH label> and the label stack2 is <EVPN related labels>. In potential other use cases (we’re talking about “generic” fragmentation), the payload after the GFH may be other types.

Zzh> The intention is to have a generic solution, not tied to EVPN-MPLS or PW. You can see that while the immediate use case is EVPN-MPLS, the solution is not tied to it. *In theory*, this can be used for fragmenting at Ethernet level between two Ethernet nodes.
Zzh> Thanks!
Zzh> Jeffrey

- Stewart