Subject: [mpls] comments for MPLS-RT review of draft-akiya-mpls-lsp-ping-lag-multipath

From: Yimin Shen <[email-masked]>
Date: Sat, 13 Dec 2014 05:07:30 +0000

Hi authors,

I was asked by the MPLS WG chairs to review this draft. Thanks Nobo for clarifying some points for me during the review. Here are my primary comments. 

This draft is trying to address multi-path discovery and verification for LSPs over LAGs, which is a valid but challenging use case with a long history. From this perspective, I think the draft shows a good effort.

The mechanism tries to extend LSP traceroute, which is a layer-3 OAM tool, to explore and exercise layer-2 multi-paths. It may be debatable whether this is a layer violation, but I'd like to leave it to the WG. 

Nevertheless, I can see some challenges that the mechanism faces. These challenges are essentially related to the lack of layer-2 visibility on layer-3 routers. Specifically, the draft uses an interface index to indentify each LAG member link, which is a layer-2 link. However, the two routers at the endpoints of a LAG normally do not have the info of the LAG member link indices of each other. This results in some limitations in multi-path verification:

[1] It doesn't support incoming LAG member link verification for Echo Request on the downstream router of a LAG. IOW, the ingress router may set an Echo Request's destination to a 127/8 prefix according to the multi-path info of a particular LAG member link, but the downstream router has no way to verify that the upstream router indeed sent the Echo Request over the claimed LAG member link. This is due to the lack of local and remote LAG member link association, and the lack of upstream LAG member link identification in the Echo Request. Without incoming LAG member link verification, this traceroute would not provide more useful information than traditional LSP tracerotue over LAG as a single layer-3 interface.

[2] For the above reason, it does not support verification for individual end-2-end paths, which is a desired mode in troubleshooting and pinpointing a problem path.

[3] For the same reason, the draft has to require the ingress router to always verify all paths. In particular, the ingress needs to match the count of Echo Request sent and the count of Echo Reply received, and verify the 1-1 relationship between upstream member links and downstream member links. This may give the users some idea about the forwarding behavior on the LAG, but not the ability to detect all failure scenarios. 

Therefore, the current draft does not seem to be sufficient or reliable enough. I'd suggest the authors to address the above issue before WG adoption.

Thanks and regards,

--
Yimin Shen
Juniper Networks

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Sun, 14 Dec 2014 14:48:03 +0000

Hi Yimin,

Thank you for taking time to review this document and providing comments.

Please see in-line for [NOBO].

> -----Original Message-----
> From: Yimin Shen [mailto:[email-masked]]
> Sent: Saturday, December 13, 2014 12:08 AM
> To: [email-masked]; Nobo Akiya
> (nobo); Tarek Saad (tsaad)
> Cc: [email-masked]; [email-masked]
> Subject: comments for MPLS-RT review of draft-akiya-mpls-lsp-ping-lag-
> multipath
> 
> Hi authors,
> 
> I was asked by the MPLS WG chairs to review this draft. Thanks Nobo for
> clarifying some points for me during the review. Here are my primary
> comments.
> 
> This draft is trying to address multi-path discovery and verification for LSPs
> over LAGs, which is a valid but challenging use case with a long history. From
> this perspective, I think the draft shows a good effort.
> 
> The mechanism tries to extend LSP traceroute, which is a layer-3 OAM tool,
> to explore and exercise layer-2 multi-paths. It may be debatable whether
> this is a layer violation, but I'd like to leave it to the WG.

[NOBO] I think that's a fair statement. Although I'd like to point out that, without "some extensions" to the LSP Ping/Trace or other tools, there is no interoperable way to identify some faults of LSP over LAG today, that's a really bad thing.

> 
> Nevertheless, I can see some challenges that the mechanism faces. These
> challenges are essentially related to the lack of layer-2 visibility on layer-3
> routers. Specifically, the draft uses an interface index to indentify each LAG
> member link, which is a layer-2 link. However, the two routers at the
> endpoints of a LAG normally do not have the info of the LAG member link
> indices of each other. This results in some limitations in multi-path
> verification:

[NOBO] Yes that's a correct statement, and if we were to create a mechanism such that the two L3 endpoints across a LAG (or LAGs) has visibility of each other, that's a rather a large problem requiring much wider extensions, possibility beyond the LSP Ping/Trace tools.

> 
> [1] It doesn't support incoming LAG member link verification for Echo
> Request on the downstream router of a LAG. IOW, the ingress router may
> set an Echo Request's destination to a 127/8 prefix according to the multi-
> path info of a particular LAG member link, but the downstream router has
> no way to verify that the upstream router indeed sent the Echo Request
> over the claimed LAG member link. This is due to the lack of local and
> remote LAG member link association, and the lack of upstream LAG member
> link identification in the Echo Request. Without incoming LAG member link
> verification, this traceroute would not provide more useful information
> than traditional LSP tracerotue over LAG as a single layer-3 interface.
> 
> [2] For the above reason, it does not support verification for individual end-
> 2-end paths, which is a desired mode in troubleshooting and pinpointing a
> problem path.
> 
> [3] For the same reason, the draft has to require the ingress router to always
> verify all paths. In particular, the ingress needs to match the count of Echo
> Request sent and the count of Echo Reply received, and verify the 1-1
> relationship between upstream member links and downstream member
> links. This may give the users some idea about the forwarding behavior on
> the LAG, but not the ability to detect all failure scenarios.

[NOBO] The limitation that you identified in [1] is valid, and the approach described in [3] does indeed require more MPLS echo request/reply packets. However, what this mechanism and the approach described in [3] will provide is a guarantee that the initiator LSR is able to exercise all LAG members for a given LSP. And that provides a way to discover whether or not the traffic over LSP is black holing or not on this LAG. Yes there is still a corner case of faulty FIB that causes some LAG members to criss-cross and land up on unexpected LAG members on the other side, but that's the only case and that's not a big problem when looking at "anti-black-hole" extension.
 
> 
> Therefore, the current draft does not seem to be sufficient or reliable
> enough. I'd suggest the authors to address the above issue before WG
> adoption.

[NOBO] I tend to think this mechanism strikes a good balance between the coverage and the simplicity, and is well worth pursuing. I'd rather invest time in implementing this, rather than get "traffic is black holing!!" customer cases, and I certainly hope that operators feel the same way.

[NOBO] Through our off-list email exchanges, you provided one comment that requires a document change. Which is, what should the responder LSR do when received multipath information does not map to a LAG member link. To address this, I have added following texts below Figure 2.

   When none of the received multipath information maps to a particular
   LAG member link, then the responder LSR MUST still place the
   Interface Index Sub-TLV for that LAG member link in the DDMAP but
   inclusion of the Multipath Data Sub-TLV for that LAG member link is
   OPTIONAL.  If the Multipath Data Sub-TLV is included, then the
   Multipath Length field of the Multipath Data Sub-TLV MUST be zero.

Again thanks for your time and effort to review this document!

-Nobo

---

From: Yimin Shen <[email-masked]>
Date: Tue, 16 Dec 2014 13:35:27 +0000

Nobo, authors,

I can support the document for WG adoption. However I believe it can significantly improve if "incoming LAG member link verification" and "individual end-to-end path verification" are supported. Therefore, I'd like to make a couple of suggestions.

1. Add an "Applicability" section to describe the current limitations.
- The general lack of layer-2 visibility in a layer-3 network, especially for LAG member indices.
- No incoming LAG member link verification at downstream router.
- No individual end-to-end path verification.
- Failure scenarios that may not be detected.
- Your strategy to balance coverage and simplicity that you mentioned below. 

2. For completeness and future compatibility, include/describe the procedures of "incoming LAG member link verification" and "individual end-to-end path verification", assuming the LAG member index visibility issue is resolved.
- This visibility could be achieved through some protocol, or learning from a central controller.
- For this reason, the document may need to allow "Interface Index Sub-TLV of LAG member link" learned from Echo Reply to be reflected in the next Echo Request. 

Thanks,

/Yimin

-----Original Message-----
From: Nobo Akiya (nobo) [mailto:[email-masked]] 
Sent: Sunday, December 14, 2014 9:48 AM
To: Yimin Shen; [email-masked]; Tarek Saad (tsaad)
Cc: [email-masked]; [email-masked]
Subject: RE: comments for MPLS-RT review of draft-akiya-mpls-lsp-ping-lag-multipath

Hi Yimin,

> I was asked by the MPLS WG chairs to review this draft. Thanks Nobo 
> for clarifying some points for me during the review. Here are my 
> primary comments.
> 
> This draft is trying to address multi-path discovery and verification 
> for LSPs over LAGs, which is a valid but challenging use case with a 
> long history. From this perspective, I think the draft shows a good effort.
> 
> The mechanism tries to extend LSP traceroute, which is a layer-3 OAM 
> tool, to explore and exercise layer-2 multi-paths. It may be debatable 
> whether this is a layer violation, but I'd like to leave it to the WG.

> 
> Nevertheless, I can see some challenges that the mechanism faces. 
> These challenges are essentially related to the lack of layer-2 
> visibility on layer-3 routers. Specifically, the draft uses an 
> interface index to indentify each LAG member link, which is a layer-2 
> link. However, the two routers at the endpoints of a LAG normally do 
> not have the info of the LAG member link indices of each other. This 
> results in some limitations in multi-path
> verification:

> set an Echo Request's destination to a 127/8 prefix according to the 
> multi- path info of a particular LAG member link, but the downstream 
> router has no way to verify that the upstream router indeed sent the 
> Echo Request over the claimed LAG member link. This is due to the lack 
> of local and remote LAG member link association, and the lack of 
> upstream LAG member link identification in the Echo Request. Without 
> incoming LAG member link verification, this traceroute would not 
> provide more useful information than traditional LSP tracerotue over LAG as a single layer-3 interface.
> 
> [2] For the above reason, it does not support verification for 
> individual end- 2-end paths, which is a desired mode in 
> troubleshooting and pinpointing a problem path.
> 
> [3] For the same reason, the draft has to require the ingress router 
> to always verify all paths. In particular, the ingress needs to match 
> the count of Echo Request sent and the count of Echo Reply received, 
> and verify the 1-1 relationship between upstream member links and 
> downstream member links. This may give the users some idea about the 
> forwarding behavior on the LAG, but not the ability to detect all failure scenarios.

> 
> Therefore, the current draft does not seem to be sufficient or 
> reliable enough. I'd suggest the authors to address the above issue 
> before WG adoption.

-Nobo

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Wed, 17 Dec 2014 02:17:30 +0000

Hi Yimin,

My apologies for top posting. I wanted to respond to your (2) first and then discuss (1), and found it easier with top posting.

Sure, what I can do is to update the document so that both "incoming LAG member link verification" and "individual end- to-end path verification" are supported, but defines "how an LSR obtains interface index from remote LSRs" to be outside the scope of the document.

Specifically:

- Extend the Interface Index Sub-TLV to have (L)ocal and (R)emote flags.
- For a LAG member, the DDMAP MUST contain the Interface Index Sub-TLV with the (L)ocal flag.
- For a LAG member, the DDMAP MAY contain the Interface Index Sub-TLV with the (R)emote flag.
- State that how an LSR obtains the Interface Index of the incoming interfaces of the downstream LSR is outside the scope of this document.
- Update the responder LSR procedures of DDMAP handling.
- Update the initiator LSR procedures of DDMAP handling.
- Break up the validation section into 2 parts: One for "incoming LAG member link verification" and one for "individual end- to-end path verification".

Note, with all of these changes (i.e., including procedures for future usage when more inter-node information is available), all aspects of your suggested topics for the new Applicability section are already covered or are outside the scope of this document. Given that, do you see it as a MUST to add the new Applicability section?

Thanks!

-Nobo

> Sent: Tuesday, December 16, 2014 8:35 AM
> To: Nobo Akiya (nobo); draft-akiya-mpls-lsp-ping-lag-
> [email-masked]; Tarek Saad (tsaad)
> I can support the document for WG adoption. However I believe it can
> significantly improve if "incoming LAG member link verification" and
> "individual end-to-end path verification" are supported. Therefore, I'd like
> to make a couple of suggestions.
> 
> 2. For completeness and future compatibility, include/describe the
> procedures of "incoming LAG member link verification" and "individual end-
> to-end path verification", assuming the LAG member index visibility issue is
> resolved.
> - For this reason, the document may need to allow "Interface Index Sub-TLV
> of LAG member link" learned from Echo Reply to be reflected in the next
> Echo Request.
> 
> 
> Thanks,
> 
> /Yimin
> 
> 
> 
> [NOBO] I think that's a fair statement. Although I'd like to point out that,
> without "some extensions" to the LSP Ping/Trace or other tools, there is no
> interoperable way to identify some faults of LSP over LAG today, that's a
> really bad thing.
> 
> >
> [NOBO] Yes that's a correct statement, and if we were to create a
> mechanism such that the two L3 endpoints across a LAG (or LAGs) has
> visibility of each other, that's a rather a large problem requiring much wider
> extensions, possibility beyond the LSP Ping/Trace tools.
> 
> >
> [NOBO] The limitation that you identified in [1] is valid, and the approach
> described in [3] does indeed require more MPLS echo request/reply packets.
> However, what this mechanism and the approach described in [3] will
> provide is a guarantee that the initiator LSR is able to exercise all LAG
> members for a given LSP. And that provides a way to discover whether or
> not the traffic over LSP is black holing or not on this LAG. Yes there is still a
> corner case of faulty FIB that causes some LAG members to criss-cross and
> land up on unexpected LAG members on the other side, but that's the only
> case and that's not a big problem when looking at "anti-black-hole"
> extension.
> 
> >
> [NOBO] I tend to think this mechanism strikes a good balance between the
> coverage and the simplicity, and is well worth pursuing. I'd rather invest
> time in implementing this, rather than get "traffic is black holing!!"
> customer cases, and I certainly hope that operators feel the same way.
> 
> [NOBO] Through our off-list email exchanges, you provided one comment
> that requires a document change. Which is, what should the responder LSR
> do when received multipath information does not map to a LAG member
> link. To address this, I have added following texts below Figure 2.
>

---

From: Yimin Shen <[email-masked]>
Date: Wed, 17 Dec 2014 13:10:14 +0000

Hi Nobo,

This approach sounds fine to me. Thanks for the flexibility.

Please see inline...

Thanks,

/Yimin

-----Original Message-----
From: Nobo Akiya (nobo) [mailto:[email-masked]] 
Sent: Tuesday, December 16, 2014 9:18 PM
To: Yimin Shen; [email-masked]; Tarek Saad (tsaad)
Cc: [email-masked]; [email-masked]
Subject: RE: comments for MPLS-RT review of draft-akiya-mpls-lsp-ping-lag-multipath

Hi Yimin,

[yshen] Treating how to get interface index as out of scope is fine.

Specifically:

[yshen] Given what you've outlined above, I agree that an Applicability section may not be needed. 

Thanks!

-Nobo

> Subject: RE: comments for MPLS-RT review of 
> draft-akiya-mpls-lsp-ping-lag- multipath
> 
> Nobo, authors,
> 
> "individual end-to-end path verification" are supported. Therefore, 
> I'd like to make a couple of suggestions.
> 
> procedures of "incoming LAG member link verification" and "individual 
> end- to-end path verification", assuming the LAG member index 
> visibility issue is resolved.
> - For this reason, the document may need to allow "Interface Index 
> Sub-TLV of LAG member link" learned from Echo Reply to be reflected in 
> the next Echo Request.
> 
> 
> Thanks,
> 
> /Yimin
> 
> 
> 
> To: Yimin Shen; 
> [email-masked];
> Tarek Saad (tsaad)
> Subject: RE: comments for MPLS-RT review of 
> draft-akiya-mpls-lsp-ping-lag- multipath
> 
> Hi Yimin,
> 
> > To: [email-masked]; Nobo 
> > Akiya (nobo); Tarek Saad (tsaad)
> > Subject: comments for MPLS-RT review of 
> > draft-akiya-mpls-lsp-ping-lag- multipath
> >
> > Hi authors,
> >
> > This draft is trying to address multi-path discovery and 
> > verification for LSPs over LAGs, which is a valid but challenging 
> > use case with a long history. From this perspective, I think the draft shows a good effort.
> >
> > tool, to explore and exercise layer-2 multi-paths. It may be 
> > debatable whether this is a layer violation, but I'd like to leave it to the WG.
> 
> [NOBO] I think that's a fair statement. Although I'd like to point out 
> that, without "some extensions" to the LSP Ping/Trace or other tools, 
> there is no interoperable way to identify some faults of LSP over LAG 
> today, that's a really bad thing.
> 
> >
> > interface index to indentify each LAG member link, which is a 
> > layer-2 link. However, the two routers at the endpoints of a LAG 
> > normally do not have the info of the LAG member link indices of each 
> > other. This results in some limitations in multi-path
> > verification:
> 
> visibility of each other, that's a rather a large problem requiring 
> much wider extensions, possibility beyond the LSP Ping/Trace tools.
> 
> >
> > [1] It doesn't support incoming LAG member link verification for 
> > Echo Request on the downstream router of a LAG. IOW, the ingress 
> > router may set an Echo Request's destination to a 127/8 prefix 
> > according to the
> > Echo Request over the claimed LAG member link. This is due to the 
> > lack of local and remote LAG member link association, and the lack 
> > of upstream LAG member link identification in the Echo Request. 
> > Without incoming LAG member link verification, this traceroute would 
> > not provide more useful information than traditional LSP tracerotue 
> > over LAG
> as a single layer-3 interface.
> >
> > to always verify all paths. In particular, the ingress needs to 
> > match the count of Echo Request sent and the count of Echo Reply 
> > received, and verify the 1-1 relationship between upstream member 
> > links and downstream member links. This may give the users some idea 
> > about the forwarding behavior on the LAG, but not the ability to 
> > detect all failure
> scenarios.
> 
> [NOBO] The limitation that you identified in [1] is valid, and the 
> approach described in [3] does indeed require more MPLS echo request/reply packets.
> provide is a guarantee that the initiator LSR is able to exercise all 
> LAG members for a given LSP. And that provides a way to discover 
> whether or not the traffic over LSP is black holing or not on this 
> LAG. Yes there is still a corner case of faulty FIB that causes some 
> LAG members to criss-cross and land up on unexpected LAG members on 
> the other side, but that's the only case and that's not a big problem when looking at "anti-black-hole"
> extension.
> 
> >
> [NOBO] I tend to think this mechanism strikes a good balance between 
> the coverage and the simplicity, and is well worth pursuing. I'd 
> rather invest time in implementing this, rather than get "traffic is black holing!!"
> that requires a document change. Which is, what should the responder 
> LSR do when received multipath information does not map to a LAG 
> member link. To address this, I have added following texts below Figure 2.
>