Subject: Re: [mpls] MPLS-RT review of draft-villamizar-mpls-forwarding

From: <[email-masked]>
Date: Tue, 5 Mar 2013 14:34:13 +0100

Hello authors and WG leads,

I have been asked to review draft-villamizar-mpls-forwarding.

I have read the draft and find it very useful. Our own experience while testing and deploying various MPLS equipment have shown that there is a need for a written best practices document on MPLS forwarding.
I support the adoption as WG draft. The document sounds technically.

Some notes:

In general:
- the draft defines some topics out of scope in the text (e.g. VPN encapsulation in section 2.1.9). But it does not explicit define the scope. Only in the first sentence of section 1, it focusses the draft on "forwarding". An explicit "scope" definition would be helpful.
- Is there a reason to use capital letters for PUSH, POP and SWAP label operations?
- I like section 2.6.4, because it uses clear requirements language ("MUST/SHOULD be supported in hardware..."). In some other sections of the draft, I miss these clear requirements language.
- In some parts, 10 Gbit/s and 100 Gbit/s are mentioned to express high bandwidth/performance requirements. Shouldn't it be generalized or do we have to update the RFC when we talk about 1 Tbit/s, next year?
- While start reading, I thought that the draft is a good tutorial about MPLS forwarding. But the style changes. In some chapters, the text uses a tutorial style and includes a general discussion about the mentioned concepts (e.g. section 2.6.1 about DoS protection). In other sections, the authors expect that the reader is aware about the features and the implementation (e.g. section 2.1.1 about reserved labels).

Section 1.2, 2nd bullet, 3rd sentence: delete "the" in "If a the bottom..."

Section 2.3, 1st sentence: what is "large"?

Section 2.3, paragraph 6, 2nd sentence: replace "are" with "at" ("...minimum size packets are full line rate...")?

Section 2.4:
usage of "Entropy Label" vs. "entropy label"

Section 2.4.1:
I do not understand the last paragraph completely. Why do you tolerate a missing control word support in intra-metro (where maybe multiple parallel 1 GbE links are used?) and low bandwidth environment. My understanding of last sentence: For general applicability of MPLS equipment, the support of control word is required.

Section 2.4.5: replace "...used, the there are a limited..." by "...used, there is a limited..."

Section 2.4.5.2
Bullet 4, 3rd sentence: replace "af" by "of".
Bullet 6: DSCP and TTL/hop count could be separated in two bullets, because there a different reasons to not consider these fields.

What is the difference between the bullets and the last paragraph in this section (e.g. the SHOULD requirements of IP addresses in the bullets and the "recommended MUST" in the last paragraph)? Does it makes sense to avoid the capitalized letters in this last paragraph to avoid conflict statements in this section?

Section 2.6.1:
I am not sure to understand the reason for this section in this I-D. The draft focusses on forwarding and the required capabilities. This section describes a need of DoS protection mechanism in general. This has impact on forwarding hardware (forwarding implementations have to provide IP filtering capabilities and cryptographic hardware support to offload and distribute DoS mitigation from the CPU). In the others section, the need of a requirement on forwarding level is not discussed in general (e.g. there is no discussion about general OAM deployments). The OAM capabilities on forwarding level are defined). There are no requirements of forwarding level in this section (e.g. filtering, dampening mechanism, cryptographic capabilities, ...). Only a single reference in following section (2.6.2) to this section.

Penultimate paragraph, last sentence: I do not know the word "vigalence".  You maybe mean "vigilance"

Section 2.6.2, last paragraph, 3rd sentence: closing brace is missing.

Section 2.6.4
text style: the abbreviation CC-CV is not described, but RDI (Remote Defect Indication) is.

Section 3:
Q#34 and Q#35 style: in other parts of section 3, the different conformance questions where listed as separate question items. But Q#34 and Q#35 contains multiple questions.

Section 4:
T#23: replace "EI" with "EL"

Style: in section 4, DoS and OAM have different "headlines" ("DoS protection" and "OAM Capabilities and Performance") whereas in section 3, DoS and OAM are under single headline ("OAM and DoS Protection").

Sorry for the required extended review time.

Regards
Thomas

> -----Original Message-----
> From: Loa Andersson [mailto:[email-masked]]
> Sent: Wednesday, February 13, 2013 9:35 AM
> To: Eric Osborne (eosborne); Gregory Mirsky; Thomas Nadeau;
> Beckhaus, Thomas
> Cc: [email-masked];
> [email-masked]; Martin Vigoureux
> Subject: MPLS-RT review of draft-villamizar-mpls-forwarding
>
> Eric, Greg, Tom and Thomas,
>
> You have been selected as an MPLS Review team reviewers for
> draft-villamizar-mpls-forwarding-01.txt.
>
> Note to authors: You have been CC'd on this email so that you can know
> that this review is going on. However, please do not review your own
> document.
>
> Reviews should comment on whether the document is coherent, is it
> useful (ie, is it likely to be actually useful in operational
> networks), and is the document technically sound?  We are interested
> in knowing whether the document is ready to be considered for WG
> adoption (ie, it doesn't have to be perfect at this point,
> but should be
> a good start).
>
> Reviews should be sent to the document authors, WG co-chairs and
> secretary, and CC'd to the MPLS WG email list. If necessary, comments
> may be sent privately to only the WG chairs.
>
> Are you able to review this draft by February 27, 2013?
> I'd had a problem picking a good date, if you have problems meeting
> February 27, please suggest another date.
>
>
>
> Thanks, Loa
> (as MPLS WG chair)
> --
>
>
> Loa Andersson                        email: [email-masked]
> MPLS Expert                                 [email-masked]
> Huawei Technologies (consult)        phone: +46 739 81 21 64
>

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 06 Mar 2013 14:02:41 -0500

In message <[email-masked]>
[email-masked] writes:
> I have read the draft and find it very useful. Our own experience
> while testing and deploying various MPLS equipment have shown that
> there is a need for a written best practices document on MPLS
> forwarding.  I support the adoption as WG draft. The document sounds
> technically.

Thomas,

Thank you for the review.

Responses to your notes are inline below.

> Some notes:
>  
> In general:
>  
> - the draft defines some topics out of scope in the text (e.g. VPN
>   encapsulation in section 2.1.9). But it does not explicit define the
>   scope. Only in the first sentence of section 1, it focusses the
>   draft on "forwarding". An explicit "scope" definition would be
>   helpful.

This is a good point.  I looked for the word scope and found only
statements limiting scope, excluding:

  1.  PW encapsulation (ie: AC, also hashing on NSP fields for EL)
  2.  VPN (another PW application)
  3.  active management of hash space in multipath
  4.  hashing on tunneling protocols within IP encapsulations

A statement up front saying what is in scope would help.  The
following could be added to the introduction.

   The focus of this document is MPLS forwarding, base pseudowire
   forwarding, and MPLS OAM.  The use of pseudowire control word, and
   sequence number are discussed.  Specific pseudowire AC and NSP are
   out of scope.  Specific pseudowire applications, such as various
   forms of VPN, are out of scope.

   MPLS support for multipath techniques is considered essential by
   many service providers and is useful for other high capacity
   networks.  In order to obtain sufficient entropy from MPLS traffic
   service providers and others find it essential for the MPLS
   implementation to interpret the MPLS payload as IPv4 or IPv6 based
   on the contents of the first nibble of payload.  The use of IP
   addresses, the IP protocol field, and UDP and TCP port number
   fields in multipath load balancing are considered within scope.
   The use of any other IP protocol fields, such as tunneling
   protocols carried within IP, are out of scope.

   Implementation details are a local matter and are out of scope.

> - Is there a reason to use capital letters for PUSH, POP and SWAP

Personal preference.  They could be lower case.

> - I like section 2.6.4, because it uses clear requirements language
>   ("MUST/SHOULD be supported in hardware..."). In some other sections
>   of the draft, I miss these clear requirements language.

It is only OAM where there is an option to do some things outside of
the "fast path".  All forwarding MUST be done in the forwarding
hardware.

BTW- It has been suggested that "fast path" be defined and used rather
than "in forwarding hardware".

> - In some parts, 10 Gbit/s and 100 Gbit/s are mentioned to express
>   high bandwidth/performance requirements. Shouldn't it be generalized
>   or do we have to update the RFC when we talk about 1 Tbit/s, next
>   year?

The examples are provided to give a sense of the PPS rates (150 MPPS
for Ethernet, 250 MPPS for GFP) and small packet burst buffering
needed.  For 1 Tb/s interfaces a simple multiplication has to be
applied.  There would be no need to update the document.

> - While start reading, I thought that the draft is a good tutorial
>   about MPLS forwarding. But the style changes. In some chapters, the
>   text uses a tutorial style and includes a general discussion about
>   the mentioned concepts (e.g. section 2.6.1 about DoS protection). In
>   other sections, the authors expect that the reader is aware about
>   the features and the implementation (e.g. section 2.1.1 about
>   reserved labels).

Apparently the DoS protection is not obvious to some and there is no
document that I am aware of to point readers toward.  The reason to
beleive it is not sufficiently obvious is the number of
implementations in the past which have overlooked this requirement.
Even today, there seem to be some within the KARP WG that think that
cryptographic techniques solve everything.  AFAIK there is no standard
defining the IP filtering needed in service provider LSR, therefore
some discussion of the requirements is about all that can be provided.

The document is long, so if there is any unnecessary tutorial where we
could point the reader to an existing document, it would be worth it
to point that out.

> Section 1.2, 2nd bullet, 3rd sentence: delete "the" in "If a the

OK

> Section 2.3, 1st sentence: what is "large"?

In this sentence large refers to average packet size.  Measurements in
the 1990s were as low as 120-160 bytes but grew as 552 MTU (dial up
using PPP) disappeared.  Most people assume 400 bytes or larger today
for Internet traffic within a service provider core but point out that
it there is a lot of some types of traffic, for example RTP associated
with VOIP, then average packet size can be much lower.

The point of just using the word "large" was to imply, "a lot bigger
than the 40 byte packets commonly observed in large bursts of small
packets on the Internet".  To expand on "large" in the document would
be pointless and add bloat.

> Section 2.3, paragraph 6, 2nd sentence: replace "are" with "at"
> ("...minimum size packets are full line rate...")?

OK

> usage of "Entropy Label" vs. "entropy label"

OK.  I will down case except in section titles.

> Section 2.4.1:
> I do not understand the last paragraph completely. Why do you tolerate
> a missing control word support in intra-metro (where maybe multiple
> parallel 1 GbE links are used?) and low bandwidth environment. My
> understanding of last sentence: For general applicability of MPLS
> equipment, the support of control word is required.

If a control word is missing from an intra-metro PW and the metro does
not use multipath, then no harm is done.  As soon as the PW goes
inter-metro and crosses infrastructure that uses multipath, it stops
working.  Some equipment vendors that focus on metro products seem to
have missed this.

> Section 2.4.5:
> replace "...used, the there are a limited..." by "...used, there is a
> limited..."

OK.  "A set" is singular.

> Bullet 4, 3rd sentence: replace "af" by "of".

OK.

> Bullet 6: DSCP and TTL/hop count could be separated in two bullets,
> because there a different reasons to not consider these fields.

This bullet lists the IP header fields to avoid using.  I could make
sub-bullets.  The reasoning is the same as for the TC and TTL label
entry fields, and IP has no bottom of stack bit.

> What is the difference between the bullets and the last paragraph in
> this section (e.g. the SHOULD requirements of IP addresses in the
> bullets and the "recommended MUST" in the last paragraph)? Does it
> makes sense to avoid the capitalized letters in this last paragraph to
> avoid conflict statements in this section?

The bullets indicate compliance to RFCs.  The paragraph tries to point
out strong market realities that implementors may ignore based on
strict reading of the RFCs alone, but "may be doing so at their own
peril" if service provider requirements are considered.

You may note that some co-authors are from service providers and
others have been involved in LSR used by service providers.  All felt
a need to emphasize that while in the RFCs some functionality
discussed in this section is optional, in the real world it is
manditory.

> Section 2.6.1:
> I am not sure to understand the reason for this section in this
> I-D. The draft focusses on forwarding and the required
> capabilities. This section describes a need of DoS protection
> mechanism in general. This has impact on forwarding hardware
> (forwarding implementations have to provide IP filtering capabilities
> and cryptographic hardware support to offload and distribute DoS
> mitigation from the CPU). In the others section, the need of a
> requirement on forwarding level is not discussed in general
> (e.g. there is no discussion about general OAM deployments). The OAM
> capabilities on forwarding level are defined). There are no
> requirements of forwarding level in this section (e.g. filtering,
> dampening mechanism, cryptographic capabilities, ...). Only a single
> reference in following section (2.6.2) to this section.

The DoS protection is a set of functionality which if omitted or
inadequately addressed can render an implementation unusable in many
service provider networks.  Most surviving LSR vendors "get it" but a
few newbies don't even if the commodity chip they bought and use in
their product has adequate support.  Most chip vendors appear to get
it, but some do a marginal job and that may be because they only
understand that filtering and multiple queues to the CPU are needed
but don't fully understand why.

This subsection adds about a page and IMO is worth that extra page.

> "vigalence".  You maybe mean "vigilance"

Thanks.

> Section 2.6.2, last paragraph, 3rd sentence: closing brace is

Thanks.

> text style: the abbreviation CC-CV is not described, but RDI (Remote

Thank you for pointing that out.

> Section 3:
> Q#34 and Q#35 style: in other parts of section 3, the different
> conformance questions where listed as separate question items. But
> Q#34 and Q#35 contains multiple questions.

That is a good point.  The DoS and OAM was written later than most of
the section.  I will make it more consistent.

> T#23: replace "EI" with "EL"

EI

> Style: in section 4, DoS and OAM have different "headlines" ("DoS
> protection" and "OAM Capabilities and Performance") whereas in section
> 3, DoS and OAM are under single headline ("OAM and DoS Protection").

There is more detail in Section 4 than in Section 3 on DoS.

It has been suggested that the headings in Section 3 and Section 4
could be subsections (a few would be very small subsections).  Now
that the document has grown from its initial scope, it makes sense to
do that.  At that time I can make the subsection headings consistent
across Section 3 and Section 4.

> Sorry for the required extended review time.

Thanks for the very detailed comments.

Curtis

---

From: <[email-masked]>
Date: Fri, 8 Mar 2013 15:02:35 +0100

Curtis,

two remaining topics

> discussed in this section is optional, in the real world it is

I completely understand and absolutely agree the rationale. But my concern was about the used rfc-2119 language in the text and the RFC process. But I assume you have much more experience with RFC processes than me.

> > T#23: replace "EI" with "EL"

I thought the abbreviation stands for Entropy Label

Thomas

> From: Curtis Villamizar [mailto:[email-masked]]
> Sent: Wednesday, March 06, 2013 8:03 PM
> To: Beckhaus, Thomas
> Cc: [email-masked];
> [email-masked];
> [email-masked]; [email-masked];
> [email-masked];
> [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: MPLS-RT review of draft-villamizar-mpls-forwarding
>
>
> In message
> <[email-masked]
> .T-INTERNAL.COM>

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 08 Mar 2013 13:30:01 -0500

In message <[email-masked]>
[email-masked] writes:
 
> I completely understand and absolutely agree the rationale. But my
> concern was about the used rfc-2119 language in the text and the
> RFC process. But I assume you have much more experience with RFC
> processes than me.

If the WG or IESG objects we can change it.  IMO the paragraph is
clear about the recommendations being for practical reasons of
securing sufficient market share rather than for RFC compliance.

The paragraph starts with:

   This document makes the following recommendations.  These
   recommendations are not required to claim compliance to any existing
   RFC therefoer implementors are free to ignore them, but due to
   service provider requirements may be doing so at their own peril.

My most recent experiences in the RFI/RFP process (at a transport
vendor considering LSR capability) is that among core networks, the
inability to look past the MPLS label stack is such a non-starter with
so many providers that equipment vendors would be nuts not to support
it.  There may be some that claim not to need it but they are a small
minority and may discover later that they do need it.

The same applies to PW CW.  One metro equipment vendor decided that
since it was optional they would elect not to support it.  One
provider decided since it was metro equipment it was OK.  When it came
time to offer inter-metro services based on PW the mistake became
apparent.  We have a similar situation with PW CW where the RFC says
SHOULD but for practical reasons the provider market says MUST.  Here
too we tried to include strong wording, noting that CW support is a
MUST in the most recent work out of PWE3 WG.  On a related note:
vendors wanting to do limited MPLS-TP in metro equipment might want to
contemplate the implications of inter-metro MPLS-TP LSP.

BTW- It was already pointed out that "therefoer" is a typo.

> I thought the abbreviation stands for Entropy Label

Typo in my response (more likely mouse click after searching for EI in
the document).  My response should read "OK" or somthing like that.

Curtis