Subject: Re: [PWE3] AD review of draft-ietf-pwe3-fat-pw

From: Stewart Bryant <[email-masked]>
Date: Thu, 07 Apr 2011 19:56:58 +0100

Adrian

Here are the actions I have taken/reply to your points.

Lets see where we agree and if we are close to agreement I will
upload a new version

Stewart

> ---
>
> I found the use of language a little relaxed. You use the same word,
> "label", to refer to a label value and to an entry in the shim header.
> It would probably be helpful to use the term "label" to refer to the
> label itself, and "label entry" to indicate the label, TC bits, and TTL.
>
I have changed this to use label for the value and label stack entry (LSE)
as per RFC3032 for the entry in the label stack.

Hopefully I have caught all instances and disposed of them correctly.
> ---
>
> It seems to me that the description of what can be in the additional
> label entry is conflicted.
>
>     On receipt of the
>     pseudowire packet at the egress PE (which knows this additional label
>     is present) the flow label is discarded without processing.
>
> This means that the label will not be used for any purpose in the packet
> processing except in the DPI of ECMP hashing.
>
>     The flow label can never become the top label in
>     normal operation
>
> That is, you will never pop a label to discover the flow label except at
> the egress where it is expected and ignored.
>
>     Note that the flow label MUST NOT be an MPLS reserved label (values
>     in the range 0..15) [RFC3032], but is otherwise unconstrained by the
>     protocol.
>
> And why is that restriction required given the two previous points?
MPLS has an architectural assumption that a label value with a reserved
value has special meaning - always -.

Its safer to always assume that to be the case even if we "can get away 
with it."

>
>     The flow label can never become the top label in
>     normal operation, and hence the TTL in the flow label is never used
>     to determine whether the packet should be discarded due to TTL
>     expiry.  Therefore there are no lower restrictions on the TTL value.
>
> Why do you call out "lower"? How about saying "there are no restrictions"
Changed.

There was some discussion in the early PW days with, as I recall,
Eric Rosen convincing me that bad things would happen in some
implementations when a PW had TTL=1. This is a carry over from that.
>     The use of the TC bits (formerly known as the EXP bits) in the flow
>     label is outside the scope of this document.  Unless otherwise agreed
>     by the ingress and egress PEs these bits MUST be set to zero by the
>     ingress PE and MUST be ignored by the egress PE.
>
> "Out of scope" seems to be different from the subsequent statement about
> how he TC field must be treated. I am also not clear why the TTL
> discussion is different from the TC discussion.
Ah, there was a lot of discussion about this is the WG, with some
proposals to use them for signaling of state along the LSP. I was
pushing back on this on MPSL architectural grounds and the text
here was the result of some discussion in WG LC.

> ---
>
> Document title: please spell out PSN
done - but note that PSN is a well known term in PWE3
>
>
> ---
>
> Abstract: yuck! :-O
>
> Suggestion...
>
> OLD
>     Where the payload carried over a pseudowire carries a number of
>     identifiable flows it can in some circumstances be desirable to carry
>     those flows over the equal cost multiple paths (ECMPs) that exist in
>     the packet switched network.  Most forwarding engines are able to
>     hash based on label stacks and use this to balance flows over ECMPs.
>     This draft describes a method of identifying the flows, or flow
>     groups, to the label switched routers by including an additional
>     label in the label stack.
> NEW
>     Where the payload of a pseudowire comprises a number of distinct
>     flows, it can be desirable to carry those flows over the equal cost
>     multiple paths (ECMPs) that exist in the packet switched network.
>     Most forwarding engines are able to hash based on MPLS label stacks
>     and use this mechanism to balance MPLS flows over ECMPs.
>
>     This document describes a method of identifying the flows, or flow
>     groups, within pseudowires such that Label Switching Routers can
>     balance flows at a finer granularity than individual pseudowires.
>     The mechanism uses an additional label in the MPLS label stack.
> END
>
done
> ---
>
> Try to avoid "ECMP paths" (or ECMPP as we call them ;-)
I use ECMP and ECMPs for the plural
> ---
>
> Section 1 para 1
>
>     the use of ECMP paths for is known to be
>     beneficial (and not harmful) to the operation of the PW.
>
> ??
The (and not harmful) was a heads up to the reader that in
some cases it is harmful, and that it is assumed that the
reader would know when this was the case and use the
mechanism.

We could delete the parenthetical remark, but it is harmless
other than perhaps as an offense to the English.
> ---
>
> Throughout...
>
> The plural of "pseudowire" is "pseudowires", not "pseudowire's".
> You may safely use "PWs" in your document.
>
> You have a number of other plurals formed as apostrophe-s. Please
> search for "'s" and fix as necessary.
>
Done
> ---
>
> Section 1
>
>     Some pseudowires are used to transport large volumes of IP traffic
>     between routers at two locations.
>
> I assume that, for some definition of "location" this is also true for
> routers at the same location. So how about...
>
>     Some pseudowires are used to transport large volumes of IP traffic
>     between routers.
>
Indeed
> ---
>
> Section 1
>
>     Such pseudowire's do not require strict ordering to
>     be preserved between packets of the pseudowire.  They only require
>     ordering to be preserved within the context of each individual
>     transported IP flow.
>
> I see where this is going, but the first sentence is massively more
> relaxed than the second! How about turning it around...
>
>     Such pseudowires only require packet ordering to be preserved within
>     the context of each individual transported IP flow. They do not
>     require packet ordering to be preserved between all packets of the
>     across all IP flows within the pseudowire.
>
Changed to
These PWs only require packet ordering to be preserved within
the context of each individual transported IP flow. They do not
require packet ordering to be preserved between all packets of
across all IP flows within the pseudowire.

> ---

> Section 1
>
>     Some operators have requested the ability to
>     explicitly configure such a pseudowire to leverage the availability
>     of multiple ECMP paths.
>
> I wish you did not feel the need to include this assertion. Why not just
> get on and describe the benefits (as you do in the next sentence).
>
> Currently it really sounds like this is an idea that is strongly opposed
> and you are trying to justify it to the rest of the community. In fact,
> the community will judge according to the technical merits.
Traditionally this is an an anathema in PWs which try hard to look like
a "wire" and when we first mooted the idea we got pushback from the
ITU-T MPLS-TP community.

The reason that we say "Some operators" is because this was an
operator initiated, not a vendor initiated project.

I have changed this to

OLD
Some operators have requested the ability to explicitly configure such a 
PW to leverage the availability of multiple ECMPs. This allows for 
better capacity planning as the statistical multiplexing of a larger 
number of smaller flows is more efficient than with a smaller set of 
larger flows. Although Ethernet is used as an example above, the 
mechanisms described in this draft are general mechanisms that may be 
applied to any PW type in which there are identifiable flows, and in 
which there is no requirement to preserve the order between those flows.
NEW
The ability to explicitly configure such a PW to leverage the 
availability of multiple ECMPs allows for better capacity planning as 
the statistical multiplexing of a larger number of smaller flows is more 
efficient than with a smaller set of larger flows. Although Ethernet is 
used as an example above, the mechanisms described in this draft are 
general mechanisms that may be applied to any PW type in which there are 
identifiable flows, and in which there is no requirement to preserve the 
order between those flows.
END

> ---
>
>
> Please don't use the word "draft" in the text as it creates work for the
> RFC Editor and might get missed in the publication process. Obviously, by
> the time this is published as an RFC it will no longer be a draft. Best
> thing to do is search and replace "draft" with "document".
>
Done. It was an error. Tutorial not required :)
> ---
>
> Section 1
>
>     Typically, forwarding hardware can deduce that an IP payload is being
>     directly carried by an MPLS label stack, and is capable of looking at
>     some fields in packets to construct hash buckets for conversations or
>     flows.  However, an intermediate node has no information on the type
>     pseudowire being carried in the packet.
>
> Is the first sentence relevant to this document? I don't think so, but if
> you believe it is, I think you will need to add a statement about how
> this deduction is made.
I have changed the second sentence to

" However, when the MPLS payload is a PW, an intermediate node has..."

The point of this is to note that routers have been in the habit of doing
DPI and ECMP on MPLS LSPs based on deducing that the packet is an IP
packet, but PWs provide information hiding that prevents this.

I am not sure that a tutorial on how they do it is appropriate, particularly
as it's not strictly in conformance with the MPLS architecture, but
noting that de facto it happens to readers not familiar with LSR design
is I believe useful.

> ---
>
> Section 1
>
>     In the case of a pseudowire emulating
>     a high bandwidth trunk, the granularity obtained by hashing the
>     default label stack is inadequate for satisfactory load-balancing.
>
> Is "the default label stack" intended to convey some special meaning?
> I suspect you are trying to say "the label stack that would be
> constructed if this I-D hasn't been implemented". You will simply have
> to go down the path of saying...
>
>     In the case of a pseudowire emulating
>     a high bandwidth trunk, the granularity obtained by hashing the
>     label stack is inadequate for satisfactory load-balancing.
>
Changed
> ---
>
> Section 1
>
>
>     This draft proposes a method to introduce granularity on the hashing
>     of traffic running over pseudowires by introducing an additional
>     label, chosen by the ingress node, and placed at the bottom of the
>     label stack.
>
> As above: s/draft/document/
> Also, please be less timid. Your document defines a method.
Changed

Of course individual-00 did only propose it.

> ---
>
> Section 1.1
>
> s/1.1.  ECMP in Label Switched Routers/1.1.  ECMP in Label Switching Routers/
>
> s/Label switched routers/Label Switching Routers (LSRs)/
>
None
> ---
>
> Section 1.1
>
>     been proposed [I-D.kompella-mpls-entropy-label], however that is
>
> Reading would be significantly enhanced by  forward reference to section
> 8.
>
I put in the forward reference and removed the OOS wich seemed a bit
strange given we have a section introducing the issue.
> ---
> Section 3
>     It is recommended
> RECOMMENDED ?
Done
> ---
>
> Section 3
>
>     It is recommended that the method chosen to
>     generate the load balancing labels introduces a high degree of
>     entropy in their values, to maximise the entropy presented to the
>     ECMP path selection mechanism in the LSRs in the PSN, and hence
>     distribute the flows as evenly as possible over the available PSN
>     ECMP paths.
>
> It isn't clear to me that entropy is necessarily a good thing. It
> depends on the hash algorithm, doesn't it.
Entropy is a good thing (TM) in this application .. and the quality of the
entropy is a function of the quality of the hash algorithm.

> ---
>
> Figure 2
>
> Those four-octet things are not actually labels are they.

The fig now says LSE

| Flow LSE | 4 octets
+---------------------------+
| PW LSE | 4 octets
+---------------------------+
| MPLS Tunnel LSE (s) | n*4 octets (four octets per LSE)

The rest are not labels
> ---
>
> What is the purpose of the T bit? Why would an implementation go to
> the trouble of including the flow-label TLV only to set T=0?
It may wish to send T=0, R=1, i.e. I do not propose to
send a FAT label, but I can process (discard) one if you send it to me.

> ---
>
> Section 4
>
>     If PWE3 signalling [RFC4447] is not in use for a pseudowire, then
>     whether the flow label is used MUST be identically provisioned in
>     both PEs at the pseudowire endpoints.  If there is no provisioning
>     support for this option, the default behaviour is not to include the
>     flow label.
>
> This paragraph is misplaced in the section on signaling. Please put it
> elsewhere.
Added new section Static Pseudowires and moved the text there.
> I am not convinced by the 2119 language in this paragraph since an
> implementation cannot control the behavior.
It could be interpreted as an instruction to the NMS system and
therefore a mandatory implementation instruction at that level.
> ---
>
> Section 4
>
> Do you not think that you should say that the new sub-TLV is carried in
> the Interface Parameters TLV, and on which messages?
The text is consistent with the other PW definitions and the reader
has already been referred to RFC4447.

> ---
>
> Section 4.1
>
>     o  FL is the flow label sub-TLV identifier assigned by IANA.
>
> Can you either put in a [TBD] flag so the RFC Editor will catch this,
> or a forward pointer to section 10.
I put in the value 0x17 since it's already been allocated.

I change the IANA section to say:

"IANA is requested to amend the PW Interface Parameters Sub-TLV type 
Registry value 0x17 (Flow Label indicator) to refer to this RFC."
> ---
>
> Section 5
>
> Consider an egress that doesn't support flow label, but previous S-PEs
> that do. In this case you are content to throw the baby out with the
> bath water?
The fix would require a change to the S-PE behavior to remove
the FAT label. That is an operation that is more complex than a
simple label swap and therefore undesirable.
---
> Section 6
>
>     We believe that this
>     problem is addressed by the following two methods
>
> I would like you to be more definitive, please.

How about

This problem is mitigated by the following two considerations:
> But, that said, the subsequent bullets do not provide methods of
> addressing OAM in this case. Rather, the text attempts to say why
> the problem is not important. In my opinion it fails since, although
> ECMP is out of scope for MPLS-TP, a user has reasonable expectations
> that some of the flows on their PW will not be wiped out for the IGP
> convergence period, when (for example) rapid notification followed
> by change of the flow label could cause the traffic to be balanced
> differently and avoid the failure,

I have added
Note that this period may be reduced if a fast re-route or fast 
convergence technology is deployed in the network [RFC4090],[RFC5286].

> or when a partial failure of a PW
> could trigger a switch to a different PW.
Well there are two problems here:

1) The whole point is that it's difficult to constantly monitor
all paths.

2) It's not clear that the new PW would be diverse when the FL is
taken into a/c.

> ---
>
> Section 6
>
> s/corruption of an/corruption of a/
Done
> ---
>
> You need to get rid of the reference to I-D.ietf-mpls-lsr-self-test
> This work has clearly been abandoned.
The reference is replaced by

such as an LSP self test technology.

With no reference.
> ---
>
> s/Figure 4: Use of Router Alert LAbel/Figure 4: Use of Router Alert Label/
>
> ---
>
> Figure 4
>
> Shouldn't "payload" actually read "VCCV message"?
>
> Shouldn't you include the warning (from 5085) that inserting the router
> alert label probably breaks the ECMP hashing such that the packets will
> go another way.
Added

Note that, depending on the number of labels hashed by the LSR, the 
inclusion of the Router Alert label may cause the OAM packet to be load 
balanced to a different path from that taken by the data packets with 
identical Flow and PW labels.

> ---
>
> How funny that labels that are conventionally described as being "above"
> on the label stack are shown in your diagrams as below.
>
Yes, but the mistake is consistent though out all PWE3 docs going back
to RFC3985.
> ---
>
> Section 7
>
> s/The methods describe/The methods described/
> s/The method proposed/The method defined/
> s/in those cased/in those cases/
> s/The methods describe/The methods described/
Can't find it
> ---
>
> Section 7
>
>     order is not maintained amongst packets of different
>     flows.
>
> I know what you mean, but you have actually said something subtly
> different.
OLD
order is not maintained amongst packets of different flows
NEW
order is no longer maintained for all packets sent over the PW.
> ---
>
> Section 7
>
>     Ethernet may have this property, and for that reason
>     this document focuses on Ethernet.  Applications for other high-
>     access-bandwidth PW's (e.g.  Fibre Channel) may be defined in the
>     future.
>
> Does the document really *focus* on Ethernet? There are a couple of
> places where Ethernet is cited as an example (but where the text says
> equally applicable, blah, blah) and two paragraphs a little below this
> statement that say "the Ethernet PW". Nothing else.
>
> Can't you generalise?

Section 1 says

Although Ethernet is used as an example above, the mechanisms described 
in this document are general mechanisms that may be applied to any PW 
type in which there are identifiable flows, and in which there is no 
requirement to preserve the order between those flows.

I have removed that text and added this note to the bottom of Section 1

NOTE: Although Ethernet is frequently referenced as a use case in this 
RFC, the mechanisms described in this document are general mechanisms 
that may be applied to any PW type in which there are identifiable 
flows, and in which there is no requirement to preserve the order 
between those flows.

> ---
>
> Section 7
>
>     If sequence number support is required this
>     mechanism is not applicable.
>
> It's stronger than that, isn't it?
> You need to prohibit the mechanism.
> But shouldn't you also prohibit ECMP?
Now says:
If sequence number support is required the flow label mechanism MUST NOT 
be used.
> ---
>
> Section 7
>
>     Where it is known that the traffic carried by the Ethernet pseudowire
>     is IP the method of identifying the flows are well known and can be
>     applied. Such methods typically include hashing on the source and
>     destination addresses, the protocol ID and higher-layer flow-
>     dependent fields such as TCP/UDP ports, L2TPv3 Session ID's etc.
>
> Either s/method/methods/ or s/are/is/
done
> If these methods are so well known, you will have no difficulty in
> supplying a reference or two.
I am fairly sure that there is no IETF reference to this. It something
that all the vendors do (differently), but there has always been strong
resistance to the publication of a standard.
>
> I think you have mixed the method of identifying the flows with the
> method of hashing the flows into buckets.
The text now says:

Where it is known that the traffic carried by the Ethernet PW is IP the 
method of identifying the flows and mapping them to an ECMP are well 
known and can be applied. Such methods are typically include hashing on 
the source and destination addresses, the protocol ID and higher-layer 
flow-dependent fields such as TCP/UDP ports, L2TPv3 Session ID’s etc.
> ---
>
> Section 7.1
>
>     In the ECMP case and the link bundling case the NSP may attempt to
>     take bandwidth into consideration when allocating groups of flows to
>     a common path.  That is permitted, but it must be borne in mind that
>     the semantics of a label stack entry (LSE) as defined by [RFC3032]
>     cannot be modified, the value of the flow label cannot be modified at
>     any point on the LSP, and the interpretation of bit patterns in, or
>     values of, the flow label by an LSR are undefined.
>
> If you have something to say, you should say it. Otherwise your paragraph
> is lost on your readers.

> I think you are trying to say "There is no mechanism currently defined
> to indicate the bandwidths in use by specific flows using the fields of
> the MPLS shim header. Furthermore, since the semantics of the MPLS shim
> header are fully defined in [RFC3032] and [RFC5462], those fields cannot
> be assigned semantics to carry this information. This document does not
> define any semantic for use in the TTL or TC fields of the label entry
> that carries the flow label, but requires that the flow label itself
> be selected with a high degree of entropy suggesting that the label
> value should not be overloaded with additional meaning in any subsequent
> specification."

OK works for me, but the WG should note the change of wording.
> ---
>
> Section 7.1
>
>     A different type of load balancing is the desire to carry a
>     pseudowire over a set of PSN links in which the bandwidth of members
>     of the link set is less than the bandwidth of the pseudowire.  This
>     problem is addressed in [I-D.stein-pwe3-pwbonding].  Such a mechanism
>     can be considered complementary to this mechanism.
>
>
> The referenced I-D seems to be dead. I'm not quite sure why you need to
> reference it here. How about just taking out the paragraph?
It's in there as a result of WG discussion
> ---
>
> Section 7.2 appears to describe exactly the link bundling case that you
> discussed in section 7. I have recently noticed considerable confusion
> between link bundles and LAG and suggest you restrict LAG to be a data
> link function that is below the level of an "interface" in the IP/MPLS
> world. That does not mean that an individual node, knowing (by
> unspecified means) that it is about to use an interface that is
> constructed from a LAG, can't hash the traffic across the LAG.
>
> Ultimately, you should consider ECMP (i.e. multiple IGP paths) where
> each hop on each path may be made a link bundle, and where each
> component link of a bundle may be supported by LAG.
>
> Can you go back and check that you have used the language precisely.
The text was added as a result of WG discussion where it was proposed
that LAG might use the FL as a key in member selection.
> ---
>
> Section 7.4
>
>     If the payload on a PW is made of a single inner flow (i.e. an
>     encrypted connection between two routers), or the flow identifiers
>     are too deeply buried in the packet, then the functionality described
>     in this document does not give any benefits, though neither does it
>     cause harm relative to the existing situation.
>
> Is that s/i.e./e.g./?
Changed
> "Too deeply buried" meaning "If the NSP cannot access sufficient
> information to distinguish flows, perhaps because the protocol stack
> is too deep for its capabilities..."?
>
> Is the statement about "no harm" necessary? After all, the mechanism
> will simply not be used in this case.
Text was added as a result of WG discussion on benefits and usage of the
mechanism.
> ---
>
> Section 7.4
>
>     The most common case
>     where a single flow dominated the traffic on a PW is when it is used
>     to transport enterprise traffic.  Enterprise traffic may well consist
>     of a large single TCP flows, or encrypted flows that cannot be
>     handled by the methods described in this document.
>
> s/dominated/dominates/
yes
> s/a large single TCP flows/large TCP flows/
>
Again this was WG agreed text
> Note that this second half of the paragraph is a non-sequitur from the
> first half. You probably need to wrap the whole paragraph with a thought
> about single or dominant large flows.
That is the title of the section
> ---
>
> Section 7.4
>
>     1.  The operator can do nothing and the system will work as it does
>         without the flow label.
>
> s/can do nothing/can choose to do nothing/
done
> ---
>
> Section 7.4
>
>     An operator has six options under these circumstances:
>
> :-) You tease!
>
> There are only five listed.
:) I wonder if I deleted one at some time in the past
> ---
>
> Section 7.4
>
> s/an single flow/a single flow/
done
> ---
>
> Section 7.4
>
>     5.  The operator could configure the ingress to assign a label of
>         special significance (such as a reserved label) to all high
>         bandwidth flows so that some other action (not specified in this
>         document) could be taken on the flow.
>
> I think you need to delete this as it contradicts two specific
> statements made elsewhere...
>
> 1. MUST NOT use a reserved label as the flow label
> 2. MUST NOT assign semantics to the flow label
I have deleted (such as a reserved label).

This was added as a result of WG discussion. I think that the idea
was to use a label from the per platform space.

> ---
>
> 8.  Applicability to MPLS
>
> While I see where you are coming from, I must point out that the whole
> of your document is about MPLS. Could you come up with a better title?
Now says

Applicability to MPLS LSPs
> ---
>
> Section 9
>
>     The ingress PE SHOULD take steps to ensure that the load-balance
>     label is not used as a covert channel.
>
> Again, you are saying something between the lines :-(
No, just warning that a badly implemented hash could have side effects.
> Since the ingress PE allocates the label (isn't it called the flow label
> not the load-balance label?) the "SHOULD take steps" is very odd! And
> why is this a security requirement? How would security be compromised?
Yes - LB label was an older name for it.

I can take it out, or we can leave it to SECdir to advise.

> ---
>
> Section 9
>
> This document has signaling extensions in it, so you should reference
> RFC 4447 from this section. You should also reference RFC 5920.
Added:
The security considerations in [RFC5920] also apply.

> ---
>
> Section 9
>
>     The flow label TTL should therefore never be considered
>     by the forwarder, and hence SHOULD be set to a value of 1.  This will
>     prevent the packet being inadvertently forwarded based on the value
>     of the flow label.  Note that this may be a departure from
>     considerations that apply to the general MPLS case.
>
> But you have previously said that the flow label TTL is never used to
> discard a packet based on TTL expiry [section 1.2]. You need to make
> your mind up. And, if there are rules you wish to apply for setting
> fields, you need to put them in the main part of the protocol spec,
> not buried in the security section.
>
> And (while I'm at it) how is this a security feature?
The point is that there is no protocol reason for it to have any particular
value.

There is a security reason (stopping the packet going some bad place)
for stopping it going anywhere, but this is only a recommendation not
a requirement.

> Oh, and another thing...
>
> The sequence of your logic is broken. It would be fine to say:
> If the flow label is inadvertently examined as if it were a normal
> label, the packet might be forwarded. This can be prevented by setting
> the associated TTL to 1.
> But I don't think that "should never be considered" leads to "should be
> set to 1".
>
> Finally, the note at the end is at best unhelpful. What are you trying
> to say, and why?

The text now says:

The flow LSE is at the bottom of label stack, therefore, even when 
penultimate hop popping is employed, it will always be will preceded by 
the PW label on arrival at the PE. If the flow label is inadvertently 
examined as if it were a normal label, the packet might be forwarded. 
This can be prevented by setting the associated TTL to 1. Note that this 
may be a departure from considerations that apply to the general MPLS case.
> ---
>
> Section 10
>
> Please change this text to reference early allocation that has been
> made for you. Note that the value 17 has NOT been assigned (although
> the value 0x17 has).
Changed - it's actually ER - and has been allocated.

Stewart

---

From: Alexander Vainshtein <[email-masked]>
Date: Fri, 8 Apr 2011 07:57:45 +0300

Stewart, Adrian and all,

I would like to point (again!) to an (implicit) linkage between this draft and draft-ietf-pwe3-mpls-tp-gal-in-pw-00

Both drafts allow the PW label stack entry not to be at the bottom of the label stack.

The caveat in draft-ietf-pwe3-fat-pw ("the flow label MUST NOT be an MPLS reserved label")
discussed in this thread actually allows (at a price!) to differentiate between GAL and fat PW labels.

Whether the price is acceptable or not is a matter of opinion and depends on implementation.

Implementations that assume that the PW label is not special in any way and that its associated action in the ILM is
"Pop and forward to the specific PW forwarder (regardless of the BOS bit)"  would be fully compliant with RFC 3031,
RFC 3985 and draft-ietf-pwe3-fat-pw. But they would not natively accommodate draft-ietf-pwe3-mpls-tp-gal-in-pw-00.

Implementations that always check for the PW label being BOS and doing the ILM lookup of the next label if it is not
BOS would natively accommodate draft-ietf-pwe3-mpls-tp-gal-in-pw-00 but would not support draft-ietf-pwe3-fat-pw.

IMHO it would be useful to re-examine the architectural implications of combining these two drafts.

That said, I support draft-ietf-pwe3-fat-pw "as is".

Regards,
     Sasha

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Stewart Bryant [[email-masked]]
Sent: Thursday, April 07, 2011 9:56 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [PWE3] AD review of draft-ietf-pwe3-fat-pw

Adrian

Stewart

Changed.

done

Done
Indeed
Changed to

I have changed this to

OLD

Changed
Changed

None
Done

The fig now says LSE

| Flow LSE | 4 octets

The rest are not labels

How about

I have added

Done

With no reference.
Added

Can't find it
OLD

Section 1 says

Now says:
done
The text now says:

flow-dependent fields such as TCP/UDP ports, L2TPv3 Session ID's etc.

Changed
yes
done
done

Now says

Added:

The text now says:

Stewart
_______________________________________________
pwe3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pwe3

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 08 Apr 2011 08:25:27 +0100

Hi Sasha

Thanks for the comments
> discussed in this thread actually allows (at a price!) to differentiate between GAL and fat PW labels.
This was not included explicitly to support PW/GAL (although it does). 
It was included because
the reserved labels are a general purpose escape mechanism which should 
always be
trapped. However as you note  the text omits to say that.
> RFC 3985 and draft-ietf-pwe3-fat-pw. But they would not natively accommodate draft-ietf-pwe3-mpls-tp-gal-in-pw-00.
An interesting point.

The implication here is that implementations MUST check that a system 
receiving a flow label MUST
check that it is not a reserved label.

The text at the moment (-5 pre Adrian's comments) says:

The forwarder at the egress PE uses the pseudowire label to identify
the pseudowire.  From the context associated with the pseudowire
label, the egress PE can determine whether a flow label is present.
If a flow label is present, the label is discarded.

It should really say:

The forwarder at the egress PE uses the pseudowire label to identify
the pseudowire.  From the context associated with the pseudowire
label, the egress PE can determine whether a flow LSE is present.
If a flow LSE is present, it MUST be checked to determine whether it
carries a reserved label. If it is a reserved label the packet is
processed according to the rules associated with that reserved
label, otherwise the LSE is discarded.

I have changed the text in candidate -06.

Of course the conflict with the GAL mechanism cannot occur because
a system that was no trapping GALs would not agree to the use of
a GAL based OAM, so I think the original text is safe. However it is
bad practice not to note that an MPLS system MUST process the
reserved label set when they are ToS.

A side note, a pre-standards implementation will not be broken
by this late change, since it will have not negotiated use of GAL,
and an ingress PE will not have been sending a reserved label as
a flow label. So simple discarding the flow label is backwards
compatible but not future proof.

> BOS would natively accommodate draft-ietf-pwe3-mpls-tp-gal-in-pw-00 but would not support draft-ietf-pwe3-fat-pw.
Such implementations would not advertise that they could receive the 
flow label and hence would never
be sent one, so I can't see how the case applies.

- Stewart

---

From: Alexander Vainshtein <[email-masked]>
Date: Fri, 8 Apr 2011 18:00:37 +0300

Stewart,
Lots of thanks for a prompt and very detailed response..

One additional point (based on these responses) I'd like to make is that ability to negotiate GAL below the PW label
exists (or will exit) for signaled PWs, but not for static PWs - which seem to be the prime target application for using GAL IMHO.

Hence I am not sure that relying on the negotiation mechanisms would suffice.

This is less important for the fat PW label IMO.

My 2c,

     Sasha

________________________________________
From: Stewart Bryant [[email-masked]]
Sent: Friday, April 08, 2011 10:25 AM
To: Alexander Vainshtein
Cc: [email-masked]; [email-masked]
Subject: Re: [PWE3] AD review of draft-ietf-pwe3-fat-pw

Hi Sasha

Thanks for the comments
An interesting point.

It should really say:

- Stewart

---

From: Adrian Farrel <[email-masked]>
Date: Fri, 15 Apr 2011 17:48:28 +0100

Hi Stewart,

Thanks for this.

> Hopefully I have caught all instances and disposed of them correctly.

OK

> Its safer to always assume that to be the case even if we "can get away

Well, OK. I don't object to this reason. 

I was only probing because it seems to me that the FAT label is not a label at
all :-)

> > Why do you call out "lower"? How about saying "there are no restrictions"

OK

> implementations when a PW had TTL=1. This is a carry over from that.

Right. I can conceive of implementations that do TTL decrement and check before
they do label examination. Thus, at an egress when the PW label is popped and
the FAT label is uncovered, the TTL might have been examined. But this is not an
issue since all FAT label processing at an egress is new function.

> pushing back on this on MPLS architectural grounds and the text

I hear you when you say that the WG consensus asked you to add text "against
your better judgement" :-0

But the text you have added is conflicted. Can I suggest...

     This document does not define a use for the TC bits (formerly known
     as the EXP bits) in the flow label. Future documents may define a 
     use for these bits, therefore implementations conforming to this
     specification MUST set the TC bits to zero at the ingress and MUST
     ignore them at the egress.

> done - but note that PSN is a well known term in PWE3

Hmmm. Since I wrote this, PSN has been awarded an asterisk in
http://www.rfc-editor.org/rfc-style-guide/abbrev.expansion.txt

> >     The mechanism uses an additional label in the MPLS label stack.

Thanks

> I use ECMP and ECMPs for the plural

OK

> other than perhaps as an offense to the English.

It was "for is known" that puzzled me.

> > search for "'s" and fix as necessary.

Thanks

> >     Some pseudowires are used to transport large volumes of IP traffic

OK

> across all IP flows within the pseudowire.

s/of across/of/
or
s/of across/across/

> a "wire" and when we first mooted the idea we got pushback from the

gawd blessem

> order between those flows.

Perfect (except see below)

> Done. It was an error. Tutorial not required :)

Hmmm.
Except...
See the previous paragraph that you suggested }:-Z

> noting that de facto it happens to readers not familiar with LSR design

I can live with it.

> >     label stack is inadequate for satisfactory load-balancing.

OK

> Of course individual-00 did only propose it.

Yup

> > s/Label switched routers/Label Switching Routers (LSRs)/

Or maybe "Done"

> strange given we have a section introducing the issue.

OK

> Done

OK

> entropy is a function of the quality of the hash algorithm. 

Maybe the bad hash algorithms are so bad that we don't need to pursue this. But
you could also add (not mandatory to add):

"Although this document does not attempt to specify any hash algorithms, it is
suggested that any such algorithm should be based on the assumption that there
will be a high degree of entropy in the values assigned to the load balancing
labels."

> > Those four-octet things are not actually labels are they.
 
OK

> send a FAT label, but I can process (discard) one if you send it to me.

I see that, but are you suggesting that an LSR will change the value of the R
bit it sends depending on the value of the T bit it receives?

Well, it doesn't matter. This is just me saying "I wouldn't have designed it
like that." Nothing is broken, and there are plenty of spare bits.

No need to change anything.
 
> Added new section Static Pseudowires and moved the text there.

Good

> therefore a mandatory implementation instruction at that level.

OK. Let it go.

> has already been referred to RFC4447.

Well, I was confused :-)

I would:
OLD
   When using the signalling procedures in [RFC4447], a Pseudowire
   Interface Parameter Flow Label Sub-TLV (FL Sub-TLV) type is used to
   synchronise the flow label states between the ingress and egress PEs.
NEW
   When using the signalling procedures in [RFC4447], a new Pseudowire
   Interface Parameter Sub-TLV, the Flow Label Sub-TLV (FL Sub-TLV), is
   used to synchronise the flow label states between the ingress and 
   egress PEs.
END

That may look like a small change, but it would have made reading a lot easier.

> Registry value 0x17 (Flow Label indicator) to refer to this RFC."

OK about putting "17" in section 4.1
IANA discussed further under section 10.

> simple label swap and therefore undesirable.

OK. Just checking that there is no desire to be able to do ECMP on a MSPW in
this case.

> This problem is mitigated by the following two considerations:

Not really :-(
The second bulletin particular is not a mitigation at all.
See below for suggested text.

> convergence technology is deployed in the network [RFC4090],[RFC5286].

OK

> 2) It's not clear that the new PW would be diverse when the FL is

So what you are saying is that "OAM cannot be used under all circumstances to
verify that a pseudowire is fully functional"

I think we can handle this through:
OLD
   Where it is required to verify that a pseudowire is fully functional
   for all flows, VCCV [RFC5085] connection verification message MUST be
   sent over each ECMP path to the pseudowire egress PE.  This problem
   is difficult to solve and scales poorly.  We believe that this
   problem is addressed by the following two methods:

   1.  If a failure occurs within the PSN, this failure will normally be
       detected by the PSN's Interior Gateway protocol (IGP) link/node
       failure detection mechanism (loss of light, bidirectional
       forwarding detection [RFC5880] or IGP hello detection), and the
       IGP convergence will naturally modify the ECMP set of network
       paths between the Ingress and Egress PE's.  Hence the PW is only
       impacted during the normal IGP convergence time.

   2.  If the failure is related to the individual corruption of an
       Label Forwarding Information database (LFIB) entry in a router,
       then only the network path using that specific entry is impacted.
       If the PW is load balanced over multiple network paths, then this
       failure can only be detected if, by chance, the transported OAM
       flow is mapped onto the impacted network path, or all paths are
       tested.  This type of error may be better solved be solved by
       other means such as LSP self test [I-D.ietf-mpls-lsr-self-test].
YOURS
   Where it is required to verify that a pseudowire is fully functional
   for all flows, VCCV [RFC5085] connection verification message MUST be
   sent over each ECMP path to the pseudowire egress PE.  This problem
   is difficult to solve and scales poorly.  This problem is mitigated
   by the following two considerations:

   1.  If a failure occurs within the PSN, this failure will normally be
       detected by the PSN's Interior Gateway protocol (IGP) link/node
       failure detection mechanism (loss of light, bidirectional
       forwarding detection [RFC5880] or IGP hello detection), and the
       IGP convergence will naturally modify the ECMP set of network
       paths between the Ingress and Egress PE's.  Hence the PW is only
       impacted during the normal IGP convergence time.  Note that this 
       period may be reduced if a fast re-route or fast convergence 
       technology is deployed in the network [RFC4090],[RFC5286].

   2.  If the failure is related to the individual corruption of an
       Label Forwarding Information database (LFIB) entry in a router,
       then only the network path using that specific entry is impacted.
       If the PW is load balanced over multiple network paths, then this
       failure can only be detected if, by chance, the transported OAM
       flow is mapped onto the impacted network path, or all paths are
       tested.  This type of error may be better solved be solved by
       other means such as an LSP self test technology.
NEW
   Where it is required to verify that a pseudowire is fully functional
   for all flows, VCCV [RFC5085] connection verification message MUST be
   sent over each ECMP path to the pseudowire egress PE.  This solution
   may be difficult to achieve and scales poorly.  

   Under these circumstances, it may be sufficient to send VCCV messages
   using any load balance pseudowire path because if a failure occurs
   within the PSN the failure will normally be detected and repaired by
   the PSN.  That is, the PSN's Interior Gateway protocol (IGP) 
   link/node failure detection mechanism (loss of light, bidirectional
   forwarding detection [RFC5880] or IGP hello detection), and the IGP
   convergence will naturally modify the ECMP set of network paths
   between the Ingress and Egress PE's.  Hence the PW is only impacted
   during the normal IGP convergence time.  Note that this period may be
   reduced if a fast re-route or fast convergence technology is deployed
   in the network [RFC4090],[RFC5286].

   If the failure is related to the individual corruption of a Label
   Forwarding Information database (LFIB) entry in a router, then only
   the network path using that specific entry is impacted.  If the PW 
   is load balanced over multiple network paths, then this failure can
   only be detected if, by chance, the transported OAM flow is mapped 
   onto the impacted network path, or if all paths are tested.  Since
   testing all paths may present problems as noted above, other
   mechanisms to detect this type of error may need to be developed,
   such as an LSP self test technology.
END

??

??

> identical Flow and PW labels.

Great

> Yes, but the mistake is consistent though out all PWE3 docs going back

Oooh, quick. Raise an erratum ;-)

> > s/The methods describe/The methods described/

accidental duplication

> order is no longer maintained for all packets sent over the PW.

OK

> flows, and in which there is no requirement to preserve the order

Good

> If sequence number support is required the flow label mechanism MUST NOT

Perfect

> > Either s/method/methods/ or s/are/is/

OK

> flow-dependent fields such as TCP/UDP ports, L2TPv3 Session ID's etc.

Is OK, although I would prefer to remove the "well known" bit.

NEW
  Where it is known that the traffic carried by the Ethernet PW is IP the
  flows can be identified and mapped to an ECMP. Such methods typically
  include hashing on the source and destination addresses, the protocol
  ID and higher-layer flow-dependent fields such as TCP/UDP ports, L2TPv3
  Session ID's etc.
END

Not required that you take this change, but s/are typically/typically/

> OK works for me, but the WG should note the change of wording.

Yup

> It's in there as a result of WG discussion

If the WG intends to adopt the I-D and run with it, that is fine (provided
in-charter, and they actually do it).
Otherwise the reference is a problem. Maybe the sentence containing the
reference can be removed?

> that LAG might use the FL as a key in member selection.

Unclear whether you are saying that you think the language *is* precise, or that
you have adopted text generated in the WG and are stuck with it. In the former
case, I'm done. in the latter case my question is extended to the WG.

[Perhaps we will need to extract a couple of questions from this I-D to go to
the WG in a separate thread?]

> >     cause harm relative to the existing situation.

OK

> Text was added as a result of WG discussion on benefits and usage of the

OK.
I still think some explanation of "too deeply buried" would be beneficial.

> >     handled by the methods described in this document.

OK

> Again this was WG agreed text

Well, really. Does the WG insist on publishing stuff that is not in English?
What will happen during RFC Editor processing?

Please select from:
- a single, large TCP flow
- large TCP flows

Ask the WG if you feel the need, although to me it is a typo that has introduced
ambiguity.

> That is the title of the section

Good point.

> > s/can do nothing/can choose to do nothing/

OK

> :) I wonder if I deleted one at some time in the past

OK

> > s/an single flow/a single flow/

OK

> was to use a label from the per platform space.

OK.
Need to check back with the WG?

> Applicability to MPLS LSPs

OK

> I can take it out, or we can leave it to SECdir to advise.

I have been looking at covert channels a bit recently (not my favourite) and the
problem with requesting that the ingress PE take steps to avoid the label being
used as a covert channel is that there is not much that can be done by the
ingress to protect the label being changed within the network.

The same arguments will apply to the TTL and TC of the flow label.

The TTL and TC can be enforced by the *egress* mandating that the received
packets all have specific TTL and TC settings in the flow label.
The flow label value can only be enforced by the ingress and egress negotiating
which flow label values are in use.

Maybe:

NEW
The flow label value and the TTL and TC of the flow label LSE provide the
potential for the creation of a covert channel. An egress PE MAY monitor the
values received and report fluctuations as potential symptoms of an attack.
END

> The security considerations in [RFC5920] also apply.

OK

> may be a departure from considerations that apply to the general MPLS case.

Works well.

> Changed - it's actually ER - and has been allocated.

So it would be helpful to tell IANA in this section that they have already made
an allocation using whatever policy.

Cheers,

Adrian

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 18 Apr 2011 18:43:17 +0100

Done
> It was "for is known" that puzzled me.

The text now says:
"The new capability proposed in this document is an OPTIONAL mode which 
may be used when the use of ECMP is known to be beneficial (and not 
harmful) to the operation of the PW."
>> across all IP flows within the pseudowire.
Did the first of these

> See the previous paragraph that you suggested }:-Z
It's the order that you apply the edits that counts :) - Text is good.
> will be a high degree of entropy in the values assigned to the load balancing
Done.
Done
> IANA discussed further under section 10.
Added 0x17 to the figure and a ref to section 10 (now section 11)
Done
Done
Done
Done
>> OK works for me, but the WG should note the change of wording.
WG please note.
> reference can be removed?
It now says:

Proposals to address this problem have been made in the past 
[I-D.stein-pwe3-pwbonding]. Such a mechanism can be considered 
complementary to this mechanism.

The reference is info, so not blocking and anyone wishing to find the 
ref can do so even though it is more than 6mths old.

OK.
> I still think some explanation of "too deeply buried" would be beneficial.
Removed the harm statement, and changed words. It now says

"If the NSP cannot access sufficient information to distinguish flows, 
perhaps because the protocol stack required parsing further into the 
packet than it is able, then the functionality described in this 
document does not give any benefits. "

> Ask the WG if you feel the need, although to me it is a typo that has introduced
Used "a single, large TCP flow"
> Need to check back with the WG?
Given that the idea has not developed since it was initially proposed
I have deleted it, but will put it back if the WG request me to do so.

> values received and report fluctuations as potential symptoms of an attack.
Ingress and egress may be in a pact.

I will delete the sentence and see what the Security directorate say.
> an allocation using whatever policy.
It now says

IANA is requested to amend the PW Interface Parameters Sub-TLV type 
Registry value 0x17 (Flow Label indicator) to refer to this RFC.

I will compile and post, then hand the draft back to Matthew to see what 
he wants to check with the WG.

Thanks for the careful review.

- Stewart