Subject: Re: [nvo3] IESG review of draft-ietf-nvo3-overlay-problem-statement-03

From: Eric Gray <[email-masked]>
Date: Wed, 3 Jul 2013 03:05:56 +0000

Thomas,

                We really cannot say -

                " VLANs are a well understood construct in the networking industry,

                   providing an L2 service via a physical network in which tenant

                  forwarding information is part of the [physical] network

                   infrastructure."

- because this is not anything that VLANs are "well understood" to do.  The

term "tenant" is alien in the context of VLANs, and the (tenant associated)

information is not part of the physical network, but is (instead) included as

part of the frame-header for Ethernet frames forwarded in the context of

each VLAN.

                Perhaps if I knew why you were proposing this change to the earlier

text, I could provide a better formulation?

                I think we are trying to do is characterize the way that the L2 service

is provided by VLANs is "in-band" - by which we apparently mean that the

mechanism used for VLANs is to carry identifiers for virtual networks in

the data-frames themselves, as part of the Ethernet frame-header.

                This might be part of the changes you indicated you would do to

address Adrian's comment.  If that is the case, I would suggest that the text

should be changed to:

                "VLANs are a well understood construct in the networking industry,

                  providing an L2 service to VLAN member end-stations where VLAN

                  forwarding information is included in the PDU header of each data

                  PDU."

--

Eric

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Friday, June 28, 2013 5:27 PM
To: [email-masked]
Subject: [nvo3] IESG review of draft-ietf-nvo3-overlay-problem-statement-03

Hi.

The IESG approved the problem statement yesterday. There was one discuss, but after some dialog it was changed to a comment.

Yay!!!!

Melinda Shore also had comments as a result of her OPS directorate review, the are also covered below.

The IESG did have some comments. But because they are not "discusses", we only need to consider them and make changes as we see fit.

Going through the comments, here are my proposed responses. If these are OK, I'll reissue the document and we'll be done. :-)

I've collected all the responses into this one message.

The comments below are taken from datatracker http://datatracker.ietf.org/doc/draft-ietf-nvo3-overlay-problem-statement/ballot/

Adrian Farrel writes:

> Thanks for this document which I believe is a major step towards

> scoping and documenting the real problems in this space.  I have a

> number of fairly editorial concerns that I hope you can work through

> with your AD and document shepherd.

> ---

> In reading this document, I found it difficult to distinguish the

> requirements that arrise from the provision of multiple virtual

> networks on a common infrastrucutre (traffic isolation, address space

> isolation, virtual network creation and configuration) from those that

> are specific to the NVO3 scope (massive scaling, multi-tenancy on

> individual physical servers, no constraints on physical location of hosted services).

I'm not sure this distinction is a distinction or matters.  I.e., all of the things listed first (traffic isolation, etc) are very much requirements and in scope for NVO3, or at least closely related.

Proposed resolution: no changes to text.

> [I-D.ietf-nvo3-framework] is used as a normative reference because it

> defines terminology used in this document.

Will update document. But it does mean the document won't get published until the framework document is published...

> I would move the definiton of "in-band virtual network" from section 2

> to section 5.3 (the only place the term is used) to avoid complicating

> the definitions with concepts that appear to only be applied to L2

> networks.

Proposal:

Delete the defintion and update text in 5. (At one point this term was used more than what is in the current document.)

Old:

5.3.  802.1 VLANs

   VLANs are a well understood construct in the networking industry,

   providing an L2 service via an in-band L2 Virtual Network.  A VLAN is

   an L2 bridging construct that provides the semantics of virtual

   networks mentioned above: a MAC address can be kept unique within a

   VLAN, but it is not necessarily unique across VLANs.  Traffic scoped

   within a VLAN (including broadcast and multicast traffic) can be kept

   within the VLAN it originates from.  Traffic forwarded from one VLAN

   to another typically involves router (L3) processing.  The forwarding

   table look up operation may be keyed on {VLAN, MAC address} tuples.

New (only first sentence has changed):

   VLANs are a well understood construct in the networking industry,

   providing an L2 service via a physical network in which tenant

   forwarding information is part of the phsyical network

   infrastructure. A VLAN is an L2 bridging construct ...

> Why is the example of an Overlay Virtual Network in section picked

> from the layer 2 space when this work is supposed to consider only

> layer 3 overlays?

The solution will be over Layer 3. The example is one where overlays are used. No need to restrict that example to L3, and SPB is an obvious example to use here...

> OTOH, since this term is not used anywhere in the document, I suggest

> deleting it.

I propose to change the term to just "Overlay Network". That term is used throughout the document. (It turns out that the term "overlay virtual network" is not used elsewhere in the document). The two terms are in practice used interchangably.

> I believe section 3.1 could be rewritten without the need to say "cloud"

> or "elastic services".  This would be helpful because those marketting

> phrases do not add to the meaning.

> I think the final sentence of the paragraph captures the issues, but

> could be pulled out into a little more explanation of what happens and

> what problems it causes.

This also came up in Melinda's review. I will remove the term cloud. I think elastic is OK.

Old:

   Cloud computing involves on-demand provisioning of resources for

   multi-tenant environments.  A common example of cloud computing is

   the public cloud, where a cloud service provider offers elastic

   services to multiple customers over the same infrastructure.  In

   current systems, it can be difficult to provision resources for

   individual tenants (e.g., QoS) in such a way that provisioned

   properties migrate automatically when services are dynamically moved

   around within the data center to optimize workloads.

NEW (first sentence replaces two previous sentences):

   Some service providers offer services to multiple customers whereby

   services are dynamic and the resources assigned to support them

   must be able to change quickly as demand changes.  In current

   systems, it can be difficult to provision resources for individual

   tenants (e.g., QoS) in such a way that provisioned properties

   migrate automatically when services are dynamically moved around

   within the data center to optimize workloads.

> Section 5.3 uses the terms C-VLAN, S-VLAN, and B-VLAN, but only C-VLAN

> has been defined.

Rather than define them, I added references to IEEE-802.1Q and 802.1aq (for I-SIDs).

> Section 10 seems to me to be missing the impact that one virtual

> network might be able to have on another (for example by stressing

> network resources to cause undesirable VM mobility, or by consuming

> shared resources to make b/w or CPU unavailable).

> This is a type of self-consuming DoS.

Old:

   In the control plane, the primary security concern is ensuring that

   unauthorized control information is not installed for use in the data

   plane.  The prevention of the installation of improper control

   information, and other forms of denial of service are also concerns.

   Hereto, some environments may also be concerned about confidentiality

   of the control plane.

New:

   In the control plane, the primary security concern is ensuring that

   an unauthorized party does not compromise the control plane

   protocol in ways that improperly impact the data plane. Some

   environments may also be concerned about confidentiality of the

   control plane.

   More generally, denial of service concerns may also be an

   consideration. For example, a tenant on one virtual network could

   consume excessive network resources in a way that degrades services

   for other tenants on other virtual networks.

Benoit Claise writes:

> Comment (2013-06-27)

> Not much OPS feedback in this draft. I'm dying to see the "Operational

> Requirements submitted for IESG review" chartered item.

> Editorial:

> "Tenant Systems" should not be capitalized. Alternatively, you can

> define the term. Please expand ARMD Explain/Expand: C-VID, B-VID,

> I-VID

Tenant Systems is defined in the framework document (referenced). Reference to other terms added.

> And here is Melinda's feedback from OPS-DIR:

> I was asked to perform an OPS-DIR review of

> draft-ietf-nvo3-overlay-problem-statement.

> The document specifically targets multitenancy in large data center

> networks, describing problems arising from that scenario and how they

> may be addressed by overlay networks.

> That this document made it through working group last call at all

> should be seen as a major political accomplishment, given the level of

> rancor in the working group, and much respect is due to the chairs and

> the document authors for getting this done.

:-)

> The underlying assumption is that these virtual networks will provide

> traffic isolation.

> Minor issues:

> Section 3.1: "Cloud computing" - the document would benefit from

> eliminating that terminology and just describing the scenario ("Some

> service providers offer elastic services ... ").  "Cloud" is imprecise

> and evocative of marketing jargon.  We can talk about the need for

> dynamic provisioning more carefully, I think.

Per above, "cloud" will be removed.

> Section 3.2, second sentence: "A VM can be migrated from one server to

> another, [ ... ]."  I'm afraid it's servers all the way down - may be

> clearer to say that VMs may be migrated between hypervisors.

I think leaving this as server is OK. There is no perfect term here. Most folk understand that when you are talking about VMs, they are running servers or some other physical machine. Even worse, in industry the term "host" rather than "server" is used a lot too.

> An operational consideration for this section (3.2) is that there may

> be state associated with specific data flows to a VM that is not on

> the VM - that's resident on some sort of middlebox (firewall,

> application proxy, accelerator, cache, etc.).  I tend to think that

> network state will, in practice, be topologically close to the VM, but

> care must be taken.

This comment actually plays into the proposed NSC work...

> Doesn't really matter but it appears that the section header for

> section 3.6 is marked up incorrectly (font and bolding).

Font? Bolding? In an ascii ID? :-)

This is presumably an issue with some tool somewhere (this section title is long and doesn't fit on one line).

> 3.7 is probably one of the clearest descriptions I've seen of this

> issue - well done.

Thanks!

> 10: I'm not sure the security considerations are quite right, or at

> least not the discussion of data plane security issues.

> What are the characteristics of an overlay network that differ from a

> physical network or VPN, and how do they impact design decisions for

> the overlay?

> Also, may be worth saying something about data leakage from

> interception of control plane traffic (what inferences can be made

> from changes in topology, etc.?).

I proposed some changes above in response to Adrian's comments. But I don't see what changes the above suggests. For one thing, there is an assumption that overlays are running (primarily) in a data center, where the network is not open and tenants don't have direct access to the underlay. If they did, presumably that would raise a bunch of bigger issues than their ability to see control traffic.

Jari Arkko writes;

> Thank you for writing this document. It is well written and easy to

> read, and documents the space well.

> I had one question when reading Section 4. I was wondering why MTU was

> not mentioned, MTU issues being one of the impacts of overlay designs.

I responded:

   Well, in the data center, MTU issues are not that big of a problem

   in practice. Most hardware does jumbo frames (even if officially we

   don't like to say that, given IEEE's position on jumbo frames). And

   trying to deal with path MTU discovery and all that just doesn't

   seem worth it in the overall scheme of things. If you look at some

   of the solutions being built in this space (NVGRE and VXLAN), they

   also take a similar approach wrt MTU, i.e., avoid fragmentation.

   Also, one thing this document has had to stradle is what is a

   problem in data centers generally, and what is a problem that

   results if overlays are the solution. For the latter, that's

   arguably really for other documents to get into. If one starts

   documenting all the potential issues introduced when overlays are

   used, there is a list of issues, and we've moved beyond a problem

   statement...

To which Jari responded "OK, thanks".

Joel Jaeggli writes:

> Comment (2013-06-27)

> The document is weirdly though non-specifically ipv4-centric. I don't

> think there are any particular fixes to be applied. I would observe

> however that address reuse while common in parallel rfc1918 addressing

> planes would not I imagine be very common in ipv6 in the umbering

> plans of ipv6 enabled DCs. that the longest possible route is not a

> /32, and that a signficant scaling consideration with L3 --> L2

> mappings is the duplication between the arp cache and the NDP cache.

Interesting that I somehow didn't call out IPv6 support explicitly. :-) I guess when we say IP, we really mean both these days. I think there is a general assumption solutions will have to support both IPv4/IPv6 on the underlay/overlay. I take that as a given.

Stephen Farrell writes:

> The nodes of a virtual network, once running, can look after securing

> their own traffic. That might lead one to say that nvo3 traffic

> isolation doesn't need to consider confidentiality. However, if the

> nodes in a virtual network are VMs and if VMs can be moved, then any

> secrets required for the virtual network to secure its traffic will be

> exposed to the underlay during the move.

> I'm not clear if this wg will try address that issue or not. Section

> 10 does say that some environments might be concerned about

> confidentiality but is vague about whether or not the wg will work on

> the topic.

> Such a confidentiality service isn't a panacea of course, the underlay

> components providing the confidentiality service could leak the

> relevant keys, but it could still be useful nonetheless. (BTW, I've no

> idea if it'd make sense to have such a service that's separated from

> whatever technology is used to move the VM or not.)

> So I was wondering: will the wg actually define such a confidentiality

> service or not? The response is that yes, this'll be considered for

> the requirements documents which is fine.

> Note that I'm not trying to insist on a "yes" answer, even though I

> think that'd be good. Even a "maybe, and that'll be answered in the

> requirements specs before we re-charter"

> would be ok. But regardless of the answer, I think it'd be good to at

> least note this issue in the security considerations section.

We had some back-and-forth with the IESG on this. This is something that gets decided when we look at solutions. I.e., requirements for solutions. Too early to say that here.

> - 4.1, bullets: I found the use of ingress/egress non-intuitive here.

> You mean ingress to the underlay and egress from the underlay, right?

> It'd be good to explicitly say that, though I figured it out

> eventually (or not, if I'm wrong above:-)

I changed "egress NVE" to "egress NVE for the tunneled packet" and added "tunnel" to an earlier sentence:

New:

                  The idea behind an overlay is quite straightforward. Each

                  virtual network instance is implemented as an overlay. The

                  original packet is encapsulated by the first-hop network

                  device, called a Network Virtualization Edge (NVE), and

                  tunneled to a remote NVE. The encapsulation identifies the

                  destination of the device that will perform the

                  decapsulation (i.e., the egress NVE for the tunneled packet)

                  before delivering the original packet to the endpoint. The

                  rest of the network forwards the packet based on the

                  encapsulation header and can be oblivious to the payload

                  that is carried inside.

Ted Lemon writes:

> Comment (2013-06-27)

> Minor nit:

>    While an overlay-based approach may address some of the

>    "pain points" that were raised in ARMD (e.g., better support for

>    multi-tenancy).  Analysis will be needed to understand the scaling

>    tradeoffs of an overlay based approach compared with existing

>    approaches.

> I think you want a comma between these two chunks; otherwise it

> doesn't really parse.

> In 5.7, trill-fine-labeling is in the RFC editor queue, so I think

> that should be described as completed work, rather than something

> TRILL is investigating.

Good catch!

New:

                  TRILL is a network protocol that provides an Ethernet L2

                  service to end systems and is designed to operate over any

                  L2 link type. TRILL establishes forwarding paths using IS-IS

                  routing and encapsulates traffic within its own TRILL

                  header.  TRILL as originally defined, supports only the

                  standard (and limited) 12-bit C-VID identifier.  Work to

                  extend TRILL to support more than 4094 VLANs has recently

                  completed and is defined in

                  <xref target="I-D.ietf-trill-fine-labeling"></xref>

> In general this draft is very clearly written, and does a good job of analyzing

> the problem space.   Thanks for doing such a good job on it!

Thanks!

Thomas

_______________________________________________

nvo3 mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/nvo3

---

From: Thomas Narten <[email-masked]>
Date: Wed, 03 Jul 2013 08:18:42 -0400

Eric,

>   " VLANs are a well understood construct in the networking
>     industry, providing an L2 service via a physical network in
>     which tenant forwarding information is part of the [physical]
>     network infrastructure."

> - because this is not anything that VLANs are "well understood" to
> do.  The term "tenant" is alien in the context of VLANs, and the
> (tenant associated) information is not part of the physical network,
> but is (instead) included as part of the frame-header for Ethernet
> frames forwarded in the context of each VLAN.

Yep, I messed up on the rewording a bit here.

> Perhaps if I knew why you were proposing this change
> to the earlier text, I could provide a better formulation?

To address Adrian's comment and observation that "in-band" was barely
used and arguably didn't need to be in the terminology section...

> I think we are trying to do is characterize the way that the L2
> service is provided by VLANs is "in-band" – by which we apparently
> mean that the mechanism used for VLANs is to carry identifiers for
> virtual networks in the data-frames themselves, as part of the
> Ethernet frame-header.

Right. And a key difference between "in-band" and overlays is that in
the in-band case, the network infrastructure uses those identifiers
while making forwarding decisions. With overlays, the identifiers are
only used at the edge and not in the forwarding plane (at least not
directly -- deep packet inspection muddies this, but is really just an
optimization).

> address Adrian's comment.  If that is the case, I would suggest that
> the text should be changed to:

>   forwarding information is included in the PDU header of each data

This is better, but would prefer something like:

  VLANs are a well understood construct in the networking industry,
  providing an L2 service via a physical network in which tenant
  forwarding information (i.e., the VLAN ID) is included in the L2
  frame header and used by the physical network while making
  forwarding decisions.

(this avoids using "PDU", which the PS document hasn't used...)

Thomas

---

From: Eric Gray <[email-masked]>
Date: Wed, 3 Jul 2013 14:39:58 +0000

Thomas,

                Close, I believe.  The second occurrence of "physical network" makes

me just a bit queasy.  The frame header information is not used by the wires

(or fiber) of the physical network.   Nor is it used by other physical network

components that may still be used in some L2 networks (Hubs, repeaters, ...).

                802.1D bridges also do not use VLAN IDs to make forwarding decisions

and 802.1Q bridges operate in 802.1D compatibility mode when not configured

with VLAN information.

                How about:

  forwarding information (i.e. - the VLAN ID) is included in the L2

  frame header and used by VLAN bridges in making forwarding

  decisions."

                Also, the term "tenant" is an alien term in a general VLAN context,

but I can live with this usage because we clarify what we mean by the term

in the parenthetical text.

                Some folks question why this is needed.  If this was left out, what

do we lose?  Putting it in context, and leaving the tenant part out, this is

what we would have:

5.3.  802.1 VLANs

   providing an L2 service via an L2 Virtual Network.  A VLAN is an L2

   bridging construct that provides the semantics of virtual networks

   mentioned above: a MAC address can be kept unique within a

   VLANs are a pure L2 bridging construct and VLAN identifiers are

   carried along with data frames to allow each forwarding point to know

   what VLAN the frame belongs to.  Various types of VLANs are available

   today, which can be used for network virtualization even together.

   The C-VLAN, S-VLAN and B-VLAN IDs are 12 bits.  The 24-bit I-SID

   allows the support of more than 16 million virtual networks.

                Note that the difference between this and the text in section 5.3

now is that phrase "in-band" has been removed.  We may be going to a

lot of trouble to explain a concept that isn't needed in the context of the

draft.

--

Eric

-----Original Message-----
From: Thomas Narten [mailto:[email-masked]]
Sent: Wednesday, July 03, 2013 8:19 AM
To: Eric Gray
Cc: [email-masked]
Subject: Re: [nvo3] IESG review of draft-ietf-nvo3-overlay-problem-statement-03
Importance: High

Eric,

> - because this is not anything that VLANs are "well understood" to do.

> The term "tenant" is alien in the context of VLANs, and the (tenant

> associated) information is not part of the physical network, but is

> (instead) included as part of the frame-header for Ethernet frames

> forwarded in the context of each VLAN.

> text, I could provide a better formulation?

To address Adrian's comment and observation that "in-band" was barely used and arguably didn't need to be in the terminology section...

> service is provided by VLANs is "in-band" â€“ by which we apparently

> Ethernet frame-header.

Right. And a key difference between "in-band" and overlays is that in the in-band case, the network infrastructure uses those identifiers while making forwarding decisions. With overlays, the identifiers are only used at the edge and not in the forwarding plane (at least not directly -- deep packet inspection muddies this, but is really just an optimization).

  forwarding decisions.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 03 Jul 2013 10:59:10 -0400

Eric,

> components that may still be used in some L2 networks (Hubs,

OK, I see your queasiness. You are intepreting "physical network"
strictly, ala Tanenbaum and the original layering definitions. I've used
"physical network" more loosely, to encompass a combination of data
link and physical layers. My Bad.

> lot of trouble to explain a concept that isn't needed in the context of the

I agree.  Removing "in-band" seems like a fine way to go here.

Thomas