Subject: [nvo3] NVO3 Data Plane Requirements: QoS identifier in overlay header?

From: <[email-masked]>
Date: Mon, 18 Jun 2012 20:04:55 -0400

Catching up ... 

In draft-bl-nvo3-dataplane-requirements-00.txt, the last paragraph
in Section 3.3.1.2. Service QoS identifier says:

   Support for NVE Service CoS SHOULD be provided through a QoS field,
   inside the NVO3 overlay header. Examples of service CoS provided
   part of the service tag are 802.1p and DE bits in the VLAN and PBB
   ISID tags and MPLS EXP bits in the VPN labels.

Why "inside the NVO3 overlay header"?  Earlier in the section, IP DSCP
and Ethernet 802.1p are mentioned as examples of CoS (Class of Service)
indicators and QoS fields - when used between NVEs, these (obviously)
belong in the outer IP and outer Ethernet headers respectively.

What's the rationale for an additional QoS field in the overlay header?

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Anoop Ghanwani <[email-masked]>
Date: Wed, 27 Jun 2012 12:45:23 -0700

> What's the rationale for an additional QoS field in the overlay header?

I don't see a need for it either.

Anoop

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 28 Jun 2012 13:06:01 -0500

David,  

Sorry about the delay, was out in vacation for a while and I missed your email.

In section 3.3.1.2 the text above the paragraph you mention below is describing why this is required. Let me try a practical explanation. When using different overlay/VPN technologies Service Providers have available CoS indicators at 3 levels:
- customer/tenant controlled CoS (e.g. DSCP/dot1p bits in the inner payload)
- service CoS visible in the NVE/PE (e.g. 3 exp bits in the VPN label/dot1p bits in IEEE ISID)
- tunnel/core CoS visible in the NVE and Core Nodes (e.g. 3 exp bits in the tunnel label/dot1p bits in IEEE BVLAN)

The last two are under the control of the Service Provider and may be used for different CoS schemes. For example 6 service CoS may be used in the NVE/PEs while only 4 may be used in the core. This could be because of available queuing/different oversubscription/number of flows/operational requirements in different network devices/admin domains.

CoS usage may change as the packet transitions between different SP administrative domains. At the handoff between domains there are rules to map one SP CoS definition to the one for the next domain. The presence of these fields is useful because there is no need to be aware about customer CoS definitions to map to the next CoS domain: i.e. if I am domain 2 network admin I just need to understand domain 1 CoS definition not how their individual customers marked their CoS. Also there is no need to dig on the next encap level.

I hope this helps clarify the text...

Florin

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Monday, June 18, 2012 5:05 PM
> To: [email-masked]
> Subject: [nvo3] NVO3 Data Plane Requirements: QoS identifier in overlay
> header?
> 
> Catching up ...
> 
> In draft-bl-nvo3-dataplane-requirements-00.txt, the last paragraph in
> Section 3.3.1.2. Service QoS identifier says:
> 
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Yu-Shun Wang <[email-masked]>
Date: Thu, 28 Jun 2012 18:54:58 +0000

Hi Florin,

The rules for matching/translating the QoS/CoS-related information when encapsulating and decapsulating between the virtual networks and the physical networks should definitely be mandated, and there are lots of existing docs addressing those. I am not so keen on defining yet another CoS/QoS scheme in the overlay header. Sure, the burden is on the physical infrastructure to respect the existing QoS/CoS marking when traversing devices bridging different technologies, but that problem exists today (or preferably has guidance already) independent of NVO3. Agree with David on this one.

Thanks,
Yushun

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Balus, Florin Stelian (Florin)
Sent: Thursday, June 28, 2012 11:06 AM
To: [email-masked]; [email-masked]
Subject: Re: [nvo3] NVO3 Data Plane Requirements: QoS identifier in overlay header?

David,  

Florin

> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of [email-masked]
> Subject: [nvo3] NVO3 Data Plane Requirements: QoS identifier in 
> overlay header?
> 
> Catching up ...
> 
> and Ethernet 802.1p are mentioned as examples of CoS (Class of 
> Service) indicators and QoS fields - when used between NVEs, these 
> (obviously) belong in the outer IP and outer Ethernet headers respectively.
> 
> David L. Black, Distinguished Engineer EMC Corporation, 176 South St., 
> Hopkinton, MA  01748

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 28 Jun 2012 14:08:55 -0500

Hi Yushun,

This is not only for devices bridging different technologies. Even if there are two admin domains using both an IP/GRE core interconnected via gateway NVEs, the service CoS field is usually used to ensure appropriate level of CoS processing in the gateway NVEs without the need to understand the individual customer CoS definitions. 
Thanks,

Florin

> From: Yu-Shun Wang [mailto:[email-masked]]
> Sent: Thursday, June 28, 2012 11:55 AM
> To: Balus, Florin Stelian (Florin); [email-masked]; [email-masked]
> Subject: RE: [nvo3] NVO3 Data Plane Requirements: QoS identifier in
> overlay header?
> 
> Hi Florin,
> 
> The rules for matching/translating the QoS/CoS-related information when
> encapsulating and decapsulating between the virtual networks and the
> physical networks should definitely be mandated, and there are lots of
> existing docs addressing those. I am not so keen on defining yet
> another CoS/QoS scheme in the overlay header. Sure, the burden is on
> the physical infrastructure to respect the existing QoS/CoS marking
> when traversing devices bridging different technologies, but that
> problem exists today (or preferably has guidance already) independent
> of NVO3. Agree with David on this one.
> 
> Thanks,
> Yushun
> 
> Balus, Florin Stelian (Florin)
> In section 3.3.1.2 the text above the paragraph you mention below is
> describing why this is required. Let me try a practical explanation.
> When using different overlay/VPN technologies Service Providers have
> available CoS indicators at 3 levels:
> - tunnel/core CoS visible in the NVE and Core Nodes (e.g. 3 exp bits in
> the tunnel label/dot1p bits in IEEE BVLAN)
> 
> The last two are under the control of the Service Provider and may be
> used for different CoS schemes. For example 6 service CoS may be used
> in the NVE/PEs while only 4 may be used in the core. This could be
> because of available queuing/different oversubscription/number of
> flows/operational requirements in different network devices/admin
> domains.
> 
> CoS usage may change as the packet transitions between different SP
> administrative domains. At the handoff between domains there are rules
> to map one SP CoS definition to the one for the next domain. The
> presence of these fields is useful because there is no need to be aware
> about customer CoS definitions to map to the next CoS domain: i.e. if I
> am domain 2 network admin I just need to understand domain 1 CoS
> definition not how their individual customers marked their CoS. Also
> there is no need to dig on the next encap level.
>

---

From: <[email-masked]>
Date: Thu, 28 Jun 2012 15:33:50 -0400

Florin,

I think I see where our views diverge.  If one considers a service provider
environment running a VPN technology that supports both service and underlay
CoS, then one has three CoS layers, especially when the underlay CoS field
has expressibility limits (e.g., MPLS support for DiffServ is constrained by
the 3-bit size of the EXP field and other competing uses for those bits).

OTOH, there are plenty of other environments for nvo3 overlays where there
aren't three CoS layers and there aren't serious expressibility problems in
the outermost CoS field (e.g., the DSCP field in the IP header imposes no
limits on DiffServ expressibility beyond those in the basic DiffServ
architecture).  Enterprise environments are among those that provide examples.
IPsec VPNs also provide examples under the assumption that the tunnels are
end-to-end wrt boundaries between service provider networks.

So while I agree that there are environments in which the CoS field can be
"useful", that's as far as I would go.  Specifically, I don't think the
across-the-board "SHOULD" in the draft is justifiable or appropriate in
general.

Thanks,
--David

> Sent: Thursday, June 28, 2012 2:06 PM
> To: Black, David; [email-masked]
> describing why this is required. Let me try a practical explanation. When
> using different overlay/VPN technologies Service Providers have available CoS
> indicators at 3 levels:
> The last two are under the control of the Service Provider and may be used for
> different CoS schemes. For example 6 service CoS may be used in the NVE/PEs
> while only 4 may be used in the core. This could be because of available
> queuing/different oversubscription/number of flows/operational requirements in
> different network devices/admin domains.
> 
> administrative domains. At the handoff between domains there are rules to map
> one SP CoS definition to the one for the next domain. The presence of these
> fields is useful because there is no need to be aware about customer CoS
> definitions to map to the next CoS domain: i.e. if I am domain 2 network admin
> I just need to understand domain 1 CoS definition not how their individual
> customers marked their CoS. Also there is no need to dig on the next encap
> level.
>

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 28 Jun 2012 15:14:25 -0500

We can consider changing in the next version SHOULD to a MAY if that is the WG consensus. Would like to hear though on the list or during the session in Vancouver from more people, ideally from a cloud provider environment. 

> From: [email-masked] [mailto:[email-masked]]
> Sent: Thursday, June 28, 2012 12:34 PM
> To: Balus, Florin Stelian (Florin); [email-masked]
> I think I see where our views diverge.  If one considers a service
> provider environment running a VPN technology that supports both
> service and underlay CoS, then one has three CoS layers, especially
> when the underlay CoS field has expressibility limits (e.g., MPLS
> support for DiffServ is constrained by the 3-bit size of the EXP field
> and other competing uses for those bits).
> 
> OTOH, there are plenty of other environments for nvo3 overlays where
> there aren't three CoS layers and there aren't serious expressibility
> problems in the outermost CoS field (e.g., the DSCP field in the IP
> header imposes no limits on DiffServ expressibility beyond those in the
> basic DiffServ architecture).  Enterprise environments are among those
> that provide examples.
> IPsec VPNs also provide examples under the assumption that the tunnels
> are end-to-end wrt boundaries between service provider networks.
> 
> So while I agree that there are environments in which the CoS field can
> be "useful", that's as far as I would go.  Specifically, I don't think
> the across-the-board "SHOULD" in the draft is justifiable or
> appropriate in general.
> 
> Thanks,
> --David
> 
> > Of Balus, Florin Stelian (Florin)
> > - tunnel/core CoS visible in the NVE and Core Nodes (e.g. 3 exp bits
> > in the tunnel label/dot1p bits in IEEE BVLAN)
> >
> > presence of these fields is useful because there is no need to be
> > aware about customer CoS definitions to map to the next CoS domain:
> > i.e. if I am domain 2 network admin I just need to understand domain
> 1
> > CoS definition not how their individual customers marked their CoS.
> > Also there is no need to dig on the next encap level.
> >
> > > Why "inside the NVO3 overlay header"?  Earlier in the section, IP
> > > DSCP and Ethernet 802.1p are mentioned as examples of CoS (Class of
> > > David L. Black, Distinguished Engineer EMC Corporation, 176 South
> > > St., Hopkinton, MA  01748

---

From: <[email-masked]>
Date: Thu, 28 Jun 2012 16:46:07 -0400

Ok, but NVO3 is applicable to (and the overlays will be deployed in)
a lot more than just cloud provider environments.

Thanks,
--David

> Sent: Thursday, June 28, 2012 4:14 PM
> We can consider changing in the next version SHOULD to a MAY if that is the WG
> consensus. Would like to hear though on the list or during the session in
> Vancouver from more people, ideally from a cloud provider environment.
>

---

From: David Freedman <[email-masked]>
Date: Thu, 28 Jun 2012 21:38:24 +0000

Wearing a cloud provider hat, I have to agree with David,

Whilst I can think of a number of uses for marking
the overlay, I wouldn't want to limit us to not
being able to do customer or core controlled service
CoS should we want to.

Dave.

On 28/06/2012 21:46, "[email-masked]" <[email-masked]> wrote:

>> Subject: Re: [nvo3] NVO3 Data Plane Requirements: QoS identifier in
>>overlay
>> header?
>> 
>>

---

From: "Balus, Florin Stelian (Florin)" <[email-masked]>
Date: Thu, 28 Jun 2012 19:46:30 -0500

Dave,
Just to make sure the requirement is understood correctly I added some clarification below...

> David Freedman
> Sent: Thursday, June 28, 2012 2:38 PM
> To: [email-masked]
> Whilst I can think of a number of uses for marking the overlay, I
> wouldn't want to limit us to not being able to do customer or core
> controlled service CoS should we want to.

[FB>] The current text in the document does not exclude the customer or core controlled service CoS. We are talking about a CoS field in the service tag that may be used for marking/transporting the service CoS. One may choose at every hop the header field used in the packet header to derive the CoS treatment: i.e. the tunnel header, the service tag or even the header from the customer payload - see my initial text for use cases. 

The requirement is tagged as a SHOULD not as a MUST anyhow. 

> >Ok, but NVO3 is applicable to (and the overlays will be deployed in) a
> >lot more than just cloud provider environments.
> >
> >Thanks,
> >--David
> >
> >> We can consider changing in the next version SHOULD to a MAY if that
> >>is the WG  consensus. Would like to hear though on the list or during
> >>the session in  Vancouver from more people, ideally from a cloud
> >>provider environment.
> >>
> >> > support for DiffServ is constrained by the 3-bit size of the EXP
> >> > field and other competing uses for those bits).
> >> >
> >> > OTOH, there are plenty of other environments for nvo3 overlays
> >> > where there aren't three CoS layers and there aren't serious
> >> > expressibility problems in the outermost CoS field (e.g., the DSCP
> >> > field in the IP header imposes no limits on DiffServ
> expressibility
> >> > beyond those in
> >>the
> >> > basic DiffServ architecture).  Enterprise environments are among
> >> > those that provide examples.
> >> > IPsec VPNs also provide examples under the assumption that the
> >> > tunnels are end-to-end wrt boundaries between service provider
> networks.
> >> >
> >> > So while I agree that there are environments in which the CoS
> field
> >>can
> >> > be "useful", that's as far as I would go.  Specifically, I don't
> >> > think the across-the-board "SHOULD" in the draft is justifiable or
> >> > appropriate in general.
> >> >
> >> > Thanks,
> >> > --David
> >> >
> >> > > In section 3.3.1.2 the text above the paragraph you mention
> below
> >> > > is describing why this is required. Let me try a practical
> explanation.
> >> > > When using different overlay/VPN technologies Service Providers
> >> > > have available CoS indicators at 3 levels:
> >> > > - customer/tenant controlled CoS (e.g. DSCP/dot1p bits in the
> >> > > inner
> >> > > payload)
> >> > > - tunnel/core CoS visible in the NVE and Core Nodes (e.g. 3 exp
> >> > > bits in the tunnel label/dot1p bits in IEEE BVLAN)
> >> > >
> >> > > The last two are under the control of the Service Provider and
> >> > > may
> >>be
> >> > > in the NVE/PEs while only 4 may be used in the core. This could
> >> > > be because of available queuing/different
> oversubscription/number
> >> > > of flows/operational requirements in different network
> >> > > devices/admin
> >> > domains.
> >> > >
> >> > > CoS usage may change as the packet transitions between different
> >> > > SP administrative domains. At the handoff between domains there
> >> > > are
> >> > rules
> >> > > > In draft-bl-nvo3-dataplane-requirements-00.txt, the last
> >> > > > paragraph in Section 3.3.1.2. Service QoS identifier says:
> >> > > >
> >> > > >    Support for NVE Service CoS SHOULD be provided through a
> QoS
> >> > field,
> >> > > >    part of the service tag are 802.1p and DE bits in the VLAN
> >> > > > and
> >> > PBB
> >> > > > Why "inside the NVO3 overlay header"?  Earlier in the section,
> >> > > > IP DSCP and Ethernet 802.1p are mentioned as examples of CoS
> >> > > > (Class
> >>of
> >> > > > What's the rationale for an additional QoS field in the
> overlay
> >> > header?
> >> > > >
> >> > > > Thanks,
> >> > > > --David
> >> > > > David L. Black, Distinguished Engineer EMC Corporation, 176
> >> > > > South St., Hopkinton, MA  01748