Subject: Re: [nvo3] I-D Action: draft-xia-nvo3-vxlan-qosmarking-01.txt

From: Behcet Sarikaya <[email-masked]>
Date: Tue, 16 Dec 2014 11:10:43 -0600

On Thu, Nov 13, 2014 at 9:06 PM, Erik Nordmark <[email-masked]> wrote:
>
>  On 11/13/14 4:00 PM, Behcet Sarikaya wrote:
>
>
> On Thu, Nov 13, 2014 at 4:47 PM, Benson Schliesser <[email-masked]>
> wrote:
>
>> Hi, Behcet -
>>
>> Stepping back from the conversation about bits... What is the problem
>> that you're trying to solve, Behcet?
>>
>> I see multiple existing QoS mechanisms both in the underlay and in the
>> overlay, and I don't see any QoS gap that needs to be addressed in the
>> overlap encap layer. I believe that my point of view is consistent with the
>> WG consensus at this point.
>>
>>
> I am not familiar with any QoS mechanism that is based on the tenant, i.e
> static mapping.
> Let me know which document discusses it?
>
>
> Google search points me at rfc2983, rfc6040; latter is for ECN.
>
> There might be other RFCs.
>

Sorry for this belated reply.

I agree that there doesn't seem to be another document on tenant-based QoS.
I read RFC 2983, certainly it is not.

This is possibly because multi tenancy is a new concept in IETF introduced
by nvo3.

Brian Carpenter once suggested to discuss this draft in tcpm, maybe this
was the reason?

We are ready to present it in tcpm and discuss this concept with QoS
experts in tcpm.

Having said that I don't take this comment as negative. I think it is a
valid point.

Regards,

Behcet

>
>
>    Erik
>
>
>
> Thx,
>
> Behcet
>
>>  Thanks,
>> -Benson
>>
>>    Dino Farinacci <[email-masked]>
>>  November 13, 2014 at 12:02 PM
>>
>> Sorry there are no EXP bits mentioned in RFC 7348. MPLS is out of scope.
>>
>> EXP is 3 bits long, DSCP is 6 bits and dividing it into two 3 bit
>> pieces, I am not sure if David will like it.
>>
>>
>>  I am referring to user-priority bits below:
>>
>>
>>  Dino
>>
>>     Benson Schliesser <[email-masked]>
>>  November 12, 2014 at 9:34 AM
>>    Hi, Behcet -
>>
>> Perhaps I'm confused about what comment (from Dino) that you are
>> referring to... But in general, I think of it this way:
>>
>> Assuming the encap stack looks something like: IP1 / Eth1 / VXLAN / UDP /
>> IP2 / Eth2  (progressing L->R as inner->outer)
>>
>> Then e.g. tenant VMs can mark the IP1 and Eth1 headers with whatever
>> appropriate markings they desire. The NVE can mark the IP2 and Eth2 headers
>> with whatever appropriate markings.
>>
>> Specifically, one could imagine the NVE copying the IP1 DSCP codepoint
>> into the IP2 header. Alternatively one could imagine the NVE imposing an
>> underlay DSCP in IP2, e.g. to discriminate between tenants. Possibly, one
>> could also imagine some kind of translation policy which maps IP1
>> codepoints into IP2 codepoints. And that's not even considering mechanisms
>> that leverage the Eth headers, use different encap stacks, etc.
>>
>> Cheers,
>> -Benson
>>
>>    Behcet Sarikaya <[email-masked]>
>>  November 12, 2014 at 9:01 AM
>>   Hi Dino,
>>
>> Regarding your comment on copying IP header QoS bits into VXLAN header,
>>
>> note that IP packet is coming from the VMs.
>>
>> Yes for dynamic marking these bits can be copied.
>> However, VMs may not be configured to mark these fields.
>>
>> For static marking these bits can not be used because VMs are not
>> aware of the VNI. So NVE has to do the static marking.
>>
>> Hope this clarifies.
>>
>> Regards,
>>
>> Behcet
>>
>> _______________________________________________
>> nvo3 mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/nvo3
>>    Behcet Sarikaya <[email-masked]>
>>  November 10, 2014 at 5:47 PM
>>
>> On Mon, Nov 10, 2014 at 9:41 PM, Brian E Carpenter<[email-masked]> <[email-masked]> wrote:
>>
>>  [resend with corrected address, sorry]
>>
>> Hi,
>>
>>
>>   The first three bits (bits 5-7) are precedence bits. They are
>>  assigned according to [RFC0791]. Precedence values '110' and '111'
>>  are selected for routing traffic.
>>
>>  The last three bits (bits 8-10) are class selector bits. Thet are
>>  assigned as follows:
>>
>> 001 - BK or background traffic
>>
>>  ...
>>
>>  As can be seen the markings are the same as in IEEE 802.1p...
>>
>>  This is not in any way compatible with RFC 2474, which also made the
>> relevant part of RFC 791 obsolete.
>>
>> If you want to be compatible with RFC 2474 you should not specify the
>> bits at all - just say that they are exactly as defined in RFC 2474
>> and the various PHB definitions that have been published.
>>
>>  I think that diffserv is less relevant in the context of VXLAN.
>>
>>
>>   If you
>> want to be compatible with IEEE 802.1p that is a different matter,
>>
>>  Yes this is more relevant for VXLAN.
>>
>>
>>  but you cannot mix the two up in this way.
>>
>>  I now understand that we confused the two very different things.
>>
>> Regards,
>>
>> Behcet
>>
>>      Brian
>>
>>
>>
>>
>>  _______________________________________________
>> nvo3 mailing [email-masked]://www.ietf.org/mailman/listinfo/nvo3
>>
>>
>
>
> _______________________________________________
> nvo3 mailing [email-masked]://www.ietf.org/mailman/listinfo/nvo3
>
>
>

---

From: "Black, David" <[email-masked]>
Date: Tue, 16 Dec 2014 22:26:19 +0000

Hmm, in http://www.ietf.org/mail-archive/web/nvo3/current/msg04211.html,
one of our WG chairs wrote:

At this point, I maintain my view that the NVO3 consensus is: there is no QoS
gap that needs to be addressed in the overlap encap layer.

If NVO3 is not interested in this draft, what’s the purpose of further work on it?

Thanks,
--David

From: nvo3 [mailto:[email-masked]] On Behalf Of Behcet Sarikaya
Sent: Tuesday, December 16, 2014 12:11 PM
To: Erik Nordmark; Brian E Carpenter
Cc: Benson Schliesser; [email-masked]; Dino Farinacci; [email-masked]
Subject: Re: [nvo3] I-D Action: draft-xia-nvo3-vxlan-qosmarking-01.txt

On Thu, Nov 13, 2014 at 9:06 PM, Erik Nordmark <[email-masked]<mailto:[email-masked]>> wrote:
On 11/13/14 4:00 PM, Behcet Sarikaya wrote:

On Thu, Nov 13, 2014 at 4:47 PM, Benson Schliesser <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Behcet -

Stepping back from the conversation about bits... What is the problem that you're trying to solve, Behcet?

I see multiple existing QoS mechanisms both in the underlay and in the overlay, and I don't see any QoS gap that needs to be addressed in the overlap encap layer. I believe that my point of view is consistent with the WG consensus at this point.

I am not familiar with any QoS mechanism that is based on the tenant, i.e static mapping.
Let me know which document discusses it?

I agree that there doesn't seem to be another document on tenant-based QoS. I read RFC 2983, certainly it is not.

This is possibly because multi tenancy is a new concept in IETF introduced by nvo3.

Brian Carpenter once suggested to discuss this draft in tcpm, maybe this was the reason?

We are ready to present it in tcpm and discuss this concept with QoS experts in tcpm.

Having said that I don't take this comment as negative. I think it is a valid point.

Regards,

Behcet

   Erik

Thx,

Behcet
Thanks,
-Benson

Dino Farinacci<mailto:[email-masked]>

Dino
Benson Schliesser<mailto:[email-masked]>
November 12, 2014 at 9:34 AM
Hi, Behcet -

Perhaps I'm confused about what comment (from Dino) that you are referring to... But in general, I think of it this way:

Assuming the encap stack looks something like: IP1 / Eth1 / VXLAN / UDP / IP2 / Eth2  (progressing L->R as inner->outer)

Then e.g. tenant VMs can mark the IP1 and Eth1 headers with whatever appropriate markings they desire. The NVE can mark the IP2 and Eth2 headers with whatever appropriate markings.

Specifically, one could imagine the NVE copying the IP1 DSCP codepoint into the IP2 header. Alternatively one could imagine the NVE imposing an underlay DSCP in IP2, e.g. to discriminate between tenants. Possibly, one could also imagine some kind of translation policy which maps IP1 codepoints into IP2 codepoints. And that's not even considering mechanisms that leverage the Eth headers, use different encap stacks, etc.

Cheers,
-Benson
Behcet Sarikaya<mailto:[email-masked]>
November 12, 2014 at 9:01 AM
Hi Dino,

Hope this clarifies.

Regards,

Behcet

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3
Behcet Sarikaya<mailto:[email-masked]>
November 10, 2014 at 5:47 PM

On Mon, Nov 10, 2014 at 9:41 PM, Brian E Carpenter

<[email-masked]><mailto:[email-masked]> wrote:

Hi,

 assigned as follows:

...

 If you

Regards,

Behcet

    Brian

nvo3 mailing list

[email-masked]<mailto:[email-masked]>

nvo3 mailing list

[email-masked]<mailto:[email-masked]>

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 18 Dec 2014 11:26:27 -0600

Hi Benson,
On Thu, Nov 13, 2014 at 8:08 PM, Benson Schliesser <[email-masked]>
wrote:
>
> Hi, Behcet -
>
> Quoting from my previous message: "one could imagine the NVE imposing an
> underlay DSCP in IP2, e.g. to discriminate between tenants."
>
>
No it is not the issue of coping DSCP from IP1 to IP2 especially in tenant
based QoS.
How can a DC operator apply QoS on each tenant separately when IP1 is
created by a VM that does not know the VNI?

> This seems so obvious to me that I doubt anybody has bothered to write it
> down...
>
> It does seem like we should document a mechanism for configuration of the
> NVE's QoS behavior. (E.g. as part of the NVO3 control plane and/or in a
> YANG model for NVE management) But that's a different topic.
>
> So, back to my question: Is there actually a problem that you trying to
> solve that cannot be solved with the existing mechanisms?
>
>
Yes there is. For example, tenant based QoS can only be done at the time
the NVE receives VM packet because it is the NVE that adds VNI value.

I suggest people to please read Rev. 02 of our draft.

In this version we also address QoS framework in VXLAN or the overlay
network as you call it.

> If so, then I will reconsider my beliefs about WG consensus. But if not,
> then I don't see why we're having this conversation.
>
>
I think this should be reconsidered in view of Rev. 02 and the above
arguments.

Regards,

>  November 13, 2014 at 4:00 PM
>
>  November 13, 2014 at 12:47 PM
>  Hi, Behcet -
>
>  November 12, 2014 at 8:06 PM
>
> Exactly. Thanks Benson.
>
> Dino
> Perhaps I'm confused about what comment (from Dino) that you are referring
> to... But in general, I think of it this way:
>

---

From: Benson Schliesser <[email-masked]>
Date: Thu, 18 Dec 2014 12:59:26 -0500

Behcet Sarikaya wrote:
>
>     Quoting from my previous message: "one could imagine the NVE
>     imposing an underlay DSCP in IP2, e.g. to discriminate between
>     tenants."
>
> No it is not the issue of coping DSCP from IP1 to IP2 especially in 
> tenant based QoS.

The DSCP in the underlay IP header can be copied, imposed, translated, 
etc. The NVE can decide what to put into those DSCP bits.Thus to 
reiterate my previous assertion, which David Black recently quoted: 
there is no QoS gap that needs to be addressed in the overlay encap layer.

By the way, I have looked at the diff between -01 and -02 of this draft 
and I still see no explanation of any problem. On the other hand, I do 
see an arbitrary redefinition of existing QoS mechanisms. As it stands, 
this is not something that NVO3 will pursue.

Unless you can explain a problem that isn't addressed by the existing 
mechanisms there is no reason to continue discussing it.

Thanks,
-Benson

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 25 Dec 2014 11:44:43 -0600

On Thu, Dec 18, 2014 at 11:59 AM, Benson Schliesser
<[email-masked]> wrote:
> The DSCP in the underlay IP header can be copied, imposed, translated, etc.

It seems you did not follow up recent email exchanges.

We are talking about tenant based QoS. The tenant id or VNI is known
by NVE. So NVE has to do the QoS assignment.

> The NVE can decide what to put into those DSCP bits.Thus to reiterate my
> previous assertion, which David Black recently quoted: there is no QoS gap
> that needs to be addressed in the overlay encap layer.

As we said in Rev 02, if tenant based QoS is not used then yes DSCP
bits can be copied from VM's values.
For tenant based QoS, Ethernet frame QoS bits are used, as we
explained in Rev 02.
We also defined a mapping to DSCP bits from Ethernet frame QoS bits
after the encapsulation is removed at the receiving NVE.

>
> By the way, I have looked at the diff between -01 and -02 of this draft and
> I still see no explanation of any problem.

Please read Rev 02. We can make a revision to explain tenant based QoS
better if you like.

> On the other hand, I do see an
> arbitrary redefinition of existing QoS mechanisms. As it stands, this is not
> something that NVO3 will pursue.

Please see above.

> mechanisms there is no reason to continue discussing it.

Two issues we are addressing:
-tenant based QoS
-QoS framework in the overlay network

What more can we do for nvo3 :-)

Happy Holidays.

Behcet