Subject: [Rtg-dt-encap-considerations] Some comments on draft

From: Tom Herbert <[email-masked]>
Date: Sat, 7 Mar 2015 15:59:37 -0800

Abstract:

in the case when->in the case that

Section 2

The encapsulation protocols typically->Encapsulation protocols typically

And NIC hardware->NIC hardware

Section 4

2nd bullet is a lttle wordy. I think this is saying we assumed mostly
stateless encapsulation protocol

Section 5

The goal to->The goal is to

the need to be aware->they need to be aware

The section on IPv6 flow label is still a little conservative I think.
We already have RFC 6438 that described use of flow labels with
tunnels. Also, this might be better positioned as a way to provide
additional entropy (14 bits for tunnels is going to be a little light
in some contexts).

Section 8

some way to indicate which encapsulation header will follow... not
clear. This saying: some way to indicate next protocol?

Bullet 2. I don't understand the part about "bringing in protocol
numbers that don't make sense".

Section 10

A single marker in the encaps... there was discussion in nvo3 such a
plan and it had some problems By using a single bit the epochs are
forced to be rather coarse. Other problems with part about using a s
single bit to do 1-way latency. I think this paragraph should be
removed or expanded with more detail.

Section 11: Still needs to be cleaned up. I will try to do that.

Section 13: Data centers may be very highly optimized for their own
internal traffic by assuming that all internal traffic in under a
specific congestion control. That hash allowed us to size and carve
buffers in very specific ways. With cloud, we introduce 3rd party
stacks, that may implement CC, but not one which is conformant with
internal traffic. This is creating problems and may be the motivation
for true cc at encap layer

Last paragraph: Rate limiting is not the same thing as congestion
control (unless there's very active management of rate)

Section 14

There is little value in the portion of the UDP checksum... true in
terms of being redundant to an encapsulated checksum, however there is
value in leveraging the outer checksum to validate inner one using
common NIC offload features.

For description of partial checksum maybe add: This checksum would
cover the encapsulation header and a pseudo header that includes at
least the IP addresses and ports.

For NAT problem this can be solved with an additional field that holds
the checksum of the original IP addresses.

Section 17

Once again the idea of spraying packets of single flows has been brought up:
https://www.cs.purdue.edu/homes/kompella/publications/infocom13.pdf

It's probably more correct to say that TCP is optimized for in order
delivery, but can handle ooo. Some other protocols (non-IP) might have
much more difficulty.

Section 18

Keep encap headers small-- can we be more specific on what small means?

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 09 Mar 2015 10:40:42 -0700

Thanks for the careful review. I've applied the edits except as noted below.

> stateless encapsulation protocol

"stateless" means different things to different readers.

But it can be made shorter. How about:
<t>Focus on the class of encapsulations that would run over IP/UDP. That 
was done to avoid being distracted by the data-plane and control-plane 
interaction, which more significant for protocols that are designed to 
run over "transports" that maintain session or path state.</t>

> additional entropy (14 bits for tunnels is going to be a little light

I reworded it after the call on Thursday. The last sentences say:
Currently the leaning is towards recommending using the UDP encaps for 
both IPv4 and IPv6 underlay. The IPv6 flow label can be used for 
additional entropy if need be.

> clear. This saying: some way to indicate next protocol?
It is really about the encaps, but the preceeding header could also 
carry non-encaps.
How about wording it as:
The transport delivery mechanism for the encapsulations we discuss in 
this document need some way to indicate which encapsulation header (or 
other payload) comes next in the packet.

> numbers that don't make sense".

It is trying to capture the questions like would it make sense to run 
OSPF, ICMPv6, etc directly on top of the encaps.

Is it more clear to say:
"that would never be used directly on top of the encaps protocol"?

> removed or expanded with more detail.
I'm puzzled because earlier versions suggested encaps reserving room for 
multiple bits and you convinced me that didn't make any sense. Now you 
seem to be saying that one bit isn't sufficient and there has to be 
multiple bits.

I'll preface it and reword as "There has been suggestions that one (or 
more) marker...".

> for true cc at encap layer
What edit are you suggesting?

> control (unless there's very active management of rate)
What edit are you suggesting?
> common NIC offload features.
What edit are you suggesting?
This will be mentioned in the hardware friendly offload text that you 
provided.

> least the IP addresses and ports.
Doesn't "the same partial-checksum concept as UDP-Lite" already say that?

> the checksum of the original IP addresses.

I don't think we need to propose solutions here. Pointing out the issue 
should be sufficient.
> delivery, but can handle ooo. Some other protocols (non-IP) might have
I don't see where that text would fit into the current text.
> Keep encap headers small-- can we be more specific on what small means?
42 bytes?  ;-)

But perhaps we don't understand the question yet.

    Erik

>
> _______________________________________________
> Rtg-dt-encap-considerations mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtg-dt-encap-considerations
>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 9 Mar 2015 11:37:14 -0700

On Mon, Mar 9, 2015 at 10:40 AM, Erik Nordmark <[email-masked]> wrote:
> <t>Focus on the class of encapsulations that would run over IP/UDP. That was
> done to avoid being distracted by the data-plane and control-plane
> interaction, which more significant for protocols that are designed to run
> over "transports" that maintain session or path state.</t>
>
Better. "which is more significant"

> Currently the leaning is towards recommending using the UDP encaps for both
> IPv4 and IPv6 underlay. The IPv6 flow label can be used for additional
> entropy if need be.
>
Okay. Generally, I do think we're going to want to use the flow label
for something within DCs, 20 bits is just too much opportunity to
waste.

> It is really about the encaps, but the preceeding header could also carry
> non-encaps.
> How about wording it as:
> The transport delivery mechanism for the encapsulations we discuss in this
> document need some way to indicate which encapsulation header (or other
> payload) comes next in the packet.
>
Maybe just generically call it the type of the payload?

> It is trying to capture the questions like would it make sense to run OSPF,
> ICMPv6, etc directly on top of the encaps.
>
> Is it more clear to say:
I think there is a difference between something that doesn't make
sense and something that doesn't work or is broken. I don't see
anything fundamentally broken about encapsulation of ICMPv6 over UDP,
this is already supported in foo-over-UDP for Linux. Maybe say "would
not normally be encapsulated"...

> multiple bits and you convinced me that didn't make any sense. Now you seem
> to be saying that one bit isn't sufficient and there has to be multiple
> bits.
>
I believe my suggestion was to allocate a single bit in GUE that
provides a field that could hold generation numbers for more
granularity in an epoch. The OAM proposal on the nvo3 list defined the
algorithm with only the two bits of input to handle both the lost
counters and 1-way delay. The constraints of only using bits puts
constraints on the algorithm and the quality/granularity of
measurement.

> I'll preface it and reword as "There has been suggestions that one (or more)
> marker...".
>
Okay.

> What edit are you suggesting?
Just pointing out that circuit breaker may no be sufficient in some instances.

> What edit are you suggesting?

Same as above, rate limiting as a means of congestion control may not
be sufficient in some cases.

> What edit are you suggesting?

The above wording.

> This will be mentioned in the hardware friendly offload text that you
Okay.

> Doesn't "the same partial-checksum concept as UDP-Lite" already say that?
I am assuming the the header checksum does not automatically include
the whole UDP header, so the UDP ports need to be part of the pseudo
header checksum (e.g. draft-herbert-guecsum-00)
> I don't think we need to propose solutions here. Pointing out the issue

Okay

> I don't see where that text would fit into the current text.

No, but this may come up at some point as the counter point to all the
efforts to strictly maintain compatibility with ECMP.

>> Keep encap headers small-- can we be more specific on what small means?
I was hoping for 128 bytes :-)

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 09 Mar 2015 12:58:20 -0700

> Better. "which is more significant"

Oops. Fixed.

> Maybe just generically call it the type of the payload?

Defer for now. The intro should perhaps explain better up front the 
different places where "next" comes into play.
Minor edits won't do that.

> not normally be encapsulated"...

The point (which might not come out in the draft) is that using 8 bits, 
but having lots (100? haven't checked) values already allocated where 
most of those are not likely to ever be used for encaps is a waste of 
bits. I think that was what we talked about.
(Doesn't mean it is a show-stopper; just one more thing to consider.)

> Just pointing out that circuit breaker may no be sufficient in some instances.

That doesn't look like an edit; by edit I mean text I can copy in paste 
into a well-defined place in the document.
(Yes, it is only a few hours left and I have some meetings to take care 
of as well.)

Same comment applies to your comments below. Not actionable by me at 
this point in time.

Thanks,

    Erik

> header checksum (e.g. draft-herbert-guecsum-00)
---