Subject: [Rtg-dt-encap-considerations] Alignment and Ethernet encapsulation

From: Tom Herbert <[email-masked]>
Date: Tue, 20 Sep 2016 09:13:27 -0700

Hello,

This issue came up on the Linux netdev list. The Oracle engineers are
trying to deal with the ramifications of unaligned IP headers (not
aligned to four bytes) when doing Ethernet encapsulation (VXLAN, GRE
with TEB for instance). The problem is that certain CPU architectures,
such as SPARC (and maybe PowerPC?), do not handle unaligned accesses
very well. Every unaligned access causes a trap to software which
kills any semblance of performance. x86 has shift logic so we don't
see any issues there (except for unaligned accesses across a page
boundary).

The Linux stack is written to assume that IP protocols are properly
four byte aligned and in non-encapsulation cases Ethernet headers are
padded to be 16 bytes. The problem seems to be specific when
performing Ethernet header encapsulation and there is no padding of
the inner Ethernet header like would be done in ETHERIP. Either the
outer or the inner IP headers will be misaligned. There does not seem
to be an easy fix for this without rewriting the stack or doing copies
on every packet.

For new encapsulation protocols please consider the effects of IP
header alignment in the presence of Ethernet encapsulation. Defining
Ethernet encapsulation with the two byte padding like in ETHERIP may
help a lot to make implementation of Ethernet encapsulation feasible
on CPU HW.

Thanks,
Tom

---

From: Joe Touch <[email-masked]>
Date: Tue, 20 Sep 2016 10:07:27 -0700

Hi, Tom,

On 9/20/2016 9:13 AM, Tom Herbert wrote:
> ...

IMO, alignment needs to be handled within each encapsulation layer
independently. I don't think it's useful to expect new encapsulation
layers to have to make sure every layer of an encapsulated packet is
aligned - just the first one ought to be sufficient. The rest is the
responsibility of whomever added the other layers already in place.

So yes, it's useful to make sure the encapsulated packet starts on a
boundary that is 4-byte aligned, but the rest *needs to be* someone
else's problem.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 Sep 2016 10:29:09 -0700

> boundary that is 4-byte aligned, but the rest *needs to be* someone
It's the Ethernet payload that we need to be four byte aligned not the
Ethernet header. Just aligning Ethernet header to four bytes is not
useful; that means the Ethernet payload, e.g. an IP packet, won't have
four byte alignment and hence the misery of trying to process the
packet. For the cost of two bytes ETHERIP gets things right in this
regard!

Tom

---

From: Joe Touch <[email-masked]>
Date: Tue, 20 Sep 2016 10:35:41 -0700

> packet. For the cost of two bytes ETHERIP gets things right in this
If you've been handed Ethernet to encapsulate, then that is the header
whose alignment you should be optimizing.

As you point out, you can't align both IP and Ethernet to 4-byte
boundaries at the same time. Aligning the IP header de-aligns the
Ethernet one. Again, if you're encapsulating ethernet, then that is what
you should align to.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 Sep 2016 10:47:54 -0700

On Tue, Sep 20, 2016 at 10:35 AM, Joe Touch <[email-masked]> wrote:
> boundaries at the same time.

You can if you insert a two byte pad before the encapsulated Ethernet
header like ETHERIP (RFC3378 does).

Tom

---

From: Joe Touch <[email-masked]>
Date: Tue, 20 Sep 2016 10:55:59 -0700

> header like ETHERIP (RFC3378 does).

You CAN do a lot of things, but you shouldn't.

If you're handed ethernet to encapsulate, then THAT is the header whose
access should be aligned.

You have no business ASSUMING that you need to optimize for IP header
access. All you're doing is messing up the ethernet access.

Frankly, anything that already parses Ethernet then IP ought to be able
to efficiently handle the difference in the two alignments - and you
might be messing THAT up.

RFC3378 defines a header used for ETHERIP; that header may or may not be
relevant to other ethernet-in-X encapsulations. The field needed to be
there for versioning and was expanded from 8 to 16 bits, which certainly
helps align the *ethernet* header that comes after it compared to the
earlier use of an 8-bit field. There's no mention of aligning for the
purposes of accessing the IP header, which again, I disagree with.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Tue, 20 Sep 2016 11:38:40 -0700

> access. All you're doing is messing up the ethernet access.
Sorry, but I disagree with that statement. Keeping IP headers aligned
for stack processing is a HUGE concern in real implementation.
Probably every device driver on the planet will put in a two byte pad
in receiver buffers to ensure that the Ethernet payload (and hence IP
header, EHs, TCP headers, etc.). We need two byte alignment of
Ethernet headers, and four byte alignment on IP protocols. If we don't
have this a lot of things will break (as VXLAN is currently broken on
SPARC)! Maybe in theory alignment is no longer considered an issue and
is just a historical footnote, but on real devices in deployment today
unaligned protocol headers still can be problematic.

Tom

---

From: Joe Touch <[email-masked]>
Date: Tue, 20 Sep 2016 12:04:50 -0700

> for stack processing is a HUGE concern in real implementation.
It is, and something that handles Ethernet messages should already know
what to do here.

So if it already expects to shift the packet over as it copies it in,
then you've just defeated that. The same is true for non-IP inside
Ethernet, which might have its own padding to re-align its contents.

I.e., if it's ethernet, treat it as ethernet. That's what an L2 tunnel
should do.

Joe