Subject: [Int-area] draft-williams-overlaypath-ip-tcp-rfc

From: Brandon Williams <[email-masked]>
Date: Thu, 20 Dec 2012 12:20:01 -0500

Dear all,

A new version of this draft has been submitted that attempts to lay out 
a more clear argument for the use of both TCP and IP options, with 
references to other efforts in the same arena.

Comments are welcome.

Cheers,
--Brandon

-------- Original Message --------
Subject: New Version Notification for 
draft-williams-overlaypath-ip-tcp-rfc-03.txt
Date: Thu, 20 Dec 2012 11:55:33 -0500
From: [email-masked] <[email-masked]>
To: Williams, Brandon <[email-masked]>

A new version of I-D, draft-williams-overlaypath-ip-tcp-rfc-03.txt
has been successfully submitted by Brandon Williams and posted to the
IETF repository.

Filename:	 draft-williams-overlaypath-ip-tcp-rfc
Revision:	 03
Title:		 Overlay Path Option for IP and TCP
Creation date:	 2012-12-20
WG ID:		 Individual Submission
Number of pages: 15
URL: 
http://www.ietf.org/internet-drafts/draft-williams-overlaypath-ip-tcp-rfc-03.txt
Status: 
http://datatracker.ietf.org/doc/draft-williams-overlaypath-ip-tcp-rfc
Htmlized: 
http://tools.ietf.org/html/draft-williams-overlaypath-ip-tcp-rfc-03
Diff: 
http://www.ietf.org/rfcdiff?url2=draft-williams-overlaypath-ip-tcp-rfc-03

Abstract:
    Data transport through overlay networks often uses either connection
    termination or network address translation (NAT) in such a way that
    the public IP addresses of the true endpoint machines involved in the
    data transport are invisible to each other.  This document describes
    IPv4, IPv6, and TCP options for communicating this information from
    the overlay network to the endpoint machines.

The IETF Secretariat

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 20 Dec 2012 14:16:12 -0500

> references to other efforts in the same arena.

(note, I've cross-posted to INTAREA and TCPM, since similar
announcements went to each list)

Hi Brandon, *many* thanks for writing this; it does help me (at least)
to understand what you're doing with this option.

As I now understand it, instead of a tunneling approach that would
normally be applied for building overlay networks, this approach
pushes and pops IP addresses from the protocol options fields.

Can you discuss why normal tunneling protocols aren't used to build
the overlay?

Since those are easily and widely available, I wonder why they aren't
used and why something more elaborate, fragile, and not as compatible
with the Internet architecture is really needed or felt to be a good
idea?  I understand that it basically *works* ... but just am not
seeing how it makes sense?

-- 
Wes Eddy
MTI Systems

---

From: Brandon Williams <[email-masked]>
Date: Thu, 20 Dec 2012 15:49:27 -0500

Hi Wes,

Thanks for your comments.

It looks like I might have managed to make the use of the proposed 
option less clear, instead of more clear. Or maybe I'm misunderstanding 
the point that you're making.

The mechanics of our system are tunnel-based, as with most overlay 
architectures that I've looked at. The tunneling starts at an overlay 
ingress machine close to one of the endpoints (i.e. the client or 
server) and ends at an overlay egress machine close to the other 
endpoint. Since the ingress and egress are on the public internet, the 
overlay does not extend all the way onto the endpoints' LANs. This means 
that standard internet routing cannot be used to drive the connections 
into the overlay. Instead, NAT is used on both sides of the overlay, 
which results in the server having no way to reliably identify the client.

The proposed options are not intended to be used as part of the 
mechanics of the overlay. The overlay is fully functional without the 
options. Instead, the options are intended to provide the client's 
connection identifying information to the server for use in 
load-balancing, diagnostics, etc.

Does this clarify things? further muddy the waters? or simply indicate 
that I missed your point?

--Brandon

PS: Thanks for cross-posting your comments. I should have done that to 
begin with. I primarily posted to TCPM for informational purposes, since 
the TCPM has not shown much interest in this or similar drafts in the 
past. The INTAREA list has been more actively engaged in discussion 
related to client identification. Still, if I was going to cross-post, I 
should have done it with a single thread.

-- 
Brandon Williams; Principal Software Engineer
Cloud Engineering; Akamai Technologies Inc.

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 20 Dec 2012 16:04:57 -0500

> load-balancing, diagnostics, etc.

Ah, so are there additional devices beyond what's shown in your Figure
1?  I ask because if the overlay ingress and egress are simple tunnel
endpoints, then the endpoint addresses would be totally visible to one
another.

Your figure 1 is:

                  +------------------------------------+
                  |                                    |
                  |              INTERNET              |
                  |                                    |
   +-----------+  |  +------------+                    |
   |  HOST_1   |-----| OVRLY_IN_1 |-----------+        |
   +-----------+  |  +------------+           |        |
                  |                           |        |
   +-----------+  |  +------------+     +-----------+  |  +--------+
   |  HOST_2   |-----| OVRLY_IN_2 |-----| OVRLY_OUT |-----| SERVER |
   +-----------+  |  +------------+     +-----------+  |  +--------+
                  |                           |        |
   +-----------+  |  +------------+           |        |
   |  HOST_3   |-----| OVRLY_IN_3 |-----------+        |
   +-----------+  |  +------------+                    |
                  |                                    |
                  +------------------------------------+

                                 Figure 1

If there were tunnels between the OVRLY_IN and OVERLY_OUT boxes,
then the inner IP headers would have the HOST_X and SERVER
addresses, and the outer ones in the tunnel would have the
overlay headers.  Since the inner packets would be delivered
ultimately after egressing the tunnels, the HOST_X addresses
are totally visible to the server, and vice versa.

Your document shows instead:

   ---------------------------------------------------------------------

             ip hdr contains:               ip hdr contains:
   SENDER -> src = sender   --> OVERLAY --> src = overlay2  --> RECEIVER
             dst = overlay1                 dst = receiver

   ---------------------------------------------------------------------

So, this is not really showing tunnels to me ... this is rewriting
(NAT) of the destination address.

Or am I misunderstanding?

-- 
Wes Eddy
MTI Systems

---

From: Brandon Williams <[email-masked]>
Date: Thu, 20 Dec 2012 17:04:48 -0500

> endpoints, then the endpoint addresses would be totally visible to one

Yes. There are additional devices between the HOST and OVRLY_IN, and 
also between OVRLY_OUT and SERVER, but those devices are just the 
internet's standard routing infrastructure. There are also potential 
intermediate devices between OVRLY_IN and OVRLY_OUT that can be used for 
optimized routing between the overlay's ingress and egress.

> Your figure 1 is:

There are indeed tunnels between OVRLY_IN and OVRLY_OUT, and the inner 
IP headers will typically use either the client-side addresses or the 
server-side addresses. However, neither OVRLY_IN nor OVRLY_OUT can be 
assumed to be reliably in-path between HOST and SERVER, which means that 
internet routing cannot be relied upon to cause packets to arrive at the 
overlay ingress. Instead, HOST_1 must directly address OVRLY_IN_1 in 
order to send its packets into the tunnel, and SERVER must directly 
address OVRLY_OUT in order to send the return traffic into the tunnel.

> Your document shows instead:

As noted above, the use of tunnels and NAT in this case are not mutually 
exclusive. NAT is used to allow the overlay ingress to intercept 
packets, which are then tunneled to the overlay egress, where NAT is 
used to deliver the packets to the receiver and ensure that return 
traffic also uses the overlay.

--Brandon

PS: Sorry to double send this to you Wes. It was bounced by the ietf 
lists the first time.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 21 Dec 2012 14:34:43 +0100

Brandon, 

> > If there were tunnels between the OVRLY_IN and OVERLY_OUT 
> boxes, then 
> > the inner IP headers would have the HOST_X and SERVER 
> addresses, and 
> > the outer ones in the tunnel would have the overlay headers.  Since 
> > the inner packets would be delivered ultimately after egressing the 
> > tunnels, the HOST_X addresses are totally visible to the 
> server, and 
> > vice versa.
> 
> There are indeed tunnels between OVRLY_IN and OVRLY_OUT, and 
> the inner IP headers will typically use either the 
> client-side addresses or the server-side addresses. However, 
> neither OVRLY_IN nor OVRLY_OUT can be assumed to be reliably 
> in-path between HOST and SERVER, which means that internet 
> routing cannot be relied upon to cause packets to arrive at 
> the overlay ingress. Instead, HOST_1 must directly address 
> OVRLY_IN_1 in order to send its packets into the tunnel, and 
> SERVER must directly address OVRLY_OUT in order to send the 
> return traffic into the tunnel.

Thanks for this explanation - this indeed helps to understand the architecture. But actually I still don't fully understand the motivation of bypassing Internet routing this way. As a non-expert on routing, it indeed looks to me like reinventing source routing - but this is outside my core expertise.

Regarding TCPM's business: If I correctly understand the approach, OVRLY_IN will "transparently" add and remove TCP options. This is kind of dangerous from an end-to-end perspective... Sorry if that has been answered before, but I really wonder what to do if OVRLY_IN can't add this option, either because of lack of TCP option space, or because the path MTU is exceeded by the resulting IP packet. (In fact, I think that this problem does not apply to TCP options only.)

Unless I miss something, the latter case could become much more relevant soon: TCPM currently works on the fast-open scheme that adds data to SYNs. With that, I think it is possible that all data packets from a sender to a receiver are either full sized or large enough that the proposed option does not fit in. Given that this option can include full-sized IPv6 addresses, this likelihood is much larger than for other existing TCP option, right?

In some cases, I believe that the proposed TCP option cannot be added in the overlay without either IP fragmentation, which is unlikely to be a good idea with NATs, or TCP segment splitting, which probably can cause harm as well. For instance, what would OVRLY_IN do if it receives an IP packet with a TCP SYN segment that already sums up to 1500 byte? And, to make the scenario more nasty, if the same applies to the first data segments as well?

Thanks

Michael

---

From: Brandon Williams <[email-masked]>
Date: Fri, 21 Dec 2012 11:31:45 -0500

Hi Michael,

Your are correct that the option could be problematic if added to a 
full-sized packet, or even a nearly full one. I can see that the 
document should have some discussion of this issue.

In a case like ours, where the overlay network uses tunneling, 
transparently adding the option is not a critical problem to be solved. 
It is already the case that the overlay entry point must advertise a 
reduced MSS in order to accommodate the tunnel overhead. The amount of 
space consumed by the option will always be smaller than the tunnel 
overhead, and the option can be added at OVRLY_OUT, so the two are not 
additive. That said, I can see that an overlay network that does not use 
tunnels internally, or one that in fact does apply the option on 
OVRLY_IN, would have a bigger problem, though.

The issue of the proposed fast-open scheme is one that we have not 
considered, but I don't think it adds any problems for the TCP option 
that aren't already a problem for tunneled connectivity in general. I 
will have to spend some time with that proposal and think about how they 
interrelate.

--Brandon

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 21 Dec 2012 20:14:10 +0100

Brandon, 

> Your are correct that the option could be problematic if 
> added to a full-sized packet, or even a nearly full one. I 
> can see that the document should have some discussion of this issue.

Yes.

> In a case like ours, where the overlay network uses 
> tunneling, transparently adding the option is not a critical 
> problem to be solved. 
> It is already the case that the overlay entry point must 
> advertise a reduced MSS in order to accommodate the tunnel 
> overhead. The amount of space consumed by the option will 
> always be smaller than the tunnel overhead, and the option 
> can be added at OVRLY_OUT, so the two are not additive. That 
> said, I can see that an overlay network that does not use 
> tunnels internally, or one that in fact does apply the option 
> on OVRLY_IN, would have a bigger problem, though.

So, the new TCP option is basically required between OVRLY_OUT and the receiver/server, because the relevant information is already somehow transported in the overlay, right?

This raises another question (sorry if it is naive): Why can't the overlay tunnel just be extended to the server? This somehow implies that OVRLY_OUT would be kind of co-located with the server - obviously, there can be further routers/overlay nodes in between.

I am asking this because processing the information contained in the TCP option will require anyway a modified TCP stack in the server, i. e., the server will not be fully backward compatible if it has to process the proposed option. But if the TCP/IP stack has to be modified anyway, I could imagine that one could just add to the server whatever encap/decap is required for the overlay transport. Then, I have the impression that the proposed TCP option would not be needed at all.

I don't want to dig into the overlay design, because this is not really in scope of TCPM. But if there is a system architecture that does not require adding TCP options in middleboxes, thus affecting TCP end-to-end semantics, it would really be important to understand why such an architecture cannot be used.

Thanks

Michael

> The issue of the proposed fast-open scheme is one that we 
> have not considered, but I don't think it adds any problems 
> for the TCP option that aren't already a problem for tunneled 
> connectivity in general. I will have to spend some time with 
> that proposal and think about how they interrelate.
> 
> --Brandon
> 
> >> There are indeed tunnels between OVRLY_IN and OVRLY_OUT, and the 
> >> inner IP headers will typically use either the client-side 
> addresses 
> >> or the server-side addresses. However, neither OVRLY_IN 
> nor OVRLY_OUT 
> >> can be assumed to be reliably in-path between HOST and 
> SERVER, which 
> >> means that internet routing cannot be relied upon to cause 
> packets to 
> >> arrive at the overlay ingress. Instead, HOST_1 must 
> directly address
> >> OVRLY_IN_1 in order to send its packets into the tunnel, 
> and SERVER 
> >> must directly address OVRLY_OUT in order to send the 
> return traffic 
> >> into the tunnel.
> >
> > Thanks for this explanation - this indeed helps to 
> understand the architecture. But actually I still don't fully 
> understand the motivation of bypassing Internet routing this 
> way. As a non-expert on routing, it indeed looks to me like 
> reinventing source routing - but this is outside my core expertise.
> >
> > Regarding TCPM's business: If I correctly understand the approach, 
> > OVRLY_IN will "transparently" add and remove TCP options. 
> This is kind 
> > of dangerous from an end-to-end perspective... Sorry if 
> that has been 
> > answered before, but I really wonder what to do if OVRLY_IN 
> can't add 
> > this option, either because of lack of TCP option space, or because 
> > the path MTU is exceeded by the resulting IP packet. (In 
> fact, I think 
> > that this problem does not apply to TCP options only.)
> >
> > Unless I miss something, the latter case could become much 
> more relevant soon: TCPM currently works on the fast-open 
> scheme that adds data to SYNs. With that, I think it is 
> possible that all data packets from a sender to a receiver 
> are either full sized or large enough that the proposed 
> option does not fit in. Given that this option can include 
> full-sized IPv6 addresses, this likelihood is much larger 
> than for other existing TCP option, right?
> >
> > In some cases, I believe that the proposed TCP option 
> cannot be added in the overlay without either IP 
> fragmentation, which is unlikely to be a good idea with NATs, 
> or TCP segment splitting, which probably can cause harm as 
> well. For instance, what would OVRLY_IN do if it receives an 
> IP packet with a TCP SYN segment that already sums up to 1500 
> byte? And, to make the scenario more nasty, if the same 
> applies to the first data segments as well?
> >
> > Thanks
> >
> > Michael
> >
> 
> --
> Brandon Williams; Principal Software Engineer Cloud 
> Engineering; Akamai Technologies Inc.
>

---

From: Brandon Williams <[email-masked]>
Date: Fri, 21 Dec 2012 15:07:19 -0500

Michael,

Extending the overlay all the way to the application server would mean 
that existing solutions for load balancing, SSL offload, intrusion 
detection, diagnostic logging, etc. would not work. In other words, 
there are many systems in a common enterprise environment that would 
benefit from more accurate host identification, and all would require 
changes in order for the mechanism to work. On the other hand, there is 
existing middleware that can already handle an arbitrary tcp option, 
using its value for the above listed purposes. So using a TCP option for 
this purpose is deployable today, but extending the overlay is not.

At the same time, use of the option does not carry significant risk of 
breaking existing connectivity, even in cases where the option is not 
understood by the TCP stack. Testing has shown that only about 1.7% of 
the top 100,000 web servers fail to establish connections when the 
option is included (see draft-abdo-hostid-tcpopt-implementation). This 
is mostly likely a characteristic of the common TCP stacks in use today, 
and so probably extends to non-HTTP application servers, too.

--Brandon