Subject: [tcpm] TCP Connection ID

From: Alexander Azimov <[email-masked]>
Date: Fri, 15 May 2020 18:07:45 +0300

Dear WG,

As you all know L3 LB is a common element of today's cloud infrastructure.
Its original design was based on the connection state that maps backend IPs
(real IPs) to connection. Recently there were several papers that were
suggesting the move to the stateless L3 LB, where the state (connection id)
becomes the property of connection and it kept at the client, so the L3 LB
can work without keeping state for every connection. As far as I
understand, for QUIC it becomes native with destination connection ID in
its header.

I've seen suggestions for getting this functionality in TCP by overloading
the Timestamps option or even TCP SEQ - can't say that such design looks
good for me. I wonder was there an attempt to add connection id in the TCP
options, and if it was - why it wasn't successful?

-- 
Best regards,
Alexander Azimov

---

From: Joseph Touch <[email-masked]>
Date: Sat, 16 May 2020 20:38:58 -0700

RFC7974

> On May 15, 2020, at 8:07 AM, Alexander Azimov <[email-masked]> wrote:
> 
> Dear WG,
> 
> As you all know L3 LB is a common element of today's cloud infrastructure. Its original design was based on the connection state that maps backend IPs (real IPs) to connection. Recently there were several papers that were suggesting the move to the stateless L3 LB, where the state (connection id) becomes the property of connection and it kept at the client, so the L3 LB can work without keeping state for every connection. As far as I understand, for QUIC it becomes native with destination connection ID in its header.
> 
> I've seen suggestions for getting this functionality in TCP by overloading the Timestamps option or even TCP SEQ - can't say that such design looks good for me. I wonder was there an attempt to add connection id in the TCP options, and if it was - why it wasn't successful?
> 
> -- 
> Best regards,
> Alexander Azimov
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Alexander Azimov <[email-masked]>
Date: Wed, 20 May 2020 11:29:03 +0300

Dear colleagues,

I haven't got any reply which increases level my confusion.

If there was no such proposal I will be glad to volunteer to write such a
document. But if you believe that such functionality is not needed in TCP -
I will be glad to learn your opinion in advance.

-- 
Best regards,
Alexander Azimov

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 20 May 2020 11:34:07 +0300

> On 20 May, 2020, at 11:29 am, Alexander Azimov <[email-masked]> wrote:
> 
> If there was no such proposal I will be glad to volunteer to write such a document. But if you believe that such functionality is not needed in TCP - I will be glad to learn your opinion in advance. 

Joe Touch replied with a simple mention of RFC-7974.  Did you not see that?

I would also point to the Flow ID field in the IPv6 header, which is intended to carry this sort of information.

 - Jonathan Morton

---

From: Alexander Azimov <[email-masked]>
Date: Wed, 20 May 2020 16:08:11 +0300

Hi all,

The email from Joe got into the spam box, so I was able to find it only
after your notice. Thank you.

The Host Id presented in RFC7974 is a very different approach: it is
client-side id generated at the moment of the SYN which should help to
distinguish different hosts behind NAT.

What I'm looking for is a connection id, generated at the server-side and
sent back with a SYN/ACK, which is next mirrored by the client. It is not
meant for host identification, but it can be used for connection
identification by the middlebox under the same administrative control. And
there is no need to have a 1:1 map id to a server, instead, each server
should use randomized ids with encryption.

And speaking about IPv6 Flow Label it's not working for IPv4 + it current
implementation in Linux already has great use cases, so even in the world
of IPv6 only I would prefer to keep it without overloading its functions.

---

From: "Scharf, Michael" <[email-masked]>
Date: Wed, 20 May 2020 13:50:52 +0000

Hi Alexander,

Are you aware of RFC 7413 (TCP Fast Open)?

The use case is a different, but RFC 7413 discusses load balancers in Section 6.3.4.

Michael

From: tcpm <[email-masked]> On Behalf Of Alexander Azimov
Sent: Wednesday, May 20, 2020 3:08 PM
To: Jonathan Morton <[email-masked]>
Cc: [email-masked]
Subject: Re: [tcpm] TCP Connection ID

Hi all,

The email from Joe got into the spam box, so I was able to find it only after your notice. Thank you.

The Host Id presented in RFC7974 is a very different approach: it is client-side id generated at the moment of the SYN which should help to distinguish different hosts behind NAT.

What I'm looking for is a connection id, generated at the server-side and sent back with a SYN/ACK, which is next mirrored by the client.. It is not meant for host identification, but it can be used for connection identification by the middlebox under the same administrative control. And there is no need to have a 1:1 map id to a server, instead, each server should use randomized ids with encryption.

And speaking about IPv6 Flow Label it's not working for IPv4 + it current implementation in Linux already has great use cases, so even in the world of IPv6 only I would prefer to keep it without overloading its functions.

---

From: cait <[email-masked]>
Date: Wed, 20 May 2020 12:32:00 -0500

Read the caveats in the RFC that Joe cited. How would a server chosen direct index fair any better than client selected?

 

 
>  
> On May 20, 2020 at 8:08 AM, Alexander Azimov  <[email-masked]>  wrote:
>  
>    
> Hi all, 
>
>  
>  _______________________________________________ tcpm mailing list [email-masked] https://www.ietf.org/mailman/listinfo/tcpm 
>

---

From: Joseph Touch <[email-masked]>
Date: Wed, 20 May 2020 10:52:48 -0700

Right - I think that if RFC7974 doesn’t work for you, you’ll need to provide a much more precise description of what you’re seeking.

From your description below, it’s not a connection ID you want; you really seem to be re-inventing cookies (something that lets the server re-establish state with a client BETWEEN connections, not just within one), and you want that cookie to govern how the middle box works.

Good luck with that. I think any decent security mechanism would (or should) prevent that.

Joe

> On May 20, 2020, at 10:32 AM, cait <[email-masked]> wrote:
> 
>> On May 20, 2020 at 8:08 AM, Alexander Azimov <[email-masked] <mailto:[email-masked]>> wrote: 
>> 
>> Hi all,
>>

---

From: Alexander Azimov <[email-masked]>
Date: Wed, 20 May 2020 21:44:39 +0300

Thank you all for the comments. I will combine my answers all-in-one reply.

Michael:
The mechanics of TFO has significant intersection, but with a certain
difference. The TFO is applied on the client src IP, while the connection
ID should depend on the server IP, which is invisible for the client. And
while TFO is just cookie (you can't retrieve the value since it was
truncated) connection ID should be reversible. And of course, the
Connection Id must be present in all packets except the first SYN.

Cait (I don't see your full name, sorry if I got it wrong):
Can you please clarify your comment? Are you speaking about IESG note? If
yes, I don't see how privacy concerns of host id are applicable in this
case.

Joseph:
If you read my suggestions as something cached between connection you got
it wrong. What I'm trying to discuss is an option similar to DSID in QUIC,
which is put in the unencrypted part of the header for packets sent from
the client to the server. The straightforward way is to apply AES on 128bit
block where the first 64 bits go from high bits server IP (real server IP,
not VIP) + random 64 bits/timestamp. The middlebox (L3 LB), knowing the
secret, will be able to retrieve real IP and make balancing without keeping
the state of the connection. Lack of space in TCP options might be a
problem for such an approach.

ср, 20 мая 2020 г. в 20:52, Joseph Touch <[email-masked]>:

> Right - I think that if RFC7974 doesn’t work for you, you’ll need to
> provide a much more precise description of what you’re seeking.
>
> From your description below, it’s not a connection ID you want; you really
> seem to be re-inventing cookies (something that lets the server
> re-establish state with a client BETWEEN connections, not just within one),
> and you want that cookie to govern how the middle box works.
>
> Read the caveats in the RFC that Joe cited. How would a server chosen
> direct index fair any better than client selected?
>
>
>
> sent back with a SYN/ACK, which is next mirrored by the client.. It is not
> tcpm mailing [email-masked]://www.ietf.org/mailman/listinfo/tcpm
>
>

-- 
Best regards,
Alexander Azimov

---

From: Joseph Touch <[email-masked]>
Date: Wed, 20 May 2020 20:03:13 -0700

> On May 20, 2020, at 11:44 AM, Alexander Azimov <[email-masked]> wrote:
> 
> Joseph: 
> If you read my suggestions as something cached between connection you got it wrong. What I'm trying to discuss is an option similar to DSID in QUIC, which is put in the unencrypted part of the header for packets sent from the client to the server. The straightforward way is to apply AES on 128bit block where the first 64 bits go from high bits server IP (real server IP, not VIP) + random 64 bits/timestamp. The middlebox (L3 LB), knowing the secret, will be able to retrieve real IP and make balancing without keeping the state of the connection. Lack of space in TCP options might be a problem for such an approach.

You can do this by combining the existing TCP Host ID option (RFC 7974) with IP LSR.

You only really need:
	- a TCP option in the SYN and SYN-ACK (8-bytes using the RFC7874 variant) but not in later segments
	- IP LSR at the client
	- a middlebox that rewrites IP headers of SYN packets with LSR

Here’s how:

Servers are assigned two addresses each - one unique (P, Q, R, …), one common (let’s say “X”)
All servers are configured to forward

Clients send SYN to LSR X,X with TCP HID option with an empty value
	You need IP to have LSR because the middlebox should NEVER make a packet longer; it could mess up MTU discovery
	You need the TCP HID option by the rules of that option
LB middlebox picks a server (P) and rewrites the IP LSR X, X with the LSR P,X
	So load balancing happens only on SYNs
The SYN proceeds to server P, which performs LSR; in the process of forwarding, it accepts the packet at X
Server P needs to see the IP LSR P,X and extract P
Server P puts P in the SYN-ACK TCP HID option and sends it back to the client
	now that you receive the TCP HID option back, the client no longer needs to include it in future packets

The client sees P in the SYN-ACK option and configures that TCP connection to send all future packets with IP LSR P,X

At that point the middlebox doesn’t need to do anything anymore; everything is handled by the client, who has to keep state for the connection anyway.

This can work even with end-to-end TCP encryption or authentication. The only thing it needs is an outermost IP header whose LSR can be rewritten.

Joe

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 21 May 2020 15:38:51 +0200

Dear Alexander,

> I've seen suggestions for getting this functionality in TCP by overloading the Timestamps option or even TCP SEQ - can't say that such design looks good for me. I wonder was there an attempt to add…

Overloading TCP sequence numbers or TCP timestamps to support this feature could lead to problems that are difficult to troubleshoot as there are middleboxes that modify these fields.

The latest version of Multipath TCP (RFC8684) includes a feature to support load-balancers. The connection is established via the load balancer that forwards the SYN to a server and this server advertises its own address to the client. This enables the client to stick the new subflows of the Multipath TCP connection to this server and thus completely bypass the load balancer. This is described in https://inl.info.ucl.ac.be/publications/making-multipath-tcp-friendlier-load-balancers-and-anycast.html <https://inl.info.ucl.ac.be/publications/making-multipath-tcp-friendlier-load-balancers-and-anycast.html>

Olivier
-- 

Disclaimer: https://www.tessares.net/mail-disclaimer/ 
<https://www.tessares.net/mail-disclaimer/>

---

From: Alexander Azimov <[email-masked]>
Date: Thu, 21 May 2020 18:47:13 +0300

Hi everyone, here is another all-in-one reply.

Joseph:
As far as I understand you are suggesting to use IP LSR/RH which
SHOULD be turned
off by default (RFC7196). Besides, P and X, in your terms may belong to
different address families. It seems that you still have confusion on the
purpose of my proposal:

   - There is a browser, that sends a WEB-request. The request is sent to
   the virtual IP (VIP). The browser/OS is out of our control.
   - There is a load balancer inside the datacenter, which receives an SYN
   packet and needs to send it to the server with real IP. Normally it is done
   via tunnel: IPIP or GRE. Both load balancer and server are under the same
   administrative control.
   - The idea is to introduce a connection id, that is generated by the
   server at the time of SYN/ACK and kept at the client for the time of TCP
   session, thus removing the need to keep the state at the load balancer.
   This brings advantages in the case of DDoS attacks, load drain from
   servers, and traffic rerouting between LB.

Oliver:
Thank you for the link! I was not aware of this paper. The option to
advertise VIP looks interesting, but seem to have limitations:

   - What if real server IP is IPv6 address while the client is working
   from IPv4 only network?
   - The attacker may learn the full list of backend servers, which can be
   used for DDoS attacks. The dynamic IP allocation for connections may work
   as a partial workaround, have you implemented it in the lab?
   - The load balancer still needs to keep states of the connections. But
   it wasn't the problem that was addressed in the paper.

Anyway, the connection id, as I see previously described, is a function on
top of real server IP, and with proper encoding/decoding it will be working
for Anycast when connection jumps from one to another LB. It should
be working for MPTCP case too.

PS: previously in this thread, I used the term L3LB, while in Oliver's
paper it is named L4 load balancer. I hope it hasn't brought additional
confusion.

чт, 21 мая 2020 г. в 16:38, Olivier Bonaventure <
[email-masked]>:

> Dear Alexander,
>
>
>> As you all know L3 LB is a common element of today's cloud
>> infrastructure. Its original design was based on the connection state that
>> maps backend IPs (real IPs) to connection. Recently there were several
>> papers that were suggesting the move to the stateless L3 LB, where the
>> state (connection id) becomes the property of connection and it kept at the
>> client, so the L3 LB can work without keeping state for every connection.
>> As far as I understand, for QUIC it becomes native with destination
>> connection ID in its header.
>>
>> I've seen suggestions for getting this functionality in TCP by
>> overloading the Timestamps option or even TCP SEQ - can't say that such
>> design looks good for me. I wonder was there an attempt to add connection
>> id in the TCP options, and if it was - why it wasn't successful?
>>
>
> Overloading TCP sequence numbers or TCP timestamps to support this feature
> could lead to problems that are difficult to troubleshoot as there are
> middleboxes that modify these fields.
>
> The latest version of Multipath TCP (RFC8684) includes a feature to
> support load-balancers. The connection is established via the load balancer
> that forwards the SYN to a server and this server advertises its own
> address to the client. This enables the client to stick the new subflows of
> the Multipath TCP connection to this server and thus completely bypass the
> load balancer. This is described in
> https://inl.info.ucl.ac.be/publications/making-multipath-tcp-friendlier-load-balancers-and-anycast.html
>
> Olivier
>
> ------------------------------
>

-- 
Best regards,
Alexander Azimov

---

From: Joseph Touch <[email-masked]>
Date: Thu, 21 May 2020 09:39:02 -0700

> On May 21, 2020, at 8:47 AM, Alexander Azimov <[email-masked]> wrote:
> 
> As far as I understand you are suggesting to use IP LSR/RH which SHOULD be turned off by default (RFC7196).

There’s no mention of LSR in RFC7196. Perhaps you mean RFC7126 - in which case, yes - another blow to the flexibility of the Internet by BCPs that treat anything unexpected as an attack ;-(

> Besides, P and X, in your terms may belong to different address families.

You’re adding new constraints, but OK.

> It seems that you still have confusion on the purpose of my proposal:
> There is a browser, that sends a WEB-request. The request is sent to the virtual IP (VIP). The browser/OS is out of our control.
Then you wouldn’t be talking about TCP option anymore (options cannot be added en-route). You also wouldn’t be talking about MPTCP either - in general.
> There is a load balancer inside the datacenter, which receives an SYN packet and needs to send it to the server with real IP. Normally it is done via tunnel: IPIP or GRE. Both load balancer and server are under the same administrative control.
> The idea is to introduce a connection id, that is generated by the server at the time of SYN/ACK and kept at the client for the time of TCP session,
The server TCP sequence number and timestamps are the only things that most TCP clients already keep track of that the server has control ove). 

Both of those have other meanings and change during a connection, which means that the middlebox would have to keep state to track them. But you also said the middlebox doesn’t keep state.
> thus removing the need to keep the state at the load balancer. This brings advantages in the case of DDoS attacks, load drain from servers, and traffic rerouting between LB.

I see more clearly what you want, but no way to get there if the client browser/OS is unmodified.

Joe

---

From: Alexander Azimov <[email-masked]>
Date: Thu, 21 May 2020 22:48:39 +0300

Joseph,

Then you wouldn’t be talking about TCP option anymore (options cannot be
> added en-route). You also wouldn’t be talking about MPTCP either - in
> general.
>
I believe that regular TCP options (not experimental) in general have
higher chances to be supported globally.

The server TCP sequence number and timestamps are the only things that most
> TCP clients already keep track of that the server has control ove).
>
> Both of those have other meanings and change during a connection, which
> means that the middlebox would have to keep state to track them. But you
> also said the middlebox doesn’t keep state.
>
That's the purpose of my proposal. To have a reverse function for
connection id that returns the real IP, so the load balancer doesn't need
to perform the connection tracking. Though you need to know the secret to
restore value(s), so it will not have negative security effects.

> browser/OS is unmodified.
I've meant that OS/browser is out of administrative control. So you can't
rely on a proprietary software/agent on the client-side. But the open
standard may be successful especially taking into account that main OS
vendors are content providers at the same time, thus already trying to
solve the problem of LB in different ways.

The incremental deployment is crucial, but IMO it looks doable.

-- 
Best regards,
Alexander Azimov

---

From: Joseph Touch <[email-masked]>
Date: Fri, 22 May 2020 21:10:25 -0700

> On May 21, 2020, at 12:48 PM, Alexander Azimov <[email-masked]> wrote:
> 
> I've meant that OS/browser is out of administrative control.

When (or if) you can formulate a clear, complete problem statement, let us know.

Joe

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 26 May 2020 08:51:40 +0200

Alexander,

> As far as I understand you are suggesting to use IP LSR/RH which SHOULD be turned off by default (RFC7196). Besides, P and X, in your terms may belong to different address families. It seems that you still have confusion on the purpose of my proposal:
>
> The idea is to introduce a connection id, that is generated by the server at the time of SYN/ACK and kept at the client for the time of TCP session, thus removing the need to keep the state at the load balancer. This brings advantages in the case of DDoS attacks, load drain from servers, and traffic rerouting between LB.

Another possibility would be to place the connection id in the network
layer, see doi:10.1109/TNET.2018.2799242

> Oliver:
> Thank you for the link! I was not aware of this paper. The option to advertise VIP looks interesting, but seem to have limitations:
>
> What if real server IP is IPv6 address while the client is working from IPv4 only network?

If your VIP is IPv4 and the server behind the LB only support IPv6,
then you need the packets to pass through the LB anyway

> The attacker may learn the full list of backend servers, which can be used for DDoS attacks. The dynamic IP allocation for connections may work as a partial workaround, have you implemented it in the lab?

We did some tests but did not deploy that on busy servers

> The load balancer still needs to keep states of the connections. But it wasn't the problem that was addressed in the paper.
>

Indeed. The main objective of the paper was to show how to efficiently
support MPTCP on load balancers

Olivier

--

---

From: Joseph Touch <[email-masked]>
Date: Wed, 27 May 2020 10:18:15 -0700

> On May 25, 2020, at 11:51 PM, Olivier Bonaventure <[email-masked]> wrote:
> 

Not unless the network layer is establishing the connection, IMO.

In IP, connections are transport layer and above; they shouldn’t be buried elsewhere. (And no, a flow ID is not a connection ID, because one flow may represent multiple connections or one connection may use multiple flows).

Joe