Subject: [multipathtcp] Time for Long Options?

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 29 Jul 2010 17:19:21 +0200

Hi,

since we decided today to put the signaling stuff into the
options (good decision), I wonder if it's the right time now
to extend the option field.

Remember, Linux for example use Timestamps and SACK per
default, so that no space is free (without loosing SACK
blocks)...

Alex

//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22220
// email: [email-masked]
// web: http://www.umic-mesh.net
//

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 29 Jul 2010 17:33:38 +0200

> default, so that no space is free (without loosing SACK

The SACK blocks are only in (some) ACKs while most of the MPTCP options will be in data packets. This shouldn't be a huge problem.

As for extending the option space: this is going to be big time incompatible with existing TCP. Does it make sense to overhaul the entire TCP header? In that case we can also sneak in a stronger checksum and some MPTCP fields.

---

From: Fred Baker <[email-masked]>
Date: Thu, 29 Jul 2010 17:35:03 +0200

Are you proposing to change the TCP header? That should be an interesting exercise. 

On Jul 29, 2010, at 5:19 PM, Alexander Zimmermann wrote:

> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

http://www.ipinc.net/IPv4.GIF

---

From: Randall Stewart <[email-masked]>
Date: Thu, 29 Jul 2010 11:40:33 -0400

Could we add chunks and flags along with the checksum.. hmm this could
be real fun ;-)

R

> http://www.ipinc.net/IPv4.GIF

------------------------------
Randall Stewart
803-317-4952 (cell)

---

From: Mark Handley <[email-masked]>
Date: Thu, 29 Jul 2010 16:42:29 +0100

I've also been wondering about the "extended option" option.  For example:

http://tools.ietf.org/html/draft-kohler-tcpm-extopt-00

It seems like it might be good to define this now, even though we know
we can't effectively use it for a long time (many years at least).
The issue is of course middleboxes that might not understand, and
parse extended options as payload.  But as a long-term exercise, it
does seem like the right thing to do.

 - Mark

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 29 Jul 2010 17:43:08 +0200

Hi Fred,

> Are you proposing to change the TCP header? That should be an interesting exercise. 

I know. However, they are already some ideas in this direction... 

* TCPM thread: [tcpm] TCP Long Options, 01/07/2008
http://www.ietf.org/mail-archive/web/tcpm/current/msg03677.html

* TCPM thread: [tcpm] TCP Long Options - Why not as payload in SYNs? 14/07/2008
http://www.ietf.org/mail-archive/web/tcpm/current/msg03700.html

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 29 Jul 2010 17:46:02 +0200

Hi,

> The SACK blocks are only in (some) ACKs while most of the MPTCP options will be in data packets. This shouldn't be a huge problem.

Maybe, nevertheless TCP is bidirectional...

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 29 Jul 2010 17:51:17 +0200

> http://www.ietf.org/mail-archive/web/tcpm/current/msg03700.html

Well, maybe I know better know... And, of course, don't blame me for
shortage of TCP option space ;)

Michael

---

From: Michio Honda <[email-masked]>
Date: Thu, 29 Jul 2010 17:52:31 +0200

Many people would concern about MPTCP control data competing SACK blocks, which could happen especially at bidirectional traffic.  If MPTCP control data push aside SACK blocks, it will lead to performance degradation.  
But I'm not sure how frequently it happens and how it affects.  
We will need experiments before decision of TCP header extension.  

- Michio

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 29 Jul 2010 17:53:09 +0200

Hi Mark,

> does seem like the right thing to do.

I fully agree. And it is also better to cook a solution within a WG
than outside => http://www.rfc-editor.org/internet-drafts/draft-simpson-tcpct-03.txt

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 29 Jul 2010 18:03:25 +0200

> Maybe, nevertheless TCP is bidirectional...

There is no rule that says that if you have data and need to ACK, the two have to happen in the same packet.

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 29 Jul 2010 18:09:50 +0200

> There is no rule that says that if you have data and need to ACK, the two have to happen in the same packet.

Ok. However, we are engineers who seeks for elegant solutions, no :-)

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 29 Jul 2010 19:09:50 +0200

Hi,

There are two separate issues: options on SYN or options during the data flow.

MPTCP will only use 6B of SYN options. There is still plenty of room left for further extensions.

For data flow, MPTCP will use 16/20B for sequence mapping, and 10B for ACKs. As people have pointed out, these will leave little space for other signalling.

I agree that an architecturally clean solution would be to do long options - and only use it when it actually does go through the network.

Another solution that is not as elegant but could work now is:
1. Mandate use of Appropriate Byte Couting when using Multipath TCP
2. Turn off delayed acks (or just do ack splitting) to generate more acks than we currently do. Use the option space in those ACKs to convery additional information.

This will be a lot easier to deploy, but is not as clean.

Cons?
Costin

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 29 Jul 2010 20:30:59 +0200

Hi,

> For data flow, MPTCP will use 16/20B for sequence mapping, and 10B for ACKs. As people have pointed out, these will leave little space for other signalling.

Timestamps need 10B.
n Sack-Blocks need 8*n+2B.

Everyone can now calculate how many space we have...

> 2. Turn off delayed acks (or just do ack splitting) to generate more acks than we currently do. Use the option space in those ACKs to convery additional information.

This is as ugly as sin ;-)

Costin you really demonstrate that we need long options

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 30 Jul 2010 06:43:51 +0100

Note that the most benefit of SACK comes from the 1st SACK block.

See "Simulation-based comparisons of Tahoe, Reno and SACK TCP"
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.4657&rep=rep
1&type=pdf.

Overall, the value of each additional SACK block diminishes rapidly. The
1st SACK block is specified to always cover the most recently added
non-sequential sequence space; Thus even with only 1 SACK block,
multiple holes being generated within the RTT can be signalled back to
the sender - not as reliable as with 2, 3 or even 4 SACK blocks, true.

I haven't run the numbers (I think I've seen a formula on how to come up
with a worst case estimate), but the effectiveness of SACK is probably
around 80/16/3/1 - loosing the last 2 SACK blocks has only a very minute
effect on conservation of packets...

Furthermore, SACK blocks are typically only exchanged in one direction
(in pure ACKs), and MPTCP allows for semantics where sub-flows are only
used unidirectional (that might cause byzantine error modes with
stateful meddling middleboxes though).

As long as MPTCP signalling can fit into 20B or less, everything is
good; if more space is needed, nothing prevents the infrequent addition
of a pure ACK even in the otherwise normal data direction, btw.

Ad delayed ACKs: With TCP Offload / LRO, it's not uncommon to see much
less than one ACK every other segment. From TCP's perspective, the MSS
with doing Offload simply becomes huge (ie. 32k-1MB). Stipulating that
the receiver has to react to every (on-wire) segment with an ACK would
introduce a lot of inefficiencies.

Regards,
   Richard 

> -----Original Message-----
> From: Alexander Zimmermann 
> [mailto:[email-masked]] 
> Sent: Donnerstag, 29. Juli 2010 20:31
> To: Costin Raiciu
> Cc: multipathtcp TCP
> Subject: Re: [multipathtcp] Time for Long Options?
> 
> Hi,
> 
> > MPTCP will only use 6B of SYN options. There is still 
> plenty of room left for further extensions.
> > 
> > For data flow, MPTCP will use 16/20B for sequence mapping, 
> and 10B for ACKs. As people have pointed out, these will 
> leave little space for other signalling.
> 
> Timestamps need 10B.
> > I agree that an architecturally clean solution would be to 
> do long options - and only use it when it actually does go 
> through the network.
> > 
> > 1. Mandate use of Appropriate Byte Couting when using 
> Multipath TCP 2. 
> > Turn off delayed acks (or just do ack splitting) to 
> generate more acks than we currently do. Use the option space 
> in those ACKs to convery additional information.
> 
> >>>>> The SACK blocks are only in (some) ACKs while most of 
> the MPTCP options will be in data packets. This shouldn't be 
> a huge problem.
> >>> 
> >>> There is no rule that says that if you have data and need 
> to ACK, the two have to happen in the same packet.
> >>

---

From: Alexander Zimmermann <[email-masked]>
Date: Fri, 30 Jul 2010 09:01:47 +0200

> the sender - not as reliable as with 2, 3 or even 4 SACK blocks, true.

1. Don't forget the RTO case. SACK reneging. See RFC3517, Sec 5.1.
I want to have the redundant SACK information to quickly recover
my SACK scorebord.

2. DSACK. If the DSACK block is above the ACK line, I need a second
Block.

3. ACK loss...

4. Further extension

> of a pure ACK even in the otherwise normal data direction, btw.

Ok, maybe it's ok for MPTCP. But we cannot design MPCTP as
"Nach mir die Sintflut" (Devil may care!)

---

From: Alexander Zimmermann <[email-masked]>
Date: Fri, 30 Jul 2010 09:33:30 +0200

OK,

regarding the "positive" comments on the list (Costin, Ilpo, Mark)
Arnd and I decided that we will do something in this direction.

Anybody who is interested to join us?

Alex

Am 30.07.2010 um 09:14 schrieb Ilpo Järvinen:

> On Fri, 30 Jul 2010, Alexander Zimmermann wrote:
> 
>> 
> ...which already exists. E.g., ACK congestion control makes the SACK 
> redundancy very important, it might even make TCP need more than what
> it can get.
> 
> -- 
> i.

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Jul 2010 00:43:34 -0700

FWIW, please do first review the *lengthy* and repeated considerations 
of this issue in TCPM.

My recollection is that LO are mostly needed in the SYN, and there is no 
viable way to extend the space in the SYN.

After the SYN, it *may* be possible to extend the option space, however, 
there are complications, esp. when options are modified, etc.

Keep also in mind that options are not ordered; there is no required 
sequence, so we can't assume that the "last" option is just extended 
silently into the payload space.

Finally, there are devices that silently repack the data payloads, e.g., 
split TCP, silent proxies, etc. That's part of why I had thought this 
group was assuming that extensions would be in TLV encoding; that same 
reason argues against TCP LO.

I'd be glad to provide feedback as this evolves, e.g., to a subgroup 
offlist, if useful.

Joe

---

From: Alexander Zimmermann <[email-masked]>
Date: Fri, 30 Jul 2010 09:55:31 +0200

Hi Joe,

Am 30.07.2010 um 09:43 schrieb Joe Touch:

> FWIW, please do first review the *lengthy* and repeated considerations of this issue in TCPM.

I know. I'm aware of the discussion.

> 
> My recollection is that LO are mostly needed in the SYN, and there is no viable way to extend the space in the SYN.
> 
> After the SYN, it *may* be possible to extend the option space, however, there are complications, esp. when options are modified, etc.
> 
> Keep also in mind that options are not ordered; there is no required sequence, so we can't assume that the "last" option is just extended silently into the payload space.
> 
> Finally, there are devices that silently repack the data payloads, e.g., split TCP, silent proxies, etc. That's part of why I had thought this group was assuming that extensions would be in TLV encoding; that same reason argues against TCP LO.
> 
> I'd be glad to provide feedback as this evolves, e.g., to a subgroup offlist, if useful.

Excellent Joe. Thank you.

---

From: Andreas Ripke <[email-masked]>
Date: Fri, 30 Jul 2010 08:54:44 +0000

Hi,

SYN+payload: we tried this the other day. But as it was already
assumed/stated on the TCPM discussion list: Most OSes that are not
aware silently drop the payload and carry on (tried with the usual
suspects like Linux, OpenSolaris, WindowsXP, and {Free,Open,Net}BSD.
Only MacOSX delivers the payload to the application. Doh! The few
NAT boxes we tested didn't remove SYN payload when mangling the
packet. But there are some middleboxes in the wild rectifying
this sin. It is not easy to evaluate as control of both endpoints
is needed. Sigh. 

Best,
Andreas

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 30 Jul 2010 10:36:54 +0100

Hi Alex,
 
> > 
> > Overall, the value of each additional SACK block diminishes 
> rapidly. 
> > The 1st SACK block is specified to always cover the most recently 
> > added non-sequential sequence space; Thus even with only 1 
> SACK block, 
> I want to have the redundant SACK information to quickly 
> recover my SACK scorebord.

Agreed; but reneging would either be signalled by no longer having any
SACK block, or if the sender misses out resending some packet,
continuing to announe that remaining hole with a SACK block.

It may take longer to get all that information across, fully agreed, but
the information won't be lost.

Also note, that nothing prevents the reveiver from sending multiple
ACKs, carrying different optiosn (but identical ACK/SEQ numbers).
Depending on your requirements of timeliness and resiliency, that should
address much of the problems one may want to solve by TCP LO...

Furthermore, MPTCP options should ideally only be sent infrequently (ie.
1-2 segments per chunk), for all the rest, all the option space would be
fully available for standard options...
 
> 2. DSACK. If the DSACK block is above the ACK line, I need a 
> second Block.

No; DSACK can be inferred by the SACK block being below the ACK. A 2nd
SACK block is not necessary to signal DSACK - the sender might think the
receiver has reneged though - but with the very next ACK, the Scoreboard
would be (slowly) rebuilt. A stipulation can be made, that the SACK
reneging for a subflow (purging of the Scoreboard) requires one ACK, not
carrying any SACK (reneging all or nothing, basically; reneging of some
data but not the hole scoreboard would still result in that info being
signalled by subsequent SACK blocks...

> > good; if more space is needed, nothing prevents the infrequent 
> > addition of a pure ACK even in the otherwise normal data 
> direction, btw.
> 

Well, TCP MD5, AO etc will not be able to be carried at the same time
with TS and SACK. However, it appears to me that real bidirectional
flows are more of an exception; Tim's suggestion was to stipulate that
MPTCP uses all sub-flows all the time only unidirectional (even though
that opens the possibility of some byzantine cornercases with meddling
middleboxes.

Richard

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Jul 2010 12:24:43 -0700

On 7/30/2010 1:54 AM, Andreas Ripke wrote:
> Only MacOSX delivers the payload to the application. Doh!

FWIW, the MacOSX is implementing TCP properly. The data in the SYN is 
part of the data path (unless, presumably, the option declares 
otherwise). Options not defined are required to be silently ignored.

I suspect the other TCPs are choosing to drop data in the SYN regardless 
of what options are set - i.e., they do this to avoid queueing data for 
connections that might never happen. While this is permitted (i.e., TCP 
can always drop data so long as it doesn't ACK it), such behavior is NOT 
required.

Joe

 > The few

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Fri, 30 Jul 2010 10:14:22 +0300 (EEST)

-- 
 i.

---

From: Robert Watson <[email-masked]>
Date: Sat, 31 Jul 2010 11:44:43 +0100 (BST)

On Fri, 30 Jul 2010, Andreas Ripke wrote:

> assumed/stated on the TCPM discussion list: Most OSes that are not aware 
> silently drop the payload and carry on (tried with the usual suspects like 
> Linux, OpenSolaris, WindowsXP, and {Free,Open,Net}BSD. Only MacOSX delivers 
> the payload to the application. Doh! The few NAT boxes we tested didn't 
> remove SYN payload when mangling the packet. But there are some middleboxes 
> in the wild rectifying this sin. It is not easy to evaluate as control of 
> both endpoints is needed. Sigh.

Did you try piggybacking data on both SYN segments in the handshake?  I'd 
expect almost all implementations to drop data arriving with the active side's 
SYN as a result of SYN cookies, reduced SYN state (FreeBSD syncache, Linux 
minisocks, etc).  However, data hung on the passive side's SYN/ACK might be 
handled differently, and it would be interesting to know how things look there 
as well.

Robert