Subject: [multipathtcp] Review of draft-scharf-mptcp-mctcp

From: Costin Raiciu <[email-masked]>
Date: Thu, 22 Jul 2010 09:46:33 +0100

Hi Michael,

I've read your draft. Thanks for writing it.

One reason we did not finish the payload draft were these findings that Mark mentioned earlier:
- explicit data ACKs are needed for  correct flow control; implicit ones are not enough
- data ACKs cannot be payload encoded
I'll be upfront and say its unclear to me what can be done to avoid these issues with payload encoding.

Some detailed comments about your proposal:

You said your solution is a hybrid between payload and options encoding, but I fail to see the big difference to pure payload encoding. Any payload encoding MUST use options on SYN packets to negotiate multipath TCP capability. Our unfinished payload draft does this too. This is also what your solution uses. Am I missing something?

Besides the encoding, one major difference between the current spec and your solution is that your solution has explicitly two phases:
Phase 1 - connection is single path, with regular payload encoding
Phase 2 - connection is multipath, with TLV chunked encoding.
There is a phase shift from 1 to 2 when multiple subflows are needed, and from 2 to 1 sometimes (not clear when).

- Your design choice brings up a good question: do we want to try and look as much as regular TCP when in single path mode? This might make sense for compatibility reasons, and to reduce costs.
Our options design can specify an infinite DSN mapping in the SYN to tell the receiver it should use subflow data sequence numbers for connection level reordering. Also, when an infinite mapping is in place, data ACKs are not needed, and the sender just uses subflow ACKs to free its send buffers. This phase of the options draft is very close to phase 1 of your algorithm.

Our current spec uses this phase as a fallback to regular TCP when things go bad - mainly to deal with middleboxes that strip options on data segments and with middleboxes that change the payload.
However, we have not specified a way to return to the regular multipath phase from this one. It is not difficult to do so: we could use techniques such as: send a new finite DSN mapping and a new segment,
and wait for the data ACK  before proceeding. Receiving the data ACK tells you the receiver know about the mode switch, 

- One of your design goals is to allow user-mode implementations, with minimal support from the kernel. This is obviously a good thing for the adoption point of view, but I don't think we can get away with so few changes to the stack. In particular, besides signalling on SYNs, the OS must manage a shared memory pool for the whole connection - otherwise you can easily get deadlocks. The alternative is to allocate, for each subflow, the memory needed
for the entire connection. In that way, we could avoid deadlocks; this memory is, at best, linear in the number of subflows. With the user space solution memory usage would be at least quadratic in the numbe of subflows.
Besides  inneficiency, this requires that the receive buffers of existing connections can be altered at runtime from user-space. Is that possible with current OSes?

- You propose to use a data NACK to allow the connection to recover from pro-active acking middleboxes that fail to deliver the data. That should work. However, it means the sender must use timers to delete this data. How big are
these timers? Probably you wouldn't want to delete data before a timeout would occur (presumably this would be used by the receiver to detect missing data). For 3g connections with rtts well into 1s, timeouts can be a lot more than
that. You really don't want to buffer all of your sent data for a few seconds at the sender.

- It's unclear to me how you can correctly implement coupled congestion control - required for TCP fairness -  from user land.

Thanks,
Costin

On 22 Jul 2010, at 08:38, SCHARF, Michael wrote:

> Dear all,
> 
> please find enclosed a draft version of my slides on options vs. payload. I hope that this facilitates the discussion in Maastricht.
> 
> Please note that the slides try to focus only on the encoding issue and leave out other aspects of MCTCP. In particular, I did not detail the kernel vs. user space question, even if I am still convinced that this is a further relevant criteria for that discussion.
> 
> I am still thinking about a slide that explains why and how data ACKs can be transported in the payload.
> 
> At the moment, I refer to the unpublished payload encoding draft in order to acknowledge that work, which was developed independently of MCTCP. I think that it is worth  mentioning that a (nearly) 1:1 mapping of the signaling into payload is doable, too. Please drop me a note if you want me to remove this reference to the draft.
> 
> Michael
> 
> --
> Michael Scharf
> Alcatel-Lucent Deutschland AG, Bell Labs, Stuttgart, Germany
> Email: [email-masked] / Phone: +49 711 821 40425
> 
> Domicile of the company: Stuttgart, District Court Stuttgart HRB4026
> 
> Chairman of the board of directors: Michael Oppenhoff; board of management:
> Alf Henryk Wulf (chairman), Hans-Jörg Daub, Dr. Rainer Fechner, Peter Kury
> <payload_options_draft.pdf>

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 22 Jul 2010 14:44:51 +0200

Hi Costin,

thanks for the detailed review!

> I'll be upfront and say its unclear to me what can be done to 
> avoid these issues with payload encoding.

If I correctly understand the design decisions document, both statements
are based on assumptions about the realization of the buffer management
in the receiver, which is actually left to the TCP/IP implementation.
Please correct me if I am wrong.

We have to carefully separate what is an inherent limitation of a
protocol, and what is a problem of a specific buffer management
implementation. The corresponding sections seem to assume fixed and very
small buffer sizes. In this case, MPTCP will IMHO run into problems
whatever encoding is used: Reassembly is difficult without additional
memory. Also, a stack might need additional memory to store the
information received in MPTCP's options. All this is highly
implementation-specific, and some of the assumptions in the design
document do not apply to MCTCP. To me, the only conclusion out of the
design document is that a MPTCP receiver will need a dynamic memory
management (dynamic receive buffer tuning). Modern TCP stacks do this
anyway. 

The design document claims (p. 8): "In fact, due to the implementation
of dynamic receive buffer sizing in modern operating systems, the
receive window often tracks only just above the level actually
required. It is likely that receive window problems can occur in such
operating systems even when flow control should not kick in."

Could you give an example that supports this claim, given that this
seems to be such a crucial issue for the a protocol design discussion?
Is this problem occuring in practice in the MPTCP Linux kernel
implementation?

Note that, to the best of my knowledge, many current operating systems
announce rather large receive windows (>64K). If they use dynamic memory
management, they can increase the receive window on a subflow as long as
the OS has some free memory. In this case, IMHO none of the deadlock
situations described in the design document can occur. But, again,
please correct me if I am wrong.

As a side note, there are further design principles in the document that
are IMHO not universially valid, or even wrong. Two examples right at
the beginning:

* Design principle 2 is only necessary if the passive opener would
trigger any multipath actions, which is not necessarily true. In case of
MCTCP, the active opener has to trigger the change to multi-connection
mode, i. e., there is no problem if the MPCAP option in the SYN/ACK gets
lost. In that case, the passive opener just observes that this
connection never enters multi-connection mode, but I don't see any
problems with this.

* Design principle 3 is apparently also wrong. If desired, I can explain
why.

It would really be helpful to separate design principles that are
independent of any protocol from requirements that only apply under
certain assumptions.

> You said your solution is a hybrid between payload and 
> options encoding, but I fail to see the big difference to 
> pure payload encoding. Any payload encoding MUST use options 
> on SYN packets to negotiate multipath TCP capability. Our 
> unfinished payload draft does this too. This is also what 
> your solution uses. Am I missing something?

As far as I can tell, the key difference seems to be that MCTCP does not
change the encoding format for single path transfer, i. e., it is
backward compatible for all middleboxes until one really wants to use
multiple paths. Apart from that, there are numerous details that MCTCP
solves in a different way, due to different design objectives.

MCTCP has been developed independently from your unpublished payload
draft. I assume that there are both similarities and differences. MCTCP
is by far not the only way how to realize payload encoding, and it would
really be worth discussing other design alternatives. Also, note that
the MCTCP draft explicitly lists some drawbacks of the design, and it is
possible to address some of them by alternative approaches.

> Besides the encoding, one major difference between the 
> current spec and your solution is that your solution has 
> explicitly two phases:
> Phase 1 - connection is single path, with regular payload 
> encoding Phase 2 - connection is multipath, with TLV chunked encoding.
> There is a phase shift from 1 to 2 when multiple subflows are 
> needed, and from 2 to 1 sometimes (not clear when).

Basically, the fall back from multi-connection mode to single-connection
mode is only needed when there are content-modifying middleboxes, or
middleboxes that entirely block TLV transport.

> - Your design choice brings up a good question: do we want to 
> try and look as much as regular TCP when in single path mode? 
> This might make sense for compatibility reasons, and to reduce costs.

Agreed.

> Our options design can specify an infinite DSN mapping in the 
> SYN to tell the receiver it should use subflow data sequence 
> numbers for connection level reordering. Also, when an 
> infinite mapping is in place, data ACKs are not needed, and 
> the sender just uses subflow ACKs to free its send buffers. 
> This phase of the options draft is very close to phase 1 of 
> your algorithm.

True. I was not aware of your idea of infinite mappings when designing
MCTCP, but it is indeed similar.

> Our current spec uses this phase as a fallback to regular TCP 
> when things go bad - mainly to deal with middleboxes that 
> strip options on data segments and with middleboxes that 
> change the payload.
> However, we have not specified a way to return to the regular 
> multipath phase from this one. It is not difficult to do so: 
> we could use techniques such as: send a new finite DSN 
> mapping and a new segment, and wait for the data ACK  before 
> proceeding. Receiving the data ACK tells you the receiver 
> know about the mode switch, 

I have thought about that solution, too. My current understanding is
that a fallback to infinite mapping is not robustly possible, but I
might be wrong.

The fundamental problem is that the usage of both infinite and finite
DSN mappings results in contradicting mappings for the sequence number
space. The sender must know that the receiver has received the updated
mapping, and the receiver must know that the sender is actually applying
it. If a middlebox on all paths starts to strip options, there is IMHO
no way how sender and receiver can both know what mapping is indeed used
(example: what happens in your example if the data ACK gets lost?). But,
as I said, I might have missed some workaround.

Please specify a way how to return to the regular phase, so that we can
check whether it is robust, or whether you risk to corrupt the
application bytestream by applying the wrong mapping (which is IMHO even
worse than closing the connection).

> - One of your design goals is to allow user-mode 
> implementations, with minimal support from the kernel. This 
> is obviously a good thing for the adoption point of view, but 
> I don't think we can get away with so few changes to the 
> stack. In particular, besides signalling on SYNs, the OS must 
> manage a shared memory pool for the whole connection - 
> otherwise you can easily get deadlocks.

There are two different ways how to implement MCTCP, as explained in the
draft:

It is either a new shim layer inside the OS, mostly on top of individual
connections. Then, it is basically operating on socket level. Or it is
realized in the user space, as an own library (such as glibc), in
combination with some small patches of the TCP stack to deal with the
TCP options and with coupled congestion control. To me, this is the more
interesting variant as long as multipath transport is not universially
supported in the Internet.

In the latter case, the MCTCP library has an own memory management
anyway, which is probably the shared memory pool that you refer to. For
a kernel solution, I would personally decouple the memory management of
MCTCP from the receive buffer of an individual TCP connection. But this
is left to an implementation, and I am not familiar with all OS
constraints. Note that in case of Linux (the only TCP implementation
that I know in detail), there is no fixed receive buffer. Memory is
dynamically allocated for each incoming packet, and there is no real
reservation of buffer capacity. Thus, the OS already uses a shared
memory pool for all TCP connections today.

Memory management in a modern OS is a highly complex issue, and IMHO
hardly relates to the assumptions of receive buffers in RFC 793. But
maybe an OS expert has to speak up here.

> The alternative is to 
> allocate, for each subflow, the memory needed for the entire 
> connection. In that way, we could avoid deadlocks; this 
> memory is, at best, linear in the number of subflows. With 
> the user space solution memory usage would be at least 
> quadratic in the numbe of subflows.
> Besides  inneficiency, this requires that the receive buffers 
> of existing connections can be altered at runtime from 
> user-space. Is that possible with current OSes?

I cannot follow this argument. I don't think that one would implement
MCTCP with very large buffers per subflow.

BTW, in case of Linux, there are socket options and sysctls that allow
to modify the "receive buffer size" from user space. AFAIK they mostly
affect the receive window announced to the sender, and only indirectly
the kernel memory management. These settings are applied in runtime,
AFAIK. But, again, this is a topic for OS experts.

> - You propose to use a data NACK to allow the connection to 
> recover from pro-active acking middleboxes that fail to 
> deliver the data. That should work. However, it means the 
> sender must use timers to delete this data. How big are these 
> timers? Probably you wouldn't want to delete data before a 
> timeout would occur (presumably this would be used by the 
> receiver to detect missing data). For 3g connections with 
> rtts well into 1s, timeouts can be a lot more than that. You 
> really don't want to buffer all of your sent data for a few 
> seconds at the sender.

A timeout is one solution. Alternatively, the sender could estimate the
worst-case BDP and store some few multiples of that. For a HSPA
connection with (optimistically) 10 Mbit/s and 200ms RTT, the BDP is 250
KB. Caching for 3s would require less than 1 MB of memory. IMHO, current
smartphones, netbooks, etc. have at least 1GB of memory. Memory is not
that scarce... Is there really an issue?

Of course, I agree that if there are pro-active acking middleboxes,
under certain constraints, a sender using an explicit data ACK may
require less buffer than a NACK-based solution, for the same level of
reliability.

But, as I pointed out in the draft, I am still not convinced that data
ACKs would actually solve a relevant problem:

- Pro-actively acking middleboxes break TCP semantics and are, in
general, a really bad idea these days. With modern high-speed congestion
control algorithms, and receive window autotuning, such middleboxes have
hardly any benefit, but cause numerous problems to all TCP connections.
I doubt that working around such boxes should be the main focus for
MPTCP.

- It probably only requires a small firmware update for such
middleboxes, and afterwards they also proavtively acknowledge MPTCP data
ACKs, in particular if they are sent in options. In this case, what is
the benefit of data ACKs?

- If the data ACKs are sent frequently, they result in a lot of
overhead. If they are sent more seldomly, the buffer requirement at the
sender will be rather large, too.

To return the question: How frequently do you recommend to send data
ACKs?

> - It's unclear to me how you can correctly implement coupled 
> congestion control - required for TCP fairness -  from user land.

I can elaborate on this in detail in Maastricht, if desired. It is
doable from the user space - I have running code for
draft-ietf-mptcp-congestion-00 (I'll comment on some lessons learnt
later). But, true, this is more straightforward in the kernel space.
MCTCP anyway requires some few support in the kernel space, and there
are actually several solutions how to efficiently do congestion control
coupling.

Thanks again

Michael

---

From: Mark Handley <[email-masked]>
Date: Thu, 22 Jul 2010 15:27:36 +0100

> connection never enters multi-connection mode, but I don't see any

I guess this depends on your design intent.

Consider a single-addressed client that connects to a multihomed
server.  The client can tell from the SYN/ACK that the server is
MP-capable, but it is the server that actually needs to do something
about enabling multipath behaviour - the client cannot tell whether
the server wishes to use multiple addresses.

So at some point the server needs to take the initiative and trigger
some multipath behaviour.

> * Design principle 3 is apparently also wrong. If desired, I can explain

I don't think this is wrong.  Please explain.

 - Mark

---

From: Mark Handley <[email-masked]>
Date: Thu, 22 Jul 2010 15:36:45 +0100

> please correct me if I am wrong.

Just to be clear: are you saying that the connection cannot deadlock
because the OS can always allocate more data even when it has already
signalled to the other end that it has used all the buffer it was
willing to allocate?

Or is there some other way you see to break the deadlock in the
example from design issues document?

 - Mark

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 22 Jul 2010 17:27:29 +0200

Mark,

> > trigger any multipath actions, which is not necessarily 
> true. In case 
> > of MCTCP, the active opener has to trigger the change to 
> > multi-connection mode, i. e., there is no problem if the 
> MPCAP option 
> > in the SYN/ACK gets lost. In that case, the passive opener just 
> > observes that this connection never enters multi-connection 
> mode, but 
> > I don't see any problems with this.
> 
> Consider a single-addressed client that connects to a 
> multihomed server.  The client can tell from the SYN/ACK that 
> the server is MP-capable, but it is the server that actually 
> needs to do something about enabling multipath behaviour - 
> the client cannot tell whether the server wishes to use 
> multiple addresses.
> 
> So at some point the server needs to take the initiative and 
> trigger some multipath behaviour.

true. But this is one of the cases where MCTCP can use a mechanism that
is currently not foreseen in draft-ietf-mptcp-multiaddressed.

In MCTCP, the server could send a "Mode" TCP option in this case, as a
hint to the client that multipath transport would be desired. It is up
to the client to decide whether to react to this option, or not.

If the MPCAP option in the SYN/ACK is lost, the only consequence is that
a client might receive a "Mode" TCP option on a connection where this is
not expected. Given that the multipath capability negotiation failed in
the setup, it must ignore this option. This should not cause any harm,
and multipath transport will not be enabled. The client can actually
infer from the reception of the option that there was probably an issue
with the SYN/ACK, but in that case it is reasonable not to use multipath
transport on that connection anyway.

Note that the client is indeed implementing MCTCP in that case, i. e.,
it is not possible that the server sends the "Mode" option to a legacy
stack (which should actually just igore it).

As I said, in MCTCP, the first additional setup is setup by the client,
and as a consequence the synchronization issue described in design
principle 2 cannot occur.

> > * Design principle 3 is apparently also wrong. If desired, I can 
> > explain why.
> 

IMHO, the only requirement is that one can savely identify _that_ it is
an additional subflow. The rest can, at least in theory, be done in the
payload. We have a bunch of header bits that we could use to carry the
signal "I am a subflow". Such as the port number. Of course, a TCP
option is a very reasonable solution for this, and for sure has several
advantages. This is why MCTCP uses an option for this, too. But one
could work out a different solution, maybe with some disadvantages.

Michael

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 22 Jul 2010 16:34:18 +0100

> could work out a different solution, maybe with some disadvantages.

The connection can't be signalled in the SYN payload - most middleboxes do not allow that through.
It also can't be done in the first segment: that would mean you have to postpone your choice to
accept or reject the subflow until after you have set it up with the three way handshake 
(otherwise middleboxes will not allow you to send data segments). This smells like a nice security feature.
Costin

---

From: Mark Handley <[email-masked]>
Date: Thu, 22 Jul 2010 16:37:36 +0100

> to the client to decide whether to react to this option, or not.

Fair enough, but you're just delaying the problem.

Consider the SYN+JOIN, SYN/ACK+JOIN case in MCTCP when you actually
want to go multipath.  If the JOIN is stripped from the SYN/ACK+JOIN
you'll have a problem unless the third packet of the exchange acks the
JOIN from the SYN/ACK.

 - Mark

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 22 Jul 2010 18:32:13 +0200

> > true. But this is one of the cases where MCTCP can use a mechanism 
> > that is currently not foreseen in draft-ietf-mptcp-multiaddressed.
> >

IMHO not

> Consider the SYN+JOIN, SYN/ACK+JOIN case in MCTCP when you 
> actually want to go multipath.  If the JOIN is stripped from 
> the SYN/ACK+JOIN you'll have a problem unless the third 
> packet of the exchange acks the JOIN from the SYN/ACK.

If the SYN/ACK does not have a JOIN option, MCTCP (similar like MPTCP,
as far as I know), mandates that the active opener closes the connection
immediately with a RST. The passive owner only enters multi-connection
mode when receiving the first ACK, i. e., when the TCP connection is
sucessfully set up. But this is not possible if the JOIN gets dropped,
i. e., there is still no inconsistency, since both sides remain in
single connection mode. Or do I miss something?

Though, there may be a valid point in your example: Maybe it is possible
that one ends up in inconsistencies if one allows MPCAP or JOIN on
retransmitted SYNs or retransmitted SYN/ACKs, in combination with
reordering, strange routing, etc. So, this is maybe not a good idea.

But, still, unless I am wrong, in MCTCP there cannot be inconstencies if
MPCAP and JOIN options are set only in the first SYN or SYN/ACK, and if
they are never retransmitted. This is what my implementation actually
does right now.

Michael

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 22 Jul 2010 20:19:11 +0200

> > announce rather large receive windows (>64K). If they use dynamic 
> > memory management, they can increase the receive window on 
> a subflow 
> > as long as the OS has some free memory. In this case, IMHO 
> none of the 
> > deadlock situations described in the design document can 
> occur. But, 
> > again, please correct me if I am wrong.
> 
> Just to be clear: are you saying that the connection cannot 
> deadlock because the OS can always allocate more data even 
> when it has already signalled to the other end that it has 
> used all the buffer it was willing to allocate?

Note that the amount of memory that a system is allocating as TCP
receive buffer must already today be larger than the receiver advertised
window, if a network stack wants to use SACKs efficiently. This is why
it is IMHO wrong to assume a TCP receive buffer with a fixed upper limit
that cannot (temporarily) store further data.

But, actually, my key point is that if MCTCP is realized as an
additional layer on top of TCP, its memory allocation can be independent
of the TCP receive buffer and the semantics of the receive window
announcement on a single subflow. As the MCTCP layer can still
temporarily allocate resources even if a TCP receiver has advertised a
receive window of zero, the described deadlock cannot occur. The memory
is only required temporarily to parse a small TLV message, and released
afterwards, so there is not permanent additional memory allocation (on
the contrary, send buffer is realized). Thus, by reserving some few
additional bytes of memory when accepting a multipath connection, which
are not announced to the sender unless the connection stalls, the
deadlock is completely avoided.

Apart from that, to my understanding, an OS can always temporarily
allocate small amounts of memory outside the TCP receive buffer, and if
not, the system will crash soon. Most notably, if we talk about MCTCP in
the user space. This case is neglected in the design document, since it
only discusses TCP receive buffers of fixed size only.

> Or is there some other way you see to break the deadlock in 
> the example from design issues document?

I already explained in this e-mail

http://www.ietf.org/mail-archive/web/multipathtcp/current/msg00924.html

why this kind of deadlock cannot occur in case of MCTCP, even if it used
data ACKs in the payload. What do I miss?

Michael

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 22 Jul 2010 21:25:33 +0100

> the contrary, send buffer is realized).

Think middleboxes. Your TCP advertises a zero window, and they see data segments flying past. Those data segments will not make it to you.  Deadlock again.

Cheers,
Costin

---

From: Mark Handley <[email-masked]>
Date: Thu, 22 Jul 2010 23:04:43 +0100

> single connection mode. Or do I miss something?

OK, reading your spec, I think you're right.  There's no big issue if
the Join gets stripped from the SYN/ACK.

But now I'm confused about something else:

Suppose you encounter an FTP-aware NAT on the initial connection that
rewrites an IP address in the payload, and in so doing changes the
length of the payload (fixing up the sequence numbers and acks
appropriately).

When you switch to multi-connection mode, your sequence numbers will
no longer match the data sent on the initial connection.  Either
there's a byte or two that never arrives (because the initial
connection is shorter than the sender assumes), or there are a byte of
two of inconsistent overlap between the initial connection and the
multi-connection session.

Can the mode switch work in this case?  The only thing I can think
that would work is to pause the connection, let all buffers drain, and
then exchange the actual sequence numbers received.  Only then can you
safely resume.

 - Mark

---

From: "SCHARF, Michael" <[email-masked]>
Date: Fri, 23 Jul 2010 09:38:14 +0200

Mark,

> Suppose you encounter an FTP-aware NAT on the initial 
> connection that rewrites an IP address in the payload, and in 
> so doing changes the length of the payload (fixing up the 
> sequence numbers and acks appropriately).
> 
> When you switch to multi-connection mode, your sequence 
> numbers will no longer match the data sent on the initial 
> connection.  Either there's a byte or two that never arrives 
> (because the initial connection is shorter than the sender 
> assumes), or there are a byte of two of inconsistent overlap 
> between the initial connection and the multi-connection session.

good point.

IMHO this is a non-tivial challenge for all multipath protocol variants.
But I think that MCTCP is robust in this case, and we have some text on
this in the draft.

> Can the mode switch work in this case?  The only thing I can 
> think that would work is to pause the connection, let all 
> buffers drain, and then exchange the actual sequence numbers 
> received.  Only then can you safely resume.

No, to my current understanding, an exchange of sequence numbers is not
required. 

The MCTCP spec recommends that there is no overlap of the global
sequence numbers on the initial connection and the first coupled
connection. In this case, the receiver has to await anyway the complete
reception of the data on the initial connection for in-order delivery to
the app (which, of course, requires that the receive buffer drains).
Before delivering the first byte of data from the coupled connection, it
should check whether there is a gap or overlap between the two sequence
number ranges. In either case, it is clear that there is a length
modifying middlebox on the path that has messed up the initial
connection.

In this case, the MCTCP spec recommends to immediately close all coupled
connection, discard all received data from these connections, and fall
back to single path transport over the initial connection. This is the
most save option, given that such an FTP-helper could mess up the TLV
structures, too. The fallback to single path is possible if the sender
can retransmit all outstanding data over the initial connection. For
releasing data, it could maybe help if we added a message handshake
"Mode switch completed", so that both ends know that there was indeed no
length change on the initial connection. We could exchange the actual
sequence numbers, too, but at the moment I think that they are not
needed. Still, this would be a straightforward extension of the MCTCP
protocol, as already mentioned in the appendix of the draft.

We have also thought about the question whether we could continue
multipath transport with such a NAPT helper on the path, by re-adjusting
the sequence numbers. In the general case, I think that this is
dangerous. If that middlebox continues to rewrite content, it depends on
the framing on the subflows whether it detects the patterns it searches
for, or not, and the result of the multipath transport is somehow
undefined (this is independent of option or payload encoding). Thus,
fallback to single path seems to be the better option.

But I'd be interested in other opinions on this. IMHO, this is a
fundamental question independent of the encoding.

Maybe we could also try to provide a special workaround for selected
application protocols such as FTP and their NAPT-helpers, but I have not
thought about this, and it would be a layer violation.

BTW, how would you deal with this situation in combination with option
encoding? I assume that you don't want to enable checksumming on the
first subflow?

Michael

---

From: "SCHARF, Michael" <[email-masked]>
Date: Fri, 23 Jul 2010 10:48:12 +0200

> > additional layer on top of TCP, its memory allocation can be 
> > independent of the TCP receive buffer and the semantics of 
> the receive 
> > window announcement on a single subflow. As the MCTCP layer 
> can still 
> > receive window of zero, the described deadlock cannot occur. The 
> > memory is only required temporarily to parse a small TLV 
> message, and 
> > released afterwards, so there is not permanent additional memory 
> > allocation (on the contrary, send buffer is realized).
> 
> Think middleboxes. Your TCP advertises a zero window, and 
> they see data segments flying past. Those data segments will 
> not make it to you.  Deadlock again.

No, this is not the way how MCTCP would avoid the deadlock (for the
hypothical case that it would use data ACKs in the payload).

My suggested solution is that an MCTCP receiver allocates slightly more
memory than it announces in its receive window in "normal" operation.
When the MCTCP endsystem announces a zero window, after some time, if
makes use of this "reserve", and announces a non-zero receive window.
Once the non-zero receive arrives at the other endsystem, it can send a
data ACK, and the deadlock is prevented. For middleboxes, this is just
the normal process of a window update. And there is no deadlock.

But, indeed, you have raised a valid point: As TLV headers are
transported in payload, the receive buffer must be slighlty larger to
accomodate the TLV headers, compared to option encoding (well, actually,
this depends on how the option information is stored in the reassembly
buffer - have you thought about this?). MCTCP may indeed need a slightly
different algorithm for calculating the receive window in an MCTCP
receiver, to deal with the overhead of the TLV headers, for the case
that the MCTCP reassembly buffer is running out of memory (if this is
done in the user space, it is really a corner issue). I missed to
comment on the overhead of TLV headers in the TCP receive buffer when
writing the MCTCP draft. But this can simply be fixed by a patch of the
algorithm calculating the receive window.

This is a local change of the memory management in the receiver. For
this we neither need to change the semantics of the receive window, nor
data ACKs.

BTW: Note that you cannot be sure that a receiver running out of memory,
and announcing zero windows, will indeed accept an empty ACK segment
with a data ACK option. If this packet and its retransmissions are just
dropped at the receiver, due to lack of memory, option encoding suffers
from the same deadlock. In other words: IMHO you implicitly assume that
with option encoding a receiver will always be able to process TCP
options, even if it is already short on memory.

I think we have to be very careful with assumptions about the behavior
of a TCP stack that is announcing zero receive windows, which a
multipath protocol could change anyway, and I doubt that this is the
most important aspect to discuss about.

Michael

---

From: Mark Handley <[email-masked]>
Date: Fri, 23 Jul 2010 10:38:27 +0100

> the normal process of a window update. And there is no deadlock.

But the middlebox can have a whole load of data segments buffered
waiting for your window to open.  As soon as you open the window, the
middlebox fills it with data.

Maybe if you're lucky you get a data ack in amongst that data.  Great
- you free the send buffer, and the app can send the next part of its
response.  But the situation can just repeat again, and you've used up
your reserve now.

 - Mark

---

From: "SCHARF, Michael" <[email-masked]>
Date: Fri, 23 Jul 2010 12:34:07 +0200

> > My suggested solution is that an MCTCP receiver allocates slightly 
> > more memory than it announces in its receive window in 
> "normal" operation.
> > Once the non-zero receive arrives at the other endsystem, 
> it can send 
> > a data ACK, and the deadlock is prevented. For middleboxes, this is 
> > just the normal process of a window update. And there is no 
> deadlock.
> 
> But the middlebox can have a whole load of data segments 
> buffered waiting for your window to open.  As soon as you 
> open the window, the middlebox fills it with data.
> 
> - you free the send buffer, and the app can send the next 
> part of its response.  But the situation can just repeat 
> again, and you've used up your reserve now.

Well, we can specify a minimum size for reserve (lets say 16K), and
mandate that the sender inserts into the bytestream a payload ACK at
least once every 16K (this means that the maximum size of a TLV data
chunk must be less than 16K). This way the receiver gets for sure at
least one payload ACK within the reserve. Probably, it also makes sense
that data ACKs are prioritized in the byte stream, so that the chance of
getting more than one payload ACKs increases further.

Another solution would be to run an own flow control on top of TCP, i.
e., the payload ACKs do not only carry an global sequence number, but
also an additional session-level receive window. If such an flow control
on top of TCP ensures that the outstanding data on all subflows is
always smaller than the sum of TCP receive buffers (plus a certain
reserve for TLV headers), at least one of the subflows must always have
a non-zero receive window, and thus be able to process incoming data
ACKs. In payload encoding, one could easily add such an additional
global receive window to the TLV message. Actually, we could also add
this new global receive window to the data ACK option.

This way, the deadlock is really safely prevented, because the TCP
receive buffers will never entirely fill up. Or did I miss something?

But, now we have actually rebuild a new TCP-like transport on top of
TCP. The next question is then indeed whether we should do fast
retransmits on data ACKs. This starts to get a somehow strange protocol
design.

And IMHO we can just avoid all this mess by just not using data ACKs.

Michael