Subject: [multipathtcp] New Version of MPTCP Architecture Draft

From: "Ford, Alan" <[email-masked]>
Date: Wed, 3 Feb 2010 11:03:24 -0000

Hi all,

We have submitted an updated version of the MPTCP architecture draft,
which can be found at:
http://www.ietf.org/id/draft-ford-mptcp-architecture-01.txt

This is a significant change from the previous version, concentrating
now on how core architectural principles will lead to high-level design
decisions for a MPTCP protocol.

It is intended to present and discuss this draft at the interim audio
conference next week. In the meantime, we'd be very interested in
feedback and discussion, so please do send comments to the list.

Cheers,
Alan

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 4 Feb 2010 14:18:28 +0100

Hi Alan,

one short remark: Is this correct?

"Packet Scheduling: This function breaks the bytestream received
      from the application layer into segments which are transmitted on
      one of the available lower (subflow) layers..."

Actually, segments can be transmitted on more than one subflow as the next 
paragraph says: "... segments with the same connection-level sequence number 
can be transmitted over multiple paths, i.e. as retransmissions or just to 
increase redundancy..." right?

And, one more question, is the discussion when actually to set up or close a 
subflow out of scope of this document? I was excepting some sentences about 
that in the Path Manager description...?

BR,
Mirja

On Wednesday 03 February 2010 12:03:24 Ford, Alan wrote:
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

web: www.ikr.uni-stuttgart.de
email: [email-masked]
tel: +49(0)711/685-67973
-------------------------------------------------------------------

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 05 Feb 2010 21:03:05 +0900 (JST)

Hello Alan, 

I have several questions on the reliability section.
Could you elaborate a bit?

In section 4.2, you mentioned that there are currently no connection 
levels acks. However, after a few lines later, you also mentioned connection
level MPTCP ACKs are not cumulative. Isn't it contradictionary?

If connection level MPTCP ACK exists, what does it look like? Why it cannot
be cumulative? 

".. The receiver can simply drop out-of-order segements if needed (for instnace,
due to memory pressure)."  Could you elaborate how this situation happens?

".. if the proxy acks a segment and then crashes, the sender will not retransmit
the lost segment on another subflow." To deal with this case, I think MPTCP 
needs to keep data for a while even if it is acked by subflow. But, it looks 
very expensive. Do we need to support this?

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: "Ford, Alan" <[email-masked]>
Subject: [multipathtcp] New Version of MPTCP Architecture Draft
Date: Wed, 3 Feb 2010 11:03:24 -0000
Message-ID: <[email-masked]>

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 5 Feb 2010 15:49:36 +0000

Hi Yoshifumi,

Let me try to answer those inline.

> If connection level MPTCP ACK exists, what does it look like? Why it cannot
This is a misleading phrase. It tries to say that currently, the acks at the connection level are inferred from subflow acks but do not have the same cumulative nature as tcp cumulative acks.

> due to memory pressure)."  Could you elaborate how this situation happens?
Well, in TCP any out of order data can be dropped, even if these were SACKed. This is called SACK reneging, and it will not happen frequently, but it is possible. You can't do this with MPTCP; if something was acked
at subflow level, it cannot be dropped in the rerder buffer at the connection level.

> very expensive. Do we need to support this?

You are right. Dealing with this case means that we need to introduce an explicit DATA ACK and that the sender must keep the data until it's acked at connection level. It's unclear what the best solution is.

Cheers
Costin

---

From: "Ford, Alan" <[email-masked]>
Date: Fri, 5 Feb 2010 16:57:54 -0000

Hi Mirja,

Comments inline...

> -----Original Message-----
> From: [email-masked]
[mailto:[email-masked]] On
> Behalf Of Mirja Kuehlewind
> Sent: 04 February 2010 13:18
> To: [email-masked]
> Subject: Re: [multipathtcp] New Version of MPTCP Architecture Draft
> 
> Hi Alan,
> 

That's a good point, it is not restricted to "one". I shall make that
clearer.

> that in the Path Manager description...?

The decision to set up and tear down subflows should rest with the
implementations at the endpoints. Decisions to do this may be affected
not only by generic local policy and configuration defaults, but also by
application preferences indicated through an extended API. This is
discussed further in [draft-scharf-mptcp-api].

This should probably be stated explicitly, thanks for pointing it out!

Cheers,
Alan

---

From: "Ford, Alan" <[email-masked]>
Date: Fri, 5 Feb 2010 16:57:53 -0000

Hi all,

Inline...

> From: Costin Raiciu [mailto:[email-masked]]
> Sent: 05 February 2010 15:50
> To: Yoshifumi Nishida
> Cc: Ford, Alan; [email-masked]
> This is a misleading phrase. It tries to say that currently, the acks
at the
> connection level are inferred from subflow acks but do not have the
same
> cumulative nature as tcp cumulative acks.

Yes indeed, what we're saying there is that although a connection-level
ACK can be inferred, we have not necessarily ACKed all connection-level
data up to the highest subflow-level ACKed byte, so it cannot be treated
as cumulative as in regular TCP.

We'll rephrase it for the next version, I agree it's not clear!

> > ".. The receiver can simply drop out-of-order segements if needed
(for
> instnace,
> Well, in TCP any out of order data can be dropped, even if these were
SACKed.
> This is called SACK reneging, and it will not happen frequently, but
it is
> possible. You can't do this with MPTCP; if something was acked
> at subflow level, it cannot be dropped in the rerder buffer at the
connection
> level.
> 
> >
> > ".. if the proxy acks a segment and then crashes, the sender will
not
> retransmit
> > needs to keep data for a while even if it is acked by subflow. But,
it
> looks
> You are right. Dealing with this case means that we need to introduce
an
> explicit DATA ACK and that the sender must keep the data until it's
acked at
> connection level. It's unclear what the best solution is.

We'd be very interested in wider opinions on this. This is an issue of
compatibility and convenience for some specific corner cases. On the one
hand, a connection-level ack would allow subflow-acked data to later be
lost, either deliberately (due to memory issues on a machine) or
accidentally (due to middlebox failure). On the other hand, it would add
overhead, and it could be argued that we're not providing a service
model any worse than you would get when using regular TCP when such a
failure occurs. An alternative approach may be to use a NACK here,
although I'm not particularly keen since that brings its own problems,
not least issues with timeouts to work out when to send it...

Cheers,
Alan

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sun, 07 Feb 2010 06:21:51 +0900 (JST)

From: "Ford, Alan" <[email-masked]>
Subject: RE: [multipathtcp] New Version of MPTCP Architecture Draft
Date: Fri, 5 Feb 2010 16:57:53 -0000
Message-ID: <[email-masked]>

 > > This is a misleading phrase. It tries to say that currently, the acks at the
 > > connection level are inferred from subflow acks but do not have the same

OK. I got. Thanks for the clarification.

 > > You are right. Dealing with this case means that we need to introduce an
 > > explicit DATA ACK and that the sender must keep the data until it's acked at

Hmm. we need a high level decision here whether we use connection-level ack or not.
The problems you mentioned (due to memory issues on a machine and due to middlebox 
failure) looks catastrophic situations for me. Also, middle box might support
connection-level ack in future..
If there are no other strong benefits, I don't feel we need to use this..

BTW, in section 4.1, whether MPTCP uses 2 levels(connection, subflow) sequence 
number or single sequnce number looks an open issue. However, the rest of 
discussions seems to be based on 2 levels sequnce number. I prefer to make a
decision on sequence numbering soon and to proceed to other topics.

How about making a decision on sequence numbering and connection-level ack at
next interim or Anaheim?

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Costin Raiciu <[email-masked]>
Date: Sun, 7 Feb 2010 14:44:01 +0000

Hi,

In my opinion, dual sequence numbers are mandatory for MPTCP. If we use a single sequence space across all flows we have the following problems:
- each subflow's sequence number will seem to have "gaps" to middleboxes. This may not go down well with a lot of them, like performance enhancing proxies, traffic normalizers, etc. 
- we need to rely on SACK to detect which packets were received; connection level SACK blocks will be included in every packet almost  (~26B for 3 blocks). In contrast, the data sequence number takes 10-14B on the forward 
packets, and returning SACK blocks will be much much fewer. So we can argue that dual sequence space is more space efficient.
- we cannot redundantly send data on more subflows at once, because we don't know which path (or both) delivered the packet when we get an ack. This case appears at least when you retransmit data lost packets on different subflows. Further, I imagine there will be tons of clever ways to use packet level redundancy to reduce jitter, etc. Why prohibit them by design?

As for the data ack: I think we should specify it but we should not mandate that each ack packet contains it. The receiver could generate it infrequently, and should be used by the sender as a last resort mechanism to detect weird failure modes.

Cheers,
Costin

---

From: Scott Brim <[email-masked]>
Date: Tue, 09 Feb 2010 00:24:54 -0500

Looks good but it needs some tightening.  Ignoring spelling and such ...

Re a "reference scenario", yes there should be one but it should be just
the essentials.  Don't try to cover too much.

Re Goals: At one point we discussed "enable cost reduction" as a goal.
Thoughts?  That may be the prime motivator for people to incorporate it
in mobile endpoint stacks.

> 2.2.1

>    To have a deployable protocol,
>    MPTCP SHOULD adhere to the following "do no harm" philosophy:
>    multipath TCP SHOULD behave no worse (throughput wise) than running a
>    single TCP connection over any of its paths.

This could use a little tightening up ... but I would delete it anyway
because it's said better down in 2.2.3:

   The use of multiple paths MUST not significantly harm users
   using single path TCP at shared bottlenecks, beyond the impact that
   would occur from another single legacy TCP flow.

If you keep it here in 2.2.1 ...

  If it's only SHOULD, not MUST, then "the full implications must be
  understood and carefully weighed before choosing a different course"
  (RFC2119).  There is a lot of inconsistency in how developers treat
  SHOULD, ranging from MUST to something to be ignored.  For the sake
  of implementors and consistency in the Internet it's good to fill in
  "the full implications", i.e. under what conditions would one NOT
  follow the 'do no harm' rule?

  Is "behave no worse" well-understood?  The 2.2.3 wording is clearer.

> 2.2.2 Network Compatibility

>    Although the standard TCP API will still be provided to the
>    application layer, multipath-aware applications would be able to use
>    an extended sockets API to have further influence on the behaviour of
>    MPTCP, which is also specified in [5].

s/would/will/

> 3. Multipath Architecture

>    The MPTCP layer (here and in 3.1)
>    the subflow layer
>    packet scheduling layer
>    multipath layer

These things aren't layers, they are _functional_elements_ in the
transport layer.  With MPTCP, for example, either call it just "MPTCP"
or "the MPTCP sub-layer" or maybe "the MPTCP functional element", but
not layer.  Do you ever say "the TCP layer"?  And in any case sometimes
they are called functions, sometimes layers, sometimes components.  Have
pity on people like me who can't stand to see maps folded wrong.  Here
and in 3.1.

>    o  Path Management: This is the function to detect and use multiple
>       paths between two endpoints.  In the case of the MPTCP design [3],
>       this feature is implemented using multiple IP addresses at least
>       one of the endpoints.

"at" -> "at at"

>       TCP adds
>       its of sequence numbers to the segments

its?

3.1 should be better integrated.  So far it's clearly written
separately, and some is redundant.  I would mingle it with 3.0.

> 4.4 General MPTCP Operation

>    The current MPTCP protocol proposal suggests the use of TCP options
>    for this signalling, however another approach would be to embed such
>    information in the payload, and use type-length-value (TLV) encoding
>    to separate signalling and payload data.

I think we're better off with dedicated control packets (i.e. they look
like payload to the sub-flow TCP transport, but the payload is
exclusively MPTCP control, not a mixture of MPTCP control and
application payload).  Mixing MPTCP control and higher layer payload
feels like it's going to take juggling.

> 4.5.  Path Management
> 
>    Currently, the network does not expose multiple paths between
>    endpoints.  

Sometimes "the network" does.  Currently a single interface does not
expose multiple paths between endpoints.  An endpoint might be connected
to multiple networks.  An endpoint might have multiple interfaces on the
same network, with different addresses.

> 4.7.  Network Layer Compatibility
> 
>    MPTCP's modifications remain at the TCP layer, although some
>    knowledge of the underlying IP layer is required.  MPTCP MUST work
>    with IPv4 and IPv6 interchangeably, i.e. one MPTCP connection may
>    operate over both IPv4 and IPv6 networks.

MUST work with v4 and v6 interchangeably?  I would soften that to SHOULD.

> 8. Interactions with Middleboxes

Where do you want to discuss the issues about working through Intrusion
Prevention devices?  Here?  Threats?  Multiaddressed?

Thanks ... Scott

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 09 Feb 2010 20:04:01 +0900 (JST)

Hi Costin,

From: Costin Raiciu <[email-masked]>
Subject: Re: [multipathtcp] New Version of MPTCP Architecture Draft
Date: Sun, 7 Feb 2010 14:44:01 +0000
Message-ID: <[email-masked]>

 > - we cannot redundantly send data on more subflows at once, because we don't know which path (or both) delivered the packet when we get an ack. This case appears at least when you retransmit data…

I agree. if we use a single sequence space, I think we will lose network compatibility 
for middleboxes as you pointed out. also we probably cannot say the subflow is TCP.

 > As for the data ack: I think we should specify it but we should not mandate that each ack packet contains it. The receiver could generate it infrequently, and should be used by the sender as a las…

Hmm, I'm a bit skeptical about this.
I guess the following situations are conceivable for the use cases of the data ack.

1: end system's problem
   the receiver sends back an ack for a subflow, but the data somehow is not there.

2: middle-box's problem
   the middle-box receives data and sends back an ack, but it crashes before sending the
   data to the receiver.

For 1, I guess there are serious problems at the receiver. (memory corruption,
a bug in the receiver's logic) We can suspect all subflows are affected by this.

For 2, I'm not sure how often this happens. we also need at least one subflow not to
connect this middle box. also, we cannot prevent future middle-box from sending the 
data-acks. 

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Costin Raiciu <[email-masked]>
Date: Tue, 9 Feb 2010 15:46:57 +0000

Hi Yoshifumi,

You are right that these are corner cases, but we do not have an idea of what middleboxes do. All we know is they exist, and they do lots of things to TCP connections (e.g. the Performance Enhancing Proxy RFC 3135). 

Adding the data ACK brings robustness to the connection, allowing it to continue despite weird failure modes of the subflows. That, coupled with some optimizations that reduce overhead (e.g. don't add it to every single ack) sounds to me like a good enough reason.

Thoughts?
Costin