Subject: [multipathtcp] mptcp-architecture-01 review

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 9 Sep 2010 18:26:49 +0200

Here's my review of http://tools.ietf.org/html/draft-ietf-mptcp-architecture-01

After reading the document, it becomes clear that this is the architecture for a very specific type of multipath TCP. As such, I feel uncomfortable with the fact that the term "MPTCP" is used for this protocol, making it impossible to separate discussion of the general concept of MPTCP and this particular instance of MPTCP. So I think we need a name other than "MPTCP" for this particular MPTCP protocol that the wg is developing.

In the introduction:

   While multihoming and multipath functions have been
   implemented in transport protocols previously, notably SCTP[5], MPTCP
   is distinct in recognizing application and network compatibility
   goals that we believe are important for deployability of a multipath
   transport; we discuss these goals in more detail later in 
   Section 2.

It would be much better to briefly mention these important goals here; the introduction is there to introduce things, not to tell us in which section things are explained.

   This document makes three contributions

This is not a scientific paper, you know.  :-)

    Endpoint:  A host either initiating or terminating a MPTCP
      connection.

Not sure if it's a good idea to use this term here. Isn't it used for other stuff elsewhere, i.e., an interface? If an endpoint is a host, then why not simply say "host"?

May want to remove this:

   TBD - what further detail here would be useful?

   Traditional Internet architecture slots network devices in the
   network layer and lower layers of the OSI 7-layer stack, where the
   layers above the network layer - the transport layer and upper layers
   - are instantiated only at the end-hosts

It would be good to tighten sentences like this a bit, or maybe even remove them: less is more!

Middleboxes routinely interpose on
   the transport layer; sometimes even completely terminating transport
   connections, thus leaving the application layer as the first real
   end-to-end layer, as shown in Figure 3.

Breaking the current architecture doesn't constitute a new one. Although middleboxes are common, they're very far from universal. Better use "middlebox compatibility" than "network compatibility for this section".

I also disagree with picture 3. The most common types of middleboxes are firewalls and NATs, and these operate at layer 3, not layer 4. Middleboxes that make changes to layer 4 (except for the fairly common MSS clamping) are rare.

   MPTCP MUST remain backward compatible with the Internet as it exists
   today, 

This is a very problematic requireent, because the reality of the internet isn't documented anywhere so creating a design or evaluating it would be impossible.

It's also not exactly clear whether this compatibility means being able to do multipath through any and all middleboxes or recognizing middlebox interference and falling back to regular TCP.

   As a corollary to both network and application compatibility, the
   architecture must enable new Multipath TCP flows to coexist
   gracefully with existing legacy TCP flows, competing for bandwidth
   neither unduly aggressively or unduly timidly

We need numbers here, or at least some useful discussion of the parameters.

I'm confused about the middlebox issue being raised again in the middle of the introduction of the architecture and Tng stuff? One thing at a time, please.

The MPTCP component, which provides application
   compatibility through the preservation of TCP-like semantics of
   global ordering of application data and reliability, is an
   instantiation of the "application-oriented" Semantic layer

Would it be possible to say this in a more to the point way?

4.  A Functional Decomposition of MPTCP

   Having laid out the goals to be met and the architectural basis for
   MPTCP, we now provide a functional decomposition MPTCP's design.

Isn't that what the title of the section says?

   o  Packet Scheduling: This function breaks the bytestream received
      from the application into segments which are transmitted on one of
      the available lower subflows

This seems to suggest that segmentation can happen prior to path selection. This is of course problematic if the different paths have different maximum segment sizes / path MTUs.

  o  Congestion Control: This function manages congestion control
      across the subflows.  As specified, this congestion control
      algorithm must ensure that a MPTCP connection does not unfairly

must or MUST?

   These functions fit together as follows.  The Path Management looks
   after the discovery (and if necessary, initialisation) of multiple
   paths between two endpoints.  The Packet Scheduler then receives
   packets from the application

Remember, TCP is a stream protocol. Applications have no notion of packets. Neither can the path manager, as the size of the eventual packets is only known after a path has been selected.

TThe receiving subflow re-
   orders data and passes it to the MPTCP component, which performs
   connection level re-ordering,

Reorder twice? That seems unnecessary.

removes the segment boundaries

This makes little sense to me.

   The sender must be able to tell the receiver how to reorder the data,
   for delivery to the application

Doesn't the sender simply transmit the data in the order it is written by the application???

   This architecture does not mandate a mechanism for signalling such
   information, and it could conceivably have various sources.

such = ? It's not a good idea to refer to previous text like this across a paragraph boundary in any case, but I'm not entirely sure from the previous paragraph to begin with.

This is, however,
   vulnerable to middleboxes that resegment or assemble data, since
   there is no specified behaviour for coalescing TCP options.  If one
   signalled (data seqno, length), this would still be vulnerable to
   middleboxes that coalesce segments and do not correctly coalesce the
   options.

If there is no specification, how can it be done incorrectly? And what would be correct or incorrect here?

   once a packet is acked at subflow level

at the subflow level

   Under normal behaviour, MPTCP can use the data sequence mapping and
   subflow ACKs to decide when a connection-level segment was received.
   This has certain implications on end-to-end semantics.  It means that
   once a packet is acked at subflow level it cannot be discarded in the
   re-order buffer at the connection level.  Secondly, unlike in
   standard TCP, a receiver cannot simply drop out-of-order segments if
   needed (for instance, due to memory pressure).

This assumes that first the receiver ACKs on the subflow level and then decides whether to keep the data. By simply doing this the other way around there is no problem. (Although it's still useful to point out that in the other there can be problems.)

   Furthermore, it is possible to conceive of some cases where
   connection-level acknowledgements could improve robustness.  Consider
   a subflow traversing a transparent proxy: if the proxy acks a segment
   and then crashes, the sender will not retransmit the lost segment on
   another subflow, as it thinks the segment has been received.

In this case the middlebox terminates the TCP session and all bets are off. I don't think NATs ever do this, only proxies. And proxies don't propagate options AFAIK so you can't go through them anyway.

   Therefore, to provide a fully robust multipath TCP solution, MPTCP
   SHOULD feature explicit connection-level acknowledgements.

But then you'd have to use them all the time, which would introduce additional overhead, both in the form of options and as processing. I don't like this. And if you want it, it must be a MUST or there would have to be two types of data acknowledgment at the session level, which is a disaster in the waiting because it can hide implementation incompatibilities, not to mention complexity and additional work.

However, in all cases with retransmissions on
   different subflows, the lost packets SHOULD still be sent on the path
   that lost them.  This is currently believed to be necessary to
   maintain subflow integrity, as per the network compatiblity goal.

Hm, you may be resending a lot of data unnecessarily on a lossy path. Wouldn't it make more sense to NOT do this, and then if the subflow dies because a middlebox gets confused, start a new subflow with the same address pair and THEN start extra retransmissions of already segments already successfully retransmited on another subflow? Also, if a middlebox cares about this stuff then it would be a good idea to not send any out of window data until the extra retransmission is ACKed.

Also, please use "segment" rather than "packet".

   The smallest receive buffer we need to avoid stalling under any
   circumstances is max(RTO)*sum(BW).  This is, for most multipath
   connections, too expensive.

Not if you reduce the RTO significantly... The current guidelines are way out of date.

   Send Buffer: the smallest send buffer we need is sum(BDP) across all
   paths;

What's BDP? And wouldn't the send buffer need to be the same as the receive buffer at the other end? (Or the other way around.)

In practice, the senders will be web
   servers and receivers will be desktops or mobile servers.

This is ridiculous. There is more than one application in use on the internet!

   Since MPTCP will use regular TCP streams as its transport mechanism,

That would be TCP-in-TCP tunneling, which is not what we're doing here.

5.6.  Connection Identification

   Therefore,

Not a good way to start a new section.

   Legacy applications will not, however, have access to this identifier
   and in such cases a MPTCP connection will be identified by the
   5-tuple of the first TCP subflow. 

Earlier discussion was about a 4-tuple. What's the 5th value?

It is out of the scope of this
   document, however, to define the behaviour of the MPTCP
   implementation if the first TCP subflow later fails.

I think this is an important architectural decision, which should be discussed in depth and settled here.

In the case of
   applications that have specifically asked to be bound to a particular
   address or interface, MPTCP will not be used.

How is this enforced?

6.  Summary

Superfluous. There's already an abstract and an introduction.

Performance Enhancing
   Proxies (PEPs)

These often mess with the window that the receiver advertises to the sender. How is the window value in the TCP header interpreted? If a middlebox changes it on one path then it would be necessary to maintain a per-path advertised window, both to be compatible with the middlebox, but also (more importantly) to maintain performance on other paths.

The appendix seems out of place in this document.

Figures 8 and 9 look extremely similar, which is confusing.

There is no discussion of packet size and PMTUD related issues in the document!

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 Sep 2010 08:56:14 -0700

Hi, Iljitsch,

> Here's my review of http://tools.ietf.org/html/draft-ietf-mptcp-architecture-01
...
> I also disagree with picture 3. The most common types of middleboxes
> are firewalls and NATs, and these operate at layer 3, not layer 4.
> Middleboxes that make changes to layer 4 (except for the fairly common
> MSS clamping) are rare.

NATs examine and react to layer 4 info - e.g., when they examine port 
info in outbound TCP SYNs or the first outbound UDP packets to setup 
mapping associations. As such, they must operate at layer 4. Many also 
translate in-band addresses for some services, i.e., modifying the 
payload as well.

Middleboxes are known to splice TCP connections (terminate on one side 
and re-originate on the other), modify window sizes, or even rewrite the 
payloads - all of which make changes at layer 4 or above. See RFC 3234 
for numerous examples.

I agree with many of the other points you raise.

Joe