Subject: Re: [multipathtcp] AD Review: draft-ietf-mptcp-architecture-03

From: Lars Eggert <[email-masked]>
Date: Tue, 11 Jan 2011 10:50:20 +0200

I put this ID on the IESG agenda for next week. Can you ASAP do a quick rev that fixes some of the nits below?

On 2010-12-17, at 14:01, Lars Eggert wrote:

> Summary: good to go.
> 
> I'll start the IETF last call. Please take the comments below as last-call comments. I will make this last call longer than normal, due to the holiday season.
> 
> Lars
> 
> 
> Section 1., paragraph 2:
>>   By the application of resource pooling[3], these available resources
> 
>  Nit: s/pooling[3]/pooling [3]/  (also check other reference citations
>  for a missing space before them)
> 
> 
> Section 1.2., paragraph 0:
>> 1.2.  Terminology
> 
>  I'd be somewhat useful to reorder the terms such that definitions
>  appear before they are used, as much as possible.
> 
> 
> Section 1.2., paragraph 2:
>>   Path Identifier:  Within the context of a multi-addressed multipath
>>      TCP, a path is defined by the source and destination (address,
>>      port) pairs (i.e. a 4-tuple).
> 
>  Why is a Path identified by something that includes port numbers?
>  Because of ECMP?
> 
> 
> Section 1.2., paragraph 6:
>>   Subflow:  A flow of TCP packets operating over an individual path,
>>      which forms part of a larger Multipath TCP connection.
> 
>  s/packet/segment/ (please check this throughout the document, "packet"
>  is often used where "segment" is meant)
> 
> 
> Section 1.3., paragraph 4:
>>   The scenario could have any number of addresses (1 or more) on each
>>   host, so long as the number of paths available between the two hosts
>>   is 2 or more (i.e. num_addr(A) * num_addr(B) > 1).  The paths created
>>   by these address combinations through the Internet need not be
>>   entirely disjoint - shared bottlenecks will be addressed by the
>>   Multipath TCP congestion controller.
> 
>  "will be addressed" is imprecise. Maybe: "potential fairness issues
>  introduced by joint bottlenecks need to be handled by..."
> 
> 
> Section 1.3., paragraph 5:
>>   Furthermore, the paths through
>>   the Internet may be interrupted by any number of middleboxes
>>   including NATs and Firewalls.
> 
>  Middleboxes don't really "interrupt" a path. Maybe rephrase with "are
>  present"?
> 
> 
> Section 1.3., paragraph 6:
>>   Finally, although the diagram refers
>>   to the Internet, Multipath TCP may be used over any network where
>>   there are multiple paths that could be used concurrently.
> 
>  Not sure if there is much value in mentioning this. (Plus, we do
>  assume an _IP_ network.)
> 
> 
> Section 2.1., paragraph 3:
>>   o  Improve Resilience: Multipath TCP MUST support the use of multiple
>>      paths interchangeably for resilience purposes, by permitting
>>      packets to be sent and re-sent on any available path.
> 
>  I think you mean "segments" (i.e., chunks of the bytestream) here, and
>  not IP packets.
> 
> 
> Section 2.1., paragraph 5:
>>   As distribution of traffic among available paths and responses to
>>   congestion are done in accordance with resource pooling
>>   principles[3], a secondary effect of meeting these goals is that
>>   widespread use of Multipath TCP over the Internet should optimize
>>   overall network utility by shifting load away from congested
>>   bottlenecks and by taking advantage of spare capacity wherever
>>   possible.
> 
>  s/optimize/improve/ (it won't be optimal unless there is global
>  deployment *and* the mechanism itself is optimal)
> 
> 
> Section 2.2.1., paragraph 3:
>>   A multipath-capable equivalent of TCP SHOULD retain backward
>>   compatibility with existing TCP APIs, so that existing applications
>>   can use the newer transport merely by upgrading the operating systems
>>   of the end-hosts.
> 
>  Any reason why this is not a MUST? I think this is critical; otherwise
>  we'd already have seen widespread use of SCTP.
> 
> 
> Section 2.2.2., paragraph 1:
>>   In the traditional Internet architecture, network devices operate at
>>   the network layer and lower layers, with the layers above the network
>>   layer instantiated only at the end-hosts.  While this architecture,
>>   shown in Figure 2, was initially largely adhered to, this layering no
>>   longer reflects the "ground truth" in the Internet with the
>>   proliferation of middleboxes[8].  Middleboxes routinely interpose on
>>   the transport layer; sometimes even completely terminating transport
>>   connections, thus leaving the application layer as the first real
>>   end-to-end layer, as shown in Figure 3.
> 
>  And even application level protocols often define roles for
>  intermediaries, e.g., SIP and HTTP.
> 
> 
> Section 2.4., paragraph 1:
>>   standardised.  The de facto standard stream-based transport protocol
> 
>  s/de facto standard/by far most widely used/ (TCP *is* of course a
>  standard...)
> 
> 
> Section 3., paragraph 7:
>>   behaving as a TCP flow in network, is an instantiation of the
> 
>  Nit: s/in network/in the network/
> 
> 
> Section 4., paragraph 10:
>>      shared bottlneck.  An algorithm to support this is specified in
> 
>  Nit: s/bottlneck./bottleneck./
> 
> 
> Section 4., paragraph 12:
>>   own sequence number, acks, and passes them to network.  The receiving
> 
>  Nit: s/acks,/ACKs,/
> 
> 
> Section 5.1., paragraph 3:
>>   receiver must determine how subflow-level data (carying subflow
> 
>  Nit: s/(carying/(carrying/
> 
> 
> Section 5.1., paragraph 6:
>>   be permissable for the mapping to be sent periodically and cover more
> 
>  Nit: s/permissable/permissible/
> 
> 
> Section 5.2., paragraph 2:
>>   to-end semantics.  It means that once a packet is acked at the
> 
>  Nit: s/acked/ACKed/
> 
> 
> Section 5.2., paragraph 4:
>>   a subflow traversing a transparent proxy: if the proxy acks a segment
> 
>  Nit: s/acks/ACKs/
> 
> 
> Section 5.2., paragraph 6:
>>   Therefore, to provide a fully robust multipath TCP solution, MPTCP
>>   SHOULD feature explicit connection-level acknowledgements, in
> 
>  After all of this argumentation in the previous paragraphs, why is
>  this not a MUST? Esp. since in the first paragraph you already say
>  "MPTCP features..." which in essence is a MUST.
> 
> 
> Section 5.2., paragraph 10:
>>   maintain subflow integrity, as per the network compatiblity goal.  By
> 
>  Nit: s/compatiblity/compatibility/
> 
> 
> Section 5.2., paragraph 11:
>>   doing this, throughput will be wasted, and it is unclear at this
>>   point what the optimal retransmit strategy is.
> 
>  I think "throughput will be wasted" sounds too harsh. Maybe "some
>  efficiency is lost"?
> 
> 
> Section 5.3., paragraph 6:
>>   The resulting buffer size should be small enough for practical use.
>>   However, there may be extreme cases where fast, high throughput paths
>>   (e.g. 100Mb/s, 10ms RTT) are used in conjunction with slow paths
>>   (e.g. 1Mb/s, 1000ms RTT).  In that case the required receive buffer
>>   would be 12.5MB, which is likely too big.  In these cases a Multipath
>>   TCP scheduler SHOULD use only the fast path, potentially falling back
>>   to the slow path if the fast path fails.
> 
>  Please rephrase the last sentence so that it not specific to the
>  example given, since we're trying to state a general rule here. Maybe:
>  "it may be prudent to only use some of the fastest available paths for
>  the MPTCP connection" or something like that
> 
> 
> Section 5.4., paragraph 3:
>>   The MPTCP protocol design will, however, use TCP Options for this
>>   additional signalling.  This has been chosen as the mechanism most
>>   fitting in with the goals as specified in Section 2. 
> 
>  Aside: Instead of using several different TCP Option numbers, I wonder
>  if we can use one TCP Option Number for MPTCP and disambiguate based
>  on the Length field what kind of MPTCP option we have? (Assuming all
>  our different option kinds have different lengths...)
> 
> 
> Section 5.5., paragraph 1:
>>   Currently, the network does not expose multiple paths between hosts.
> 
>  This isn't really capturing it. The network *does* expose this, *if*
>  the paths begin and end at different IP addresses. I think you mean
>  that the network doesn't expose the availability of path diversity *in
>  the network* for paths between the same two IP addresses.
> 
> 
> Section 5.5., paragraph 2:
>>   that these paths, whilst not necesarily entirely non-overlapping,
> 
>  Nit: s/necesarily/necessarily/
> 
> 
> Section 5.5., paragraph 4:
>>   Multiple different (source, destination) address pairs will thus be
>>   used as path selectors in most cases.  Each path will be identified
>>   by a TCP 4-tuple (i.e. source address, destination address, source
>>   port, destination port), however, which can allow the extension of
>>   MPTCP to use such 4-tuples as path selectors.  This will allow hosts
>>   to use MPTCP to load balance to different ports, for example if the
>>   network routes different ports over different paths (which may be the
>>   case with technologies such as Equal Cost MultiPath (ECMP) routing
>>   [16]).
> 
>  So actually, you use a five-tuple, it's just that the protocol number
>  component is always 6 (TCP) in our case. There are ECMP schemes where
>  the resulting paths are different when only the protocol number
>  component is different. I'd prefer if we could talk about five-tuples
>  and path IDs, since paths are not usually something that is specific
>  to one transport protocol. (And you actually do already talk about
>  five-tuples in the next section...)
> 
> 
> Section 5.5., paragraph 6:
>>   The modularity of path management will permit alternative mechanisms
>>   to be employed if appropriate in the future.
> 
>  What modularity of path management? Nothing in this section so far
>  talks about modularity. Suggest to drop this paragraph.
> 
> 
> Section 5.8., paragraph 11:
> 
>>   o  The desire to support a 'break-before-make' (as well as a 'make-
>>      before-break') approach to adding subflows implies that a host
>>      cannot rely on using a pre-existing subflow to support the
>>      addition of a new one.
> 
>  But I don't think we want to support "unlimited" break-before-make,
>  i.e., just because we had the connection up yesterday doesn't mean I
>  should still be able to join a new first flow into it today. In other
>  words, we want to support break-before-make for some limited amount of
>  break time only.
> 
> 
> Section 7., paragraph 3:
>>   just basic datagram delivery.  A miriad of middleboxes are deployed
> 
>  Nit: s/miriad/myriad/
> 
> 
> Section 7., paragraph 15:
>>   o  Segmentation and Colescing: middleboxes (or even something as
> 
>  Nit: s/Colescing:/Coalescing:/
> 
> 
> Section 12.2., paragraph 7:
>>   [9]   Carpenter, B., "Internet Transparency", RFC 2775,
>>         February 2000.
> 
>  Unused Reference: '9' is defined on line 1108, but no explicit
>  reference was found in the text
> 
> 
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 11 Jan 2011 10:59:30 -0000

Thanks for your comments, Lars. A new version addressing them is
available at:

http://www.ietf.org/id/draft-ietf-mptcp-architecture-04.txt

Details of changes inline...

> -----Original Message-----
> From: [email-masked]
[mailto:[email-masked]] On
> Behalf Of Lars Eggert
> Sent: 17 December 2010 12:02
> To: [email-masked] Mailing List
> Subject: [multipathtcp] AD Review: draft-ietf-mptcp-architecture-03
> 
> Summary: good to go.
> 
> I'll start the IETF last call. Please take the comments below as
last-call
> comments. I will make this last call longer than normal, due to the
holiday
> season.

All fixed I think.

>   appear before they are used, as much as possible.

This is slightly tricky due to interdependence, but "Multipath TCP"
moved to the top.

>   Why is a Path identified by something that includes port numbers?

Yes, this is explained in Section 5.5.

Quite why this definition was still there I don't know, I've removed it
- it adds nothing to clarity being this early.

>   is often used where "segment" is meant)

All changed where appropriate.

>   Middleboxes don't really "interrupt" a path. Maybe rephrase with
"are

Difficult wording to get right... text changed to:

   Furthermore, the paths through the Internet often do not
   provide a pure end-to-end service, and instead may be affected by
   middleboxes such as NATs and Firewalls.

>   Not sure if there is much value in mentioning this. (Plus, we do

Removed.

>   we'd already have seen widespread use of SCTP.

Only compatibility with the most commonly used features is really
required. Anyway, text changed.

>   intermediaries, e.g., SIP and HTTP.

These are application-level intermediaries, however, which is a bit out
of scope for MPTCP. The section is concentrating on middleboxes that
exist at the transport layer. I couldn't find an appropriate way of
inserting this into this section without going off on a
non-MPTCP-relevant tangent; I hope you're ok with it still left out.

>   "MPTCP features..." which in essence is a MUST.

It doesn't need to be a MUST if the implementation can be sure of
behaviour of the network and endpoints. So I've changed it to: "MPTCP
for use on the public Internet MUST feature explicit connection-level
acknowledgements".

>   the MPTCP connection" or something like that

Now reads:

   In extreme cases such as
   this example, it may be prudent to only use some of the fastest
   available paths for the MPTCP connection, potentially using the slow
   path(s) for backup only.

>   our different option kinds have different lengths...)

Seb's already answered that one - it's a bit limiting. However, to save
on option numbers, a sub-code could be used (at the cost of an
additional octet). But this is out of scope for the architecture, anyway
(it only says that Options will be used, not how they'll be used).

>   the network* for paths between the same two IP addresses.

Now reads:

   Currently, the network does not expose path diversity between pairs
   of IP addresses.  In order to achieve path diversity from today's IP
   networks, in the typical case MPTCP uses multiple addresses at one or
   both hosts to infer different paths across the network.

>   five-tuples in the next section...)

Fair enough... All changed to "five-tuple".

>   talks about modularity. Suggest to drop this paragraph.

I was going to drop it, but then thought it makes an important point,
just badly worded. Now reads:

   The path management is a separate function from the packet
   scheduling, subflow interface, and congestion control functions of
   MPTCP, as documented in Section 4.  As such it would be feasible to
   replace this IP-address-based design with an alternative path
   selection mechanism in the future, with no significant changes to the
   other functional components.

>   words, we want to support break-before-make for some limited amount
of

I've added "time-limited", but of course it's entirely up to
implementations, just like retaining TCP state today.

However long it is, it doesn't change the implication that's stated.

>   reference was found in the text

I think the nits tool is broken :)

(Adding spaces fixed that one, it's always been there on Page 8).

I've also addressed Cullen's two minor comments by adding the following
to "Network Compatibility":

   Middleboxes may also cause some TCP features to be able to exist on
   one subflow but not another.  Typically these will be at the subflow
   level (such as SACK [10]) and thus do not affect the connection-level
   behaviour.  In the future, any proposed TCP connection-level
   extensions should consider how they can co-exist with MPTCP.

And the following to Interactions with Applications:

   TCP features the ability to send out-of-band ("Urgent") data.
   Although the use of Urgent data is not recommended (see [20]), MPTCP
   SHOULD still support this feature if requested by an application.  An
   MPTCP implementation SHOULD send the Urgent data on one subflow of
   the MPTCP connection; it MAY choose this to be the best performing
   subflow.

Thanks,
Alan