Subject: Re: [multipathtcp] Multipath deployment and fate sharing

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 14 Dec 2009 13:05:16 +0100

[I haven't been keeping up with this list, apologies if I make points that were discussed previously.]

On 25 nov 2009, at 16:52, Joe Touch wrote:

>> Coming back to fate sharing of multipath connection and initial
>> subflow: I would assume fate sharing will be a sysctl flag on the host,

That can only set the default, we need a socket option to do it per application.

>> a) connection doesn't go down with first subflow
>> b) connection dies when local interface dies
>> c) connection dies when subflow times out (this implies b)

> These are interesting configuration parameters from an implementation
> view, but from a protocol view, if we're talking about TCP, there are
> only two cases:

> 	a) connection goes down when the whole set of subflows goes down

Yes, you still need to have a _session_ timeout.

Somewhat related issue: do we explicitly close a session, or implicitly by closing all subflows? The latter leads to long delays when a subflow is down.

> 	b) connection goes down when the first subflow goes down

> What makes a subflow go down shouldn't be defined separately;

We need to avoid sending data over broken subflows, though.

> However, anything that would cause the original subflow to fail even
> when tunneled over other interfaces still, IMO, needs to cause the whole
> set to go down. The key example is loss of the IP address associated
> with the original flow - either when the interface goes down, or when
> it's lost via failed DHCP renewals, or when it's otherwise explicitly
> reconfigured.

There are several issues here. But before we delve in: why didn't we have this discussion in shim6, where the exact same problem is present? And what about mobility?

I agree that if a host loses an address, it should no longer send packets that use that address in some way. So in the multiaddress MPTCP case that means a subflow that uses that address will have to be terminated when the address is removed. Note though, that some systems remove addresses when the interface goes down. That in itself is no reason to release the address. If a DHCP server or RA indicates that an address may be used for X amount of time, you get to keep it X amount of time, even if connectivity goes away, temporarily or difinitively, at some point.

But then the question is: what happens to other subflows?

My answer: "if a tree falls in the forest, and all the people in the forest are wearing headphones at maximum volume, did the tree fall?"

The problem with a socket option to indicate care is that it's going to take a long time for all apps to use this socket option, and we still need a default behavior. I don't think it makes sense to make the default behavior "care" because then you have to go out of your way to not care, which by definition people don't do. Especially since it requires thinking about how to set a socket option that will be unknown on some systems for portable applications.

However, we can deduce that applications care if they bind to a specific address, or look up the address tied to the remote end of an incoming session. If neither happens subflows can continue without trouble.

By the way: we need to communicate the address loss event to the other side so the other side can tear down the session if we lose our address but don't care but the other side cares. Note also that although binding to a listening address happens before the three-way handshake, looking up the remote address may happen afterwards.

Another class of applications is those that care about the addresses, but are not bothered by change. For instance, I may log that I'm talking to 192.0.2.1 but then when that address goes down and communication continues using 10.0.0.1, I simply log the address addition/change and all is well.

So I think what we need:

- mechanism to indicate that a session is bound to an address
- mechanism to indicate that an address has been removed
- mechanism to indicate "no address agility" (default = agility)
- mechanism to indicate "no MPTCP" (default = MPTCP)

Note that "no address agility" and "MPTCP" means that you can have additional subflows using additional addresses, but only as long as the primary addresses have reachability. I think this is similar to MIPv6, where you need to return to the HoA at certain intervals.

I don't think it's appropriate to explicitly track interface status in TCP. If the packets get there you're up, if they don't you time out eventually and then presumably you're down.

---

From: "Laganier, Julien" <[email-masked]>
Date: Mon, 14 Dec 2009 08:38:54 -0800

Hello Iljitsch,

Iljitsch van Beijnum wrote:
> Joe Touch wrote:
> > when tunneled over other interfaces still, IMO, needs to cause the 
> > whole set to go down. The key example is loss of the IP address 
> > associated with the original flow - either when the interface goes 
> > down, or when it's lost via failed DHCP renewals, or when it's 
> > otherwise explicitly reconfigured.
>
> There are several issues here. But before we delve in: why didn't we 
> have this discussion in shim6, where the exact same problem is present?

In the MPTCP fate sharing discussion there are two situations to consider:

1- a peer stop to use the original IP address but nobody else uses it
2- a peer stop to use the original IP address but somebody else starts to use it (e.g., address is re-allocated via DHCP)

In SHIM6 only the situation #1 is likely to occur, because as Marcelo pointed out earlier, SHIM6 security relies on the use of a specific type of addresses (Hash Based Addresses and Cryptographically Generated Addresses) whose size and random nature make them unlikely to ever collide, thus situation #2 cannot happen in SHIM6.

So SHIM6 decided that in situation #1 it was perfectly okay that a TCP connection continue to work even though the IPv6 address to which the TCP connection was originally bound (Upper Layer ID in SHIM6 terms) is no longer reachable/available. I tend to think that the same applies to MPTCP under situation #1. If we were to base MPTCP security over IPv6 on the use of HBA's and CGA's as well, we could guarantee that situation #2 doesn't occur, and thus be left to deal with situation #1 only and maintain same properties as SHIM6, i.e. no fate sharing for IPv6.

With IPv4 of course the situation would be different due to the IPv4 address space exhaustion, presence of NATs, and DHCP allocation with short term leases. So in IPv4 we can't rule out situation #2, and thus we want to make fate sharing the default IPv4. 

Not a perfect situation of course, but one in which at least we provide better functionality where possible (with IPv6, by leveraging on its IID addressing space size) and maintain existing functionality and semantic for the other situations (with IPv4, with NATs and DHCP)

--julien

PS: it also makes MPTCP over IPv6 more attractive than MPTCP over IPv4 and gives a good reason to migrate to IPv6 transports :)

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 14 Dec 2009 17:55:54 +0100

> In SHIM6 only the situation #1 is likely to occur, because as Marcelo pointed out earlier, SHIM6 security relies on the use of a specific type of addresses (Hash Based Addresses and Cryptographical…

Hm, yes. However, in the case of an address with an interface identifier that comes from a MAC address with the U/L bit set to "unique" we basically have the same situation as with Shim6 and HBA, except that the uniqueness is by convention rather than cryptographically protected.

But my point in my earlier message was that in a potential case 2 (might be a case 1 but we can't be sure), after the valid lifetime of the address expires, we should stop sending packets with the now invalid address. Other subflows that use different addresses can continue from the perspective of address uniqueness. 

Now there is the second issue of whether it's ok to continue communicating on different addresses when the originally chosen addresses are no longer valid for reasons of conforming to application expectations.

There, my point was that we can do this without reservation if the applications on both sides don't perform any API calls that reveal they're especially interested in the addresses used for a session.

So this leaves the situation where we know an address is no longer valid and the application(s) are interested in the addresses. Eventually we need to move to a situation where applications need to register their wishes in this regard and allow MPTCP by default if they don't, because the opposite situation isn't workable. However, it could be reasonable to have an interim period during which the default is to make MPTCP sessions bound to addresses that are no longer valid go down if the applications have expressed interest in those addresses.

> With IPv4 of course the situation would be different due to the IPv4 address space exhaustion, presence of NATs, and DHCP allocation with short term leases. So in IPv4 we can't rule out situation #…

I don't see that. With NAT you don't know who you're going to encounter behind a given address, anyway.

Iljitsch

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Dec 2009 09:38:53 -0800

...
> I agree that if a host loses an address, it should no longer send
> packets that use that address in some way. So in the multiaddress MPTCP
> case that means a subflow that uses that address will have to be
> terminated when the address is removed. Note though, that some systems
> remove addresses when the interface goes down. That in itself is no
> reason to release the address. If a DHCP server or RA indicates that an
> address may be used for X amount of time, you get to keep it X amount of
> time, even if connectivity goes away, temporarily or difinitively, at
> some point.

DHCP assignments shouldn't override explicit interface actions.

> My answer: "if a tree falls in the forest, and all the people in the
> forest are wearing headphones at maximum volume, did the tree fall?"
> 
> The problem with a socket option to indicate care is that it's going
> to take a long time for all apps to use this socket option, and we still
> need a default behavior. 

IMO, you need to make the assumption that causes the least amount of
chnage to existing apps. I understand why you want to call this TCP and
use TCP's protocol number, but that comes with this cost.

> I don't think it makes sense to make the
> default behavior "care" because then you have to go out of your way to
> not care, which by definition people don't do. Especially since it
> requires thinking about how to set a socket option that will be unknown
> on some systems for portable applications.

Yes - that's exactly the cost of playing with TCP.

> However, we can deduce that applications care if they bind to a
> specific address, or look up the address tied to the remote end of an
> incoming session. If neither happens subflows can continue without trouble.

Not sure this is sufficient. You can bind to a set of addresses (or
ANY), and still care that the IP address doesn't change (e.g., in-band
info can be IP-address specific, such as access control, etc.).

...
> Another class of applications is those that care about the addresses,
> but are not bothered by change. For instance, I may log that I'm talking
> to 192.0.2.1 but then when that address goes down and communication
> continues using 10.0.0.1, I simply log the address addition/change and
> all is well.

This presumes that you know the purpose of a log. If the log is just
informational, the above may be true. But if the log is used to verify
access, or to coordinate future responses, just knowing it's logged to a
file isn't sufficient.

> - mechanism to indicate that a session is bound to an address

I think you need the negative of this, i.e., to find out whether a
session is NOT bound to an address, that can respond, YES, NO, or DON'T
KNOW (which should behave like YES).

> - mechanism to indicate "no address agility" (default = agility)

As above, default = no agility.

> - mechanism to indicate "no MPTCP" (default = MPTCP)

Under what conditions would you want to set "no MPTCP"? If you think it
will ever change TCP semantics, then the default needs to be "no MPTCP".
If not, then why do you ever need this option?

> Note that "no address agility" and "MPTCP" means that you can have
> additional subflows using additional addresses, but only as long as the
> primary addresses have reachability. I think this is similar to MIPv6,
> where you need to return to the HoA at certain intervals.
> 
> I don't think it's appropriate to explicitly track interface status
> in TCP. If the packets get there you're up, if they don't you time
> out eventually and then presumably you're down.

See RFC1122 Sec 3.3.4.3, which explicitly ties the transport selection
of a source address to an address selected based on the state of the
interface. I.e., your assertion changes an existing standard.

Joe

---

From: "Laganier, Julien" <[email-masked]>
Date: Mon, 14 Dec 2009 09:40:20 -0800

Iljitsch van Beijnum wrote:
> > With IPv4 of course the situation would be different due to the IPv4
> > address space exhaustion, presence of NATs, and DHCP allocation with
> > short term leases. So in IPv4 we can't rule out situation #2, and thus
> > we want to make fate sharing the default IPv4.
> 

This is a valid point. 

Thus it seems that whether or not MPTCP implements fate sharing, a host can never be sure who's really behind a given IPv4 address. Essentially the address semantic is broken end-to-end.

For purely local matters, fate-sharing can be used to maintain some applications' expectations regarding the relationship between local addresses and the transport connections that are bound on them.

--julien

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Dec 2009 09:39:05 -0800

> So this leaves the situation where we know an address is no longer
> valid and the application(s) are interested in the addresses. Eventually
> we need to move to a situation where applications need to register their
> wishes in this regard and allow MPTCP by default if they don't, because
> the opposite situation isn't workable. 

Pushing MPTCP on by default in the absence of knowing it won't change
TCP semantics is unworkable too.

There are two options:

	1- use TCP as the basis of this new thing
	2- use a new protocol number

(1) lets you traverse NATs and be backward compatible with legacy apps,
but *ONLY* if you don't change the semantics.

It isn't reasonable to pick an alternative that could break existing
apps, but to push forward because it furthers the agenda of this WG.

> However, it could be reasonable
> to have an interim period during which the default is to make MPTCP
> sessions bound to addresses that are no longer valid go down if the
> applications have expressed interest in those addresses

I disagree, as per above. You're playing with TCP by your choice. It's
your responsibility to "first do no harm", which means to leave things
in legacy mode (captain goes down with the ship) until you know better.

If this is that useful, apps. will override that mode explicitly.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Dec 2009 09:45:52 -0800

Laganier, Julien wrote:
>Thus it seems that whether or not MPTCP implements fate sharing, a host
> can never be sure who's really behind a given IPv4 address. Essentially
> the address semantic is broken end-to-end.
> 
> For purely local matters, fate-sharing can be used to maintain some
> applications' expectations regarding the relationship between local
> addresses and the transport connections that are bound on them.

Apps have those expectations - local or remote. That's why some apps
break when they go through NATs. Let's not compound that situation with
by assuming fate sharing can be ignored.

Joe

---

From: "Laganier, Julien" <[email-masked]>
Date: Mon, 14 Dec 2009 09:56:37 -0800

Joe Touch wrote:
> >>> With IPv4 of course the situation would be different due to the
> >>> IPv4 address space exhaustion, presence of NATs, and DHCP 
> >>> allocation with[Laganier, Julien]  short term leases. So in IPv4
> >>> we can't rule out situation #2, and thus we want to make fate 
> >>> sharing the default IPv4.
> >>
> >> I don't see that. With NAT you don't know who you're going to
> >> encounter behind a given address, anyway.
> >
> > This is a valid point.
> >
> > Thus it seems that whether or not MPTCP implements fate sharing, 
> > a host can never be sure who's really behind a given IPv4 address.
> > Essentially the address semantic is broken end-to-end.
> >

Not what I intended to say, should have been more precise.

Fate sharing does not solve (neither its absence make worse) the issue of a remote address X not being uniquely associated with a peer. For example, with fate sharing and normal TCP there's the issue of an application connected to a peer application on remote address X not being able to connect to the same peer when issuing another connect(). 

This is different from the issue of an application expecting a connection to be up if and only if the remote address X is still in use by its peer. That one is influenced by our decision regarding fate-sharing.

--julien

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Dec 2009 10:36:24 -0800

Laganier, Julien wrote:
> Fate sharing does not solve (neither its absence make worse) the 
> issue of a remote address X not being uniquely associated with a
> peer. For example, with fate sharing and normal TCP there's the issue
> of an application connected to a peer application on remote address X
> not being able to connect to the same peer when issuing another
> connect().
>
> This is different from the issue of an application expecting a
> connection to be up if and only if the remote address X is still in use
> by its peer. That one is influenced by our decision regarding fate-sharing.

I agree fate sharing doesn't solve the issue, but its absence definitely
does make it "worse", where "worse" is considered (by me) to be "not
what existing TCP does".

IMO, if you want to use TCP as the basis of MPTCP, then you need to have
defaults that do what TCP does, as visible to the app. layer, w.r.t.
addresses.

Joe

---

From: "Laganier, Julien" <[email-masked]>
Date: Mon, 14 Dec 2009 10:45:50 -0800

Joe Touch wrote: 
> >> break when they go through NATs. Let's not compound that situation
> >> with by assuming fate sharing can be ignored.
> >
> > connection to be up if and only if the remote address X is still in
> > use by its peer. That one is influenced by our decision regarding
> > fate-sharing.
> 
> I agree fate sharing doesn't solve the issue, but its absence
> definitely does make it "worse", where "worse" is considered 
> (by me) to be "not what existing TCP does".

So the situation is already "worse" without MPTCP to the extent that "existing TCP" on SHIM6 already modifies the range of event that are visible to applications compared to the "existing TCP" on "existing IPv6".
 
> IMO, if you want to use TCP as the basis of MPTCP, then you need to
> have defaults that do what TCP does, as visible to the app. layer, 
> w.r.t. addresses.

If I apply that same logic to SHIM6 it should be turned off by default for all transport connections where the application hasn't opted in.

--julien

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Dec 2009 10:52:38 -0800

Laganier, Julien wrote:
...
> So the situation is already "worse" without MPTCP to the extent that
> "existing TCP" on SHIM6 already modifies the range of event that are
> visible to applications compared to the "existing TCP" on "existing IPv6".

Agreed. See below.

> If I apply that same logic to SHIM6 it should be turned off by
> default for all transport connections where the application hasn't opted in.

Agreed. That's what it should do. If SHIM6 breaks the rules, that's not
permission for MPTCP to do the same, though.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 14 Dec 2009 21:10:40 +0100

> DHCP assignments shouldn't override explicit interface actions.

If you (as a TCP session) want to go down when the interface the address that you're bound to belongs to goes down, that is your choice. However, that is not current reality. First of all, there is no unambiguous up/down dichotomy for interfaces, and second, on many systems (FreeBSD, MacOS) TCP sessions will happily stick around when an interface is really, truly down, even to the degree that the address has been removed. (This is really nice because then the session is still there when the interface comes back up again.)

The point is the authority to use an address. If a DHCP server gives me a valid lifetime of 7200 seconds I get to use the address for two hours. Interface state has nothing to do with that. (How would the DHCP server know my interface is down? Or how would a router know the interface I configured from its RA is down?)

> use TCP's protocol number, but that comes with this cost.

I don't accept that requirement.

In the IETF, we get to change things. Even to the degree that the new instance does things that diverge significantly from the old instance. We do need to be prudent and cause the least possible amount of harm, but there is no requirement to stick to arbitrary old ways of doing things just because.

> info can be IP-address specific, such as access control, etc.).

Yes, when you bind to an address you care. If you don't bind to an address (but ANY instead) apparently you don't care.

> access, or to coordinate future responses, just knowing it's logged to a

Well, these are applications that need to know about multiaddressing/multipath anyway, so presumably these apps know whether they just need logging or they need more.

> KNOW (which should behave like YES).

Ok, I can go along with that at this point.

> As above, default = no agility.

Can we narrow down our differences to just this one?

> Under what conditions would you want to set "no MPTCP"?

Not sure if there is a large class of conditions that is incompatible with MPTCP, the only thing I can think of is speed tests. But as long as we're adding API calls we may as well err on the side of adding more than on the side of adding fewer, as the marginal cost of the extra ones is near-zero and the cost of producing and implementing an update are huge.

> interface. I.e., your assertion changes an existing standard.

I can live with that.

By the way, about the protocol number: is that relevant? If two ends, including the applications on both sides, agree on different semantics, why would it be necessary to use a different protocol number?

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Dec 2009 12:27:30 -0800

> Yes, when you bind to an address you care. If you don't bind to an address (but ANY instead) apparently you don't care.

You didn't care what it was when you started, but you may care during a
connection (if you read it out).

> Well, these are applications that need to know about
> multiaddressing/multipath anyway, so presumably these apps know whether
> they just need logging or they need more.

They might know, but you might not. I.e., how do you know without
looking at the source code?

> Can we narrow down our differences to just this one?

I'm not sure - is that the only place we differ?

> Not sure if there is a large class of conditions that is incompatible
> with MPTCP, the only thing I can think of is speed tests. But as long as
> we're adding API calls we may as well err on the side of adding more
> than on the side of adding fewer, as the marginal cost of the extra ones
> is near-zero and the cost of producing and implementing an update are huge.
> 
> By the way, about the protocol number: is that relevant? If two ends,
> including the applications on both sides, agree on different
> semantics, why would it be necessary to use a different protocol
> number?

Why is it necessary to use an old one? If you want an old one, why not
use UDP?

For any reason you want to stay with TCP, that's a reason to stick with
TCP semantics or realize you're redefining semantics TCP for everyone.
Given it's place as a core protocol, I don't think redefining semantics
ought to be on the table.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 14 Dec 2009 22:00:39 +0100

> connection (if you read it out).

Absolutely. That's why either the ends must communicate to the other side the fact that they lost an address so the other side can take action if it cares, or they must indicate that they care at any time that they start caring about the specific addresses used for the session.

> looking at the source code?

Is there any reason for the user to know? If the application uses a socket option to tell the system, the right thing will happen without user intervention.

> I'm not sure - is that the only place we differ?

The wg needs to come to rough consensus, if we can have that on everything but this point that provides a way forward.

> Why is it necessary to use an old one? If you want an old one, why not

Because of backward compatibility. If we start with TCP and only go to the new semantics when both ends agree this is readily deployable, but using a new protocol number or implementing TCPng on top of UDP means we need something to discover whether the other side supports the new protocol or not and of course middlebox issues.

> TCP semantics or realize you're redefining semantics TCP for everyone.

How am I redefining it for everyone? I only execute the new semantics when both ends support them. (Remember that for the purposes of this sub-discussion I'm assuming that the applications at both ends have opted in to address agility etc.) This is no different from the TCP window scale option, in my opinion.

> ought to be on the table.

If we can redifine bits in the IP header with not even an opt in from the original sender of the packet (diffserv) why not TCP semantics...

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Dec 2009 16:51:18 -0800

Iljitsch van Beijnum wrote:
> Is there any reason for the user to know? If the application uses a
> socket option to tell the system, the right thing will happen without
> user intervention.

The user can use the socket option to get information, and then use it
to log or use it to open a new connection. You can't know the difference
just from the first option access.

> The wg needs to come to rough consensus, if we can have that on everything but this point that provides a way forward.

I agree; I just was checking.

> Because of backward compatibility. If we start with TCP and only go 
> to the new semantics when both ends agree this is readily deployable,
> but using a new protocol number or implementing TCPng on top of UDP
> means we need something to discover whether the other side supports
> the new protocol or not and of course middlebox issues.

You need to know whether the other side supports the new protocol either
way. You save only the extra legacy TCP exchange for legacy backup.

As to middleboxes, good luck - I can't see any option-based system that
a middlebox won't mess up.

> How am I redefining it for everyone? I only execute the new semantics
> when both ends support them. (Remember that for the purposes of this
> sub-discussion I'm assuming that the applications at both ends have
> opted in to address agility etc.) This is no different from the TCP
> window scale option, in my opinion.

I think you're assuming that if the transport protocols agree, the
applications are OK with using the new semantics. I'm not.

You'd get application buy-in by making this default-off, with an option
to turn it on. I suspect many on this list won't accept that solution,
because:
	a) it means no benefit for legacy apps that aren't
	revised to support the new protocol

	b) if you have to revise the app, you might as well have
	it use a new protocol altogether.

> If we can redifine bits in the IP header with not even an opt in from
> the original sender of the packet (diffserv) why not TCP semantics...

Because IP is best effort, and diffserv doesn't break "best effort"
semantics.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 15 Dec 2009 12:46:24 +0100

> just from the first option access.

Not sure what the issue is... Obviously we don't want the user to micro-manage. The way I see it, all of this is pretty similar to the V6ONLY socket option which obviated the need to manually mess with the availability of IPv4-mapped addresses with the IPv6 socket API.

> way. You save only the extra legacy TCP exchange for legacy backup.

This is easier to do with TCP options than with knocking at a door that may or may not be there, in that case you need a timeout, adding delay, or try to set up an old style and a new style session at the same time, adding overhead.

> a middlebox won't mess up.

Actually middleboxes don't seem to be too hard on TCP options.

> applications are OK with using the new semantics. I'm not.

I was specifically addressing the situation where the applications are ok with the new semantics. If you feel that even under those circumstances changes to TCP semantics can't be made, my conclusion would have to be that you won't be able to support ANY type of multipath TCP.

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sat, 19 Dec 2009 05:03:26 +0900 (JST)

Hello,

From: Joe Touch <[email-masked]>
Subject: Re: [multipathtcp] Multipath deployment and fate sharing
Date: Mon, 14 Dec 2009 10:36:24 -0800
Message-ID: <[email-masked]>

 > does make it "worse", where "worse" is considered (by me) to be "not

In my understanding, when mptcp starts a subflow, it uses a token which was used
in the initial negotiation for the first connection. If we check this token
properly, we can at least distinguish whether the peer for the subflow is the right 
one or not. So, in my feeling, mptcp does not make the situation worse much. 

--
Yoshifumi Nishida
[email-masked]