Subject: [tcpm] poll for adoption of long connectivity disruptions draft

From: "Eddy, Wesley M. (GRC-MS00)[Verizon]" <[email-masked]>
Date: Tue, 1 Sep 2009 14:27:43 -0500

Hello, the authors of the "Long Connectivity Disruptions" draft:
http://www.ietf.org/internet-drafts/draft-zimmermann-tcp-lcd-02.txt
have given a couple presentations to the WG now and asked if it
could be considered as an item for the WG for Experimental.  If
I'm not mistaken, there has been some positive feedback on the
most recent incarnation, though not a lot, and not really any
negative feedback that I could see.

As chairs, David and I are asking the WG to voice whether we should:

A) adopt this as a WG item for Experimental
B) NOT adopt this as a WG item
C) wait to decide (not ready)

Please let us know what you think.

---------------------------
Wes Eddy
Network & Systems Architect
Verizon FNS / NASA GRC
Office: (216) 433-6682
---------------------------

---

From: Pasi Sarolahti <[email-masked]>
Date: Tue, 8 Sep 2009 12:53:36 -0700

Hi,

I just read the LCD draft. Technically it looks good, and I think it  
would be ok for the TCPM start working on this as a WG item for  
Experimental RFC.

Generally, I wonder if some kind of holistic overview of the  
documented TCP enhancements would be in place at some point. In the  
recent years quite a few of these have been introduced, that each help  
in a specific problem in some communication environments, but might  
not be so important for	others. It might be useful to check that there  
are no unwanted interactions between the different schemes, or whether  
from the documents it is clear to the implementors how to implement  
RFCs A + B + C in a coherent manner with minimal additional  
implementation complexity.

Then, some high-level comments on LCD draft:

* The title would benefit from some tweaking. Maybe its just a matter  
of changing "Make" => "Making"

* It might we worth noting in Section 3 when discussing ICMP message  
content that it assumes that the IP payload is not encrypted by IPsec.  
Probably other forms of tunneling might cause problems for ICMP, too.

* Section 4.2, algorithm: this is really a nit, but might want to  
clarify in step (5) that if ICMP DU contains non-TCP header it should  
be ignored, without affecting the algorithm (right?)

* It would be interesting to have an accompanying technical report on  
some (even preliminary) results on the performance improvements using  
the algorithm. It would also be interesting to have some understanding  
how often ICMP Destination unreachables actually arrive in case of a  
real connectivity disruption in real world.

* Section 4.3, second paragraph says that with this algorithm "it has  
been proved that the retransmission was not lost due to congestion".  
This is a little questionable claim. I believe the algorithm is safe  
enough in congestion control aspects, but I'm not convinced that  
successful termination of the algorithm proves there is no congestion.  
So some rewording might be in place here.

* Section 4.4: "TCP sender MAY use the following algorithm...." -- I  
believe something is missing here. There is no algorithm in this 3- 
line section.

* You might consider if you just want to normatively refer directly  
RFC 1323 instead of the 1323bis draft. Is there something that has  
changed in 1323bis that is relevant for this document? (sometimes  
normative references to work-in-progress drafts have caused additional  
delays in the RFC ed queue, although I don't think that is much of a  
risk here)

I will send some additional detailed nits directly to the authors.

- Pasi

On Sep 1, 2009, at 12:27 PM, Eddy, Wesley M. (GRC-MS00)[Verizon] wrote:

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Tue, 08 Sep 2009 15:49:35 -0700

Overall, I think this is OK to adopt as experimental.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 08 Sep 2009 15:49:17 -0700

Pasi Sarolahti wrote:
...
> clarify in step (5) that if ICMP DU contains non-TCP header it should be
> ignored, without affecting the algorithm (right?)

It's not clear whether that is a feature or a bug.

If you have TCP that's idle, and other packets from your host traverse a
path and generate ICMP DUs, your TCP could benefit from reacting. I
didn't see anything in 1122 that would prevent that.

- --

FWIW, this draft continues the erroneous assumption that ICMPs are sent
in a timely fashion. Routers aren't required to do this, and so the
sequence number inside an ICMP should never be used as critical
information. It'd be useful (if not important) to explain the impact of
this on the algorithm in the draft.

Hie

---

From: Alexander Zimmermann <[email-masked]>
Date: Wed, 9 Sep 2009 15:42:52 +0200

Hi Pasi,

thank you so much for the review. Comments are inline.

Alex

> of changing "Make" => "Making"

Ok, we will change the title to "Making TCP more Robust to Long  
Connectivity
Disruptions". I'm also fine with changing it to something completely  
different,
are there any suggestions?

> Probably other forms of tunneling might cause problems for ICMP, too.

Ack. We will come up with something describing the issues you mention.
This seems to be a general problem for the IP layer (see below).

> be ignored, without affecting the algorithm (right?)

In our algorithm we assume that the ICMP error message is passed
up from the IP layer, for the corresponding connection.
(See RFC 1122 Section 4.2.3.9 ). Ack, we will make this more clear.
However, I think the fact that the ICMP message already got
demultiplexed implies that the ICMP DU contains a TCP header, otherwise
we would "not see" this ICMP DU at this layer.

> real connectivity disruption in real world.

Depends on what you mean with "real world" we have already presented
some measurements with Linux routers, see the slides from the last
IETF meetings. And the ICMP DU arrive surprisingly reliable.
However, we did not test juniper or cisco yet...

> So some rewording might be in place here.

Ack.

> believe something is missing here. There is no algorithm in this 3-

Yeah, I know :-) See Appendix A: TODO list

> delays in the RFC ed queue, although I don't think that is much of a

I don't think there is something relevant. I only thought
that 1323bis would be published before "our" RFC and wanted to
avoid an outdated reference list ;-).

>> Please let us know what you think.

//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22220
// email: [email-masked]
// web: http://www.umic-mesh.net
//

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 10 Sep 2009 13:27:12 +0200

Hi Joe,

thanks for your remarks. As always, comments are inline...

Alex

Am 09.09.2009 um 00:49 schrieb Joe Touch:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Pasi Sarolahti wrote:
> ...

In my opinion the algorithm should only respond to ICMPs which can be  
demultiplexed to a specific connection (see e-mail reply to Pasi).  
Exploiting other ICMP DUs may be possible, as a future work, but I  
fear it may be dangerous and collide with policy routing etc. For  
instance I wouldn't be surprised if there are "creative" ISPs which  
route HTTP connections differently than SMTP connections.

> this on the algorithm in the draft.

Ack that we should it explain more deeply. However I have to disagree  
with you: we explicitly allow routers to delay ICMPs, and try to  
benefit from those "delayed" ICMPs as long as those delayed ICMPs  
belong to the current RTO-induced loss recovery phase. Moreover, there  
are two important things to note:

1.) We check for an exact match of SND.UNA (no window etc.) with the  
sequence number included in the ICMP DU. What are the chances that a  
delayed ICMP matches SND.UNA, but not belonging to that RTO period  
arrives while this same TCP connection (ports and IPs match) is in  
timeout-based loss recovery? I say you better play lottery ;-)

2,) Even if such an very unlikely event ever occurs the impact of such  
an ICMP DU is obviously very low.

Partly this is explained in Section 4.3 in the paragraph about  
sequence number wraps. But we probably separately address the
special timing issues of ICMPs.

>
> Hie
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkqm32wACgkQE5f5cImnZruZ6QCgmy/Op6oqLMTq8XFoNSMNki6p
> 3zwAnRxph15X2K6pRQV1voL2PqJ3qVwf
> =rPsC
> -----END PGP SIGNATURE-----

---

From: Joe Touch <[email-masked]>
Date: Thu, 10 Sep 2009 14:33:06 -0700

...
> timeout-based loss recovery? I say you better play lottery  ;-) 

A router that goes down and queues pending outgoing ICMPs, then emits
them when up, could win the lottery by design.

The question is whether it matters for your algorithm. If the connection
didn't proceed anyway and this is still a useful signal, then say so.

If the connection could proceed on other paths, and as a result the
sequence numbers would need to wrap and be in-window to work, then the
probability of this happening depends on the width of the window - which
has been getting larger, and could make such a lottery more likely than
you're planning for. Again, does it matter?

Joe

---

From: Pasi Sarolahti <[email-masked]>
Date: Fri, 11 Sep 2009 00:11:58 -0700

> are there any suggestions?

No. I think "Making TCP..." is just fine.

> we would "not see" this ICMP DU at this layer.

You are right, but I guess details might depend on implementation. And  
there might be possibilities for further optimization with non-TCP  
generated ICMPs, as per Joe's earlier mail.

> However, we did not test juniper or cisco yet...

Yep, the graphs in the meeting slides are nice. As an optional further  
step (if someone had the time) they could be wrapped with some  
explanatory text into a paper format to be used as a reference for  
people trying to assess the benefits.

Regarding "real world", I had something like today's 3G & WiFi  
networks in mind.

- Pasi

---

From: Alexander Zimmermann <[email-masked]>
Date: Tue, 15 Sep 2009 19:59:24 +0200

Hi Joe,

first of all, thank you very much for your valuable comments! Since  
I'm not sure
I understand you correctly, I break down our problem to a concrete  
example with
we can play with.

* If I understand you correctly in this threat as well in the threat  
"WG Last
Call for ICMP Attacks", the "possibility" and the "impact" of a false  
positive
is something different and both should be included in our I-D. Till  
now only the
"possibility" is included. I will fix that.

* Every time I had the problem "RTO-based loss recovery AND router can  
delay to
send/generate ICMPs AND sequence space can wrap" in mind, I mean problem
"B.4.2". However, I have the feeling you mean problem "2.4.2". Right?

Alex

---

Situation:

S (Source) ---------------- R (Router) ---------------- D (Destination)

- At time t0, a steady state TCP flow with packets p1,...,pn in flight  
between a
   source S and a destination D. Packets are routed by an intermediate  
router R.

- At time t1 a connectivity disruption occurs at router R.

- At time t2, S starts to send RTO based retransmissions r1,...,rm and  
backs off
   and undoes according the algorithm.

- At time t3, the connection is re-established

- At time t4, the sequence space wraps

- At time t5, the connection is in RTO-based loss recovery again

The following scenarios are possible, when ICMP DUs for the packets  
p1,...,pn
are delayed for a longer time (or are not generated at all):

1. R does not generate ICMP DUs for p1,...,pn (congestion)
    => Backoff according to RFC 2988, no undo by design.

2. R generates ICMP DUs for p1,...,pn
   2.1 S receives ICMP DUs BEFORE t2 (standard)
       => Impact: None. ICMP DUs are ignored by design.

   2.2 S receives ICMP DUs AFTER t2, but BEFORE t3
       => Impact: Retransmission ambiguity. To be considered as not  
harmful.
       (described in I-D section 4.3, para 5)

   2.3 S receives ICMP DUs AFTER t3, but BEFORE t5
       => Impact: None. ICMP DUs are ignored by design.

   2.4 S receives ICMP DUs AFTER t5
     2.4.1 RTO due to connectivity disruption
       => Impact: No false positives since we count backoffs.

     2.4.2 RTO due to congestion
       => Impact: one false positive possible
       => Possibility: at most n/2^32
       (not described in I-D yet!!!)

The following scenarios are possible, when ICMP DUs for the  
retransmissions
r1,...,rm are delayed for a longer time (or are not generated at all):

A. R does not generate ICMP DUs for r1,...,rm (congestion)
    => Backoff according to RFC 2988, no undo by design.

B. R generates ICMP DUs FOR ALL r1,...,rm
   B.1 S receives ICMP DUs BEFORE t3
       => ICMP DUs are exploited by design. => Undo

   B.2 S receives ICMP DUs AFTER t3, but BEFORE t5
       => Impact: None. ICMP DUs are ignored by design.

   B.3 S receives ICMP DUs AFTER t5
     B.4.1 RTO due to connectivity disruption
       => Impact: No false positive since we count backoffs.

     B.4.2 RTO due to congestion
       => Impact: at most m false positives
       => Possibility: at most 1/2^32
       (described (not completely correct) in I-D section 4.3, para 6)

B.1) A steady state TCP flow between a sender S and a destination D is  
disrupted
at some router R. The first few RTO retransmissions trigger ICMP DUs  
at R, but
are delayed by it. Then, while S is still in the same loss recovery  
phase, the
accumulated ICMPs are emitted by R and received by S, triggering  
multiple
reversions. This is safe, as the result would be the same, when ICMPs  
were
emitted instantly.

B.2) A steady state TCP flow between a sender S and a destination D is  
disrupted
at some router R. The first few RTO retransmission trigger ICMP DUs at  
R, but
are delayed by it. Then, connectivity is re-established by a successful
retransmission and S leaves the loss state. If the delayed ICMPs are  
emitted by
R now, they will have no effect on S.

2.4.2) A steady state TCP flow between a sender S and a destination D is
disrupted at some router R. Each packet p[i], which is in flight from  
S to T
passes R and may (ICMP rate limiting) trigger a DU message, which is  
delayed
at R. Afterwards, when the link outage is over, the connection is
re-established, and S slow-starts the connection. When the sender's TCP
performs a sequence number wrap and approaches the sequence numbers of  
p[i],
an RTO occurs and S enters RTO-based loss recovery. Now, R emits the  
delayed
ICMPs for the p[i] of the last sequence number cycle, which may  
trigger one
false reversion. In case of congestion, this leads to one false
retransmission, while in the case of another link outage, the additional
reversion is harmless. However, it is at most one false reversion, as  
long
as we assume only one sequence number cycle of delayed ICMPs. The  
probability of
this to happen depends on the number of packets which are in flight  
towards R.

B.4.2) A steady state TCP flow between a sender S and a destination T is
disrupted at some router R. The sender performs multiple consecutive RTO
retransmissions, but the corresponding ICMPs are delayed by R. Then,  
after
several more retransmissions, connectivity is re-established and S  
leaves
loss state. A sequence space wrap later, the path is disrupted again,  
exactly
at the time at which the current SND.UNA matches the SND.UNA form the  
previous
cycle. If the router R emits the delayed ICMPs now, but is currently  
congested,
S will undo the multiple backoffs falsely. The probability of this  
should be at
most 1 to 4 billion.
Given sufficiently many RTO retransmissions in the first loss phase, the
corresponding ICMPs can pull the RTO in the second loss phase from  
MAX_RTO
to the initial RTO. However, once the ICMPs are depleted, standard  
exponential
backoff will be performed. Roughly speaking, the standard congestion
response will be delayed by a few false retransmissions, and after
log2(MAX_RTO/MIN_RTO) of those retransmissions, MAX_RTO is reached  
again.

//

---

From: Joe Touch <[email-masked]>
Date: Tue, 15 Sep 2009 11:03:19 -0700

Hi, Alex,

> I understand you correctly, I break down our problem to a concrete  

Thank you for doing this. This is the kind of discussion that, IMO,
should appear in the document. I don't disagree with any of the
conclusions below.

Joe

---

From: "Eddy, Wesley M. (GRC-MS00)[Verizon]" <[email-masked]>
Date: Fri, 18 Sep 2009 08:35:40 -0500

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Eddy, Wesley M. (GRC-MS00)[Verizon]
>Sent: Tuesday, September 01, 2009 3:28 PM
>

Based on both mailing list feedback which primarily pointed to choice A,
and the discussion at the last 2 meetings, it looks like this should be
added to our milestones.  My guess is that 1 year from now is an
appropriate time to put the milestone at.  The authors can go ahead and
submit a draft-ietf-tcpm-... version when they're ready.