Subject: Re: [nwcrg] [LOOPS] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room

From: Carsten Bormann <[email-masked]>
Date: Sun, 15 Dec 2019 21:22:03 +0100

Thanks again everyone for the participation at the LOOPS side meeting and in our slot at the NWCRG meeting.

I have compiled more detailed notes, with links to/copies of the meeting materials, at

https://github.com/loops-wg/ietf106

Please do have a look at the notes, specifically [1] and [2], and send corrections and additions.  Also, there were a few items in the NWCRG discussion where there may be results that we already can point to; as Marie-Jose said, please send them to both mailing lists.

Grüße, Carsten

[1]: https://github.com/loops-wg/ietf106/blob/master/ietf106-loops-side-meeting.md
[2]: https://github.com/loops-wg/ietf106/blob/master/ietf106-nwcrg-loops.md

---

From: Liyizhou <[email-masked]>
Date: Tue, 17 Dec 2019 04:14:47 +0000

Hi Carsten and all,

Thanks for putting all these together. 

I would like to have a follow-up discussion regarding single retransmission vs. multiple retransmission.

There is a potential choice to design a single retransmission mode *only* mechanism. The reason we might want to do it is the design is easier comparing with a generic retransmission mechanism allowing both single or multiple retransmission by parameter configuration.

For example,  LOOPS egress can mark a lost packet directly as "received" after it feedbacks a few ACKs to indicate a lost packet. Ingress removes the packet from its cache after a single retransmission. Once the retransmitted packet reaches the egress, egress simply forwards it. Such a mechanism only works for single retransmission.  
In this way, egress has a simple logic to decide when to stop sending the indication about a "lost" packet.  And the ingress does not need a logic to control whether and when 2nd and later retransmissions should be performed.

In multiple retransmission, it has to be more carefully designed to sync up when the egress gives up asking for retransmission and  when the ingress can remove the packet from its cache since no more retransmission is required.  As LOOPS does not provide 100% reliability, normally egress side should be the one determines when to give up asking for retransmission. Then egress needs to know the local forward RTT if multiple retransmission is expected.

So single retransmission *only* can be worth considering. It is a design choice more than an implementation choice to me. 
If there is a simple and efficient way to unify both, I would be more than happy to learn.  

Thanks,
Yizhou

-----Original Message-----
From: nwcrg [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Monday, December 16, 2019 4:22 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nwcrg] [LOOPS] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room

Grüße, Carsten

_______________________________________________
nwcrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/nwcrg

---

From: Michael Welzl <[email-masked]>
Date: Tue, 17 Dec 2019 08:23:09 +0100

Hi,

I would say it depends on how much loss we’re optimizing for (and the LOOPS vs. e2e RTT difference; lots of trade-offs here).

One point in favor of the simpler one-retransmission approach: just like a link layer doing ARQ, for its retransmissions, LOOPS will inflate the RTT measurements seen by transport protocols. How bad that really is depends on the transport protocol (e.g., TCP’s RTO is super conservative anyway), on the LOOPS vs. e2e RTT, on how often LOOPS retries…   I’d say there’s reason to hope that retransmitting once is relatively “safe” in that sense.

Cheers,
Michael

> On Dec 17, 2019, at 5:14 AM, Liyizhou <[email-masked]> wrote:
> 
> Hi Carsten and all,
> 
> -- 
> LOOPS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/loops

---

From: Colin Perkins <[email-masked]>
Date: Tue, 17 Dec 2019 11:32:53 +0000

Hi,

Is not the ingress logic the same for single or multiple retransmissions? The ingress node will maintain a queue of packets that have been sent and can be retransmitted. Packets expire from the queue if an ACK for their sequence number is received, or after a certain amount of time. Packets are retransmitted if a retransmission request arrives before they expire. You need the timeout to expire packets in case ACKs are lost, so whether the ingress supports single or multiple retransmissions is just a case of configuring the timeout.

If you somehow communicate that timeout to the egress node, then it can make a local decision on whether to make one or more retransmission requests for a lost packet. This would give the egress flexibility to pick an appropriate retransmission request strategy without needing to standardise what that strategy is. LOOPS would define the mechanism and give guidance on choice of the ingress expiration timeout. 

Colin

> On 17 Dec 2019, at 04:14, Liyizhou <[email-masked]> wrote:
> 
> Hi Carsten and all,
> 

-- 
Colin Perkins
https://csperkins.org/

---

From: "Zhouxingwang (Joe)" <[email-masked]>
Date: Thu, 19 Dec 2019 11:35:11 +0000

Hi,

Statically configuring a timeout (the period that the ingress will keep the packets in its buffer) is a little bit dangerous, because the local RTT may vary and is not known in advance. If the timeout value is shorter than the real local RTT, it will be bad for LOOPS. So setting an N (how many times a lost packet to be retransmitted, once per RTT) in the system is more preferred. 
To simplify the logic, I don’t think setting N at both ingress and egress side is a good idea, because it will make things complicated and N may be set inconsistently by mistake, so setting N and triggering/terminating the retransmission only at one side is better. 

Some thinking of this:
1) Set N and make decision at ingress only
    Pros: ingress knows RTT, it will perform N times retransmission and once per RTT well, and expires the packets after N*RTT from the buffer
    Cons: egress needs to know N and RTT, and still have a logic to stop asking for the lost packets 

2) Set N and make decision at egress only
    Pros: nearly no logic at ingress, egress keeps asking for lost packets by NACK, and after N*RTT (communicated by ingress) it will pretend it has got them and send ACK to ingress to let it clear the packets from retransmission buffer. Ingress just needs a big enough buffer to hold the packets waiting to be retransmitted, and once the buffer is full the new packet just replace the earliest one
    Cons: forward RTT needs to be communicated from ingress to egress

Thanks
Joe

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Colin Perkins
Sent: Tuesday, December 17, 2019 7:33 PM
To: Liyizhou <[email-masked]>
Cc: Carsten Bormann <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [LOOPS] [nwcrg] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room

Hi,

Colin

-- 
Colin Perkins
https://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Thu, 19 Dec 2019 16:50:34 +0000

Joe,

I think you misunderstood my email. 

I did not suggest statically configuring the timeout. The appropriate value clearly depends on the RTT. LOOPS can give guidance on how it relates to the RTT.

I also did not suggest setting N at both ingress and egress. How often to request retransmissions is a purely local decision at the egress. The ingress does not need to know, or care, how often the egress might send retransmission requests.

Colin

> On 19 Dec 2019, at 11:35, Zhouxingwang (Joe) <[email-masked]> wrote:
> 
> Hi,
> 
> From: LOOPS [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of Colin Perkins
> To: Liyizhou <[email-masked] <mailto:[email-masked]>>
> Cc: Carsten Bormann <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/loops <https://www.ietf.org/mailman/listinfo/loops>

---

From: "Zhouxingwang (Joe)" <[email-masked]>
Date: Fri, 20 Dec 2019 01:36:20 +0000

Hi Colin,

Now we synced ☺

It seems egress needs to know the forward RTT to make a good logic of the retransmission requests, the RTT needs to be communicated from ingress, it is non-trivial~~

Thanks
Joe

From: Colin Perkins [mailto:[email-masked]]
Sent: Friday, December 20, 2019 12:51 AM
To: Zhouxingwang (Joe) <[email-masked]>
Cc: Liyizhou <[email-masked]>; Carsten Bormann <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [LOOPS] [nwcrg] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room

Joe,

Colin

On 19 Dec 2019, at 11:35, Zhouxingwang (Joe) <[email-masked]<mailto:[email-masked]>> wrote:

Hi,

Some thinking of this:

Thanks
Joe

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Colin Perkins
Sent: Tuesday, December 17, 2019 7:33 PM
To: Liyizhou <[email-masked]<mailto:[email-masked]>>
Cc: Carsten Bormann <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [LOOPS] [nwcrg] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room

Hi,

Colin

On 17 Dec 2019, at 04:14, Liyizhou <[email-masked]<mailto:[email-masked]>> wrote:

Hi Carsten and all,

Thanks,
Yizhou

-----Original Message-----
From: nwcrg [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Monday, December 16, 2019 4:22 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nwcrg] [LOOPS] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room

Grüße, Carsten

_______________________________________________
nwcrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/nwcrg
--
LOOPS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/loops

--
Colin Perkins
https://csperkins.org/

--
LOOPS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/loops

---

From: "Stuart William Card" <[email-masked]>
Date: Sun, 22 Dec 2019 19:02:21 -0500

Carsten et al --

There has been interesting discussion since this thread-starter, but as I
am not responding specifically to any of the points in those subsequent
messages, I'll reply to this one.

Over 15 years ago, for an enterprise with global air mobile operations, I
was deeply involved in enabling IP networking among LANs aboard aircraft
and WANs on the ground.

Then there were few air-ground links available, and they were mostly:
 - high error (or packet loss) rate
 - low data rate
 - long latency
 - half-duplex (with long transmit-receive turn-around times)
 - asymmetric
 - rapidly and widely varying in all the above characteristics

So we used store-and-forward Delay/Disruption Tolerant Networking (DTN)
techniques were we could, but also had to support TCP/IP applications.

To take advantage of all of the different narrow-band links to/from the
aircraft, we integrated what we called Concurrent Multipath Routing (CMR).
This was not just flow by flow but also packet by packet load balancing
and failover, as often we had only a single application flow demanding a
higher data rate than we could achieve over a single air-ground link, as
well as the need for multi-homing for smooth make-before-break mobility
handoffs. Of course, this resulted in frequent Out Of Order Delivery
(OOOD), utterly trashing performance of basic TCP. Each different
air-ground link was just a single hop, near the aircraft end, of a
multi-hop path, with each ground segment run by a different organization
with different QoS etc. policies. Further, some links had reliable link
layer protocols that used ARQ, causing further RTT variation, so TCP
retransmissions were often needlessly triggered, redundant with (and
slower than) link layer retransmissions.

So we integrated symmetric transparent TCP Performance Enhancing Proxies
(PEPs) based on the Space Communications Protocol Specifications -
Transport Protocol (SCPS-TP), an interoperable TCP variant with additional
(IANA listed) options, especially Selective Negative ACKnowledgment
(SNACK). We implemented and got CCSDS to standardize "Lazy SNACK", where
the transport layer receiver does not immediately SNACK a hole in the
sequence and/or the transport layer sender does not immediately retransmit
in response to getting a SNACK; this more patient strategy dramatically
improved response to CMR's OOOD and the reliable link layer's ARQ.

I relate all this because I wish to suggest that LOOPS participants brush
up on SCPS-TP (the protocol, its testing and its uses), to avoid
unnecessarily re-inventing wheels. I recognize that LOOPS is for path
_segments_: we used SCPS-TP only on a portion of our end-to-end multipath
route, so that is the same; but our TCP<->SCPS-TP PEPs were transparent,
which is largely infeasible today as it looks like a Man In The Middle
attack and is defeated by end-to-end encryption (whether IPsec or TLS).
However, it certainly is possible to tunnel IP in TCP (or SCPS-TP), albeit
with substantial overhead. I'm not suggesting an IP in SCPS-TP tunnel as a
LOOPS technique, but something _like_ that might make some sense. Anyway,
check out the CCSDS standard for SCPS-TP for good ideas on dealing with
lousy path segments. Here are some getting-started references --

https://en.wikipedia.org/wiki/Space_Communications_Protocol_Specifications
https://public.ccsds.org/publications/scps.aspx
https://public.ccsds.org/Pubs/714x0b2.pdf

Although few are aware of SCPS-TP (even those who are using it, as it is
typically embedded in their SATCOM gateways), it is required for all U.S.
military TCP over SATCOM, and there has been recent work outside DoD --

http://www.ittc.ku.edu/resilinets/papers/Nguyen-Sterbenz-wns3-2017.pdf

I can provide lots of other references for anyone interested. We are also
planning to launch a new [secure] reliable [multicast] transport mailing
list to replace the old SCPS and NACK Oriented Reliable Multicast (NORM)
lists (we have the NORM list archives).

Note I added the DTN working group to distribution on this message; you
may or may not want to keep them on distribution of replies.

Thanks all!

-- Stu

> Please do have a look at the notes, specifically [1] and [2], and send
> corrections and additions.  Also, there were a few items in the NWCRG
> discussion where there may be results that we already can point to; as
> Marie-Jose said, please send them to both mailing lists.
>
> Grüße, Carsten
>
> [1]:
> https://github.com/loops-wg/ietf106/blob/master/ietf106-loops-side-meeting.md
> [2]:
> https://github.com/loops-wg/ietf106/blob/master/ietf106-nwcrg-loops.md
>

-- 
Stuart W. Card, VP & Chief Scientist
Critical Technologies Inc. (CTI)
<[email-masked]>