Subject: Re: [EToSat] about LOOPS(Localized Optimization of Path Segments) and etosat

From: Carsten Bormann <[email-masked]>
Date: Wed, 6 Feb 2019 22:43:13 +0100

On Jan 3, 2019, at 18:48, Border, John <[email-masked]> wrote:
> 
> Clearly LOOPS should not focus only on satellite.  But, I think LOOPS is directly relevant to ETOSAT.  Techniques like LOOPS are likely to be a key part of the solution to the performance over satellite problem.
> 
> I will be sending out an email to ETOSAT in the next couple of days to try to kickstart discussion overall.

Here is my attempt at kickstarting:

Traditional approaches at enhancing end-to-end performance by deploying nodes on the way (performance enhancing proxies, PEPs) have employed heavy interactions with the end-to-end transport protocols, usually without their explicit consent.  With end-to-end encryption going under the transport (QUIC), this approach is no longer useful.  This is the underlying theme of the ETOSAT mailing list, but also an important motivator for the more general LOOPS approach.  (Please use the slides previously sent as a reference to node terminology.)

So what kind of performance enhancement can we do even without aggressively reaching into transport layers?

* Active queue management: AQM is certainly a good idea.  A potential benefit of LOOPS for AQM is that it can run measurements between sub-path ingress and sub-path egress nodes, as well as between overlay ingress and overlay egress.  So more data can be made available inside the system of LOOPS nodes, and this may help in active queue management.

* Loss recovery.  Packets can be recovered between sub-path ingress and sub-path egress nodes.  This can be based on local retransmissions and/or on adding and using FEC.  By cooperation of the sub-path ingress and egress nodes, the parameters for these (RTO, FEC rate, choice between retransmission and FEC) can be chosen wisely.  Overlay ingress and egress nodes can provide additional information, such as end-to-end RTT estimates, which might also help in choosing these parameters and avoiding continuing with sub-path retransmission when an end-to-end retransmission is likely to kick in.  The total load on the network decreases with replacing end-to-end by subpath retransmissions/FEC.  Tail-loss detection can benefit from mixing multiple end-to-end flows into one error-controlled subpath flow.

* Multipathing.  Again, both the overlay ingress/egress nodes and the sub-path ingress/egress nodes can help with detecting multiple paths and using them in the best possible way.

Challenges, and potential mitigations by employing advances in end-to-end transports, include:

* Reflecting congestion information up from the sub-paths to the end-to-end paths.  Sub-path ingress/egress nodes may employ additional congestion indicators such as sojourn time to sort loss events into congestion or non-congestion losses.

* Reordering may reduce end-to-end transport performance.  RACK may help us here, as may cwnd undo on spurious retransmission detection in current TCPs.  Or we could use resequencing in egress nodes, but this is generally not a favored approach.

* MTU.

Traditionally, satellite PEPs have treated the whole satellite section (uplink, downlink) as a single sub-path (hop) for optimization.  LOOPS would benefit from an active LOOPS node on the satellite.  Is that something that is on the table these days?

What we could design here includes:

* Sub-path “transport-layer style“ protocols, for retransmission and FEC.  This includes designing/selecting good FEC schemes for sub-path usage, as well as some basic encapsulation formats.
* Formats for making measurement information available between the nodes.
* Formats for making control information available between ingress and egress nodes.

Grüße, Carsten

---

From: "Border, John" <[email-masked]>
Date: Thu, 7 Feb 2019 17:50:08 +0000

You wrote:

"Traditionally, satellite PEPs have treated the whole satellite section (uplink, downlink) as a single sub-path (hop) for optimization.  LOOPS would benefit from an active LOOPS node on the satellite.  Is that something that is on the table these days?"

Do you mean on the satellite itself?   There have been a few satellites built with processing capability on board (mostly for government/military use) but that is not currently happening for the high speed satellites being used for Internet over satellite.

Some of the elements of LOOPS already exist for satellite links but they need to be beefed up now because transport layer solutions are disappearing.  One area that concerns me a lot is loss recovery.  While the error rate on the satellite paths is generally very low most of the time, but gets higher during link condition fades (e.g. heavy rain).  Satellite technology has come a long way in using adaptive techniques to compensate for fade but there is always some residual error rate.  And, from the outside, the mitigation techniques will look like a reduction in link capacity which ties in to the other LOOPS aspects of dealing with congestion management.

John

-----Original Message-----
From: Carsten Bormann <[email-masked]>
Sent: Wednesday, February 06, 2019 4:43 PM
To: [email-masked]
Cc: Marie-Jose Montpetit <[email-masked]>; Border, John <[email-masked]>
Subject: Re: about LOOPS(Localized Optimization of Path Segments) and etosat

On Jan 3, 2019, at 18:48, Border, John <[email-masked]<mailto:[email-masked]>> wrote:

* MTU.

Grüße, Carsten

---

From: Marie-Jose Montpetit <[email-masked]>
Date: Thu, 7 Feb 2019 13:37:43 -0500

I think this work my loop back (yes pun intended) in some of the work we are doing on adding FEC/Network Coding to QUIC which would help the error recovery without waiting for the long retransmission. And of couse the work we are proposing in COIN where we want to add processing on network paths. So stay tuned.

And of course when we were doing TCP over Satellite and other fu-over satellite work way back when we touched a lot of the topics Carsten raised: how to design a network that will survive the yes sometimes adverse conditions of high frequency satellite systems (Ka band for those of you who want to know) where the usual random packet loss hypothesis in the Internet meet deep fades, in fact beyond what a typical error recovery can deal with. Hence yes multi-path is a good idea but not sure we can always count on the 2nd path when only a satellite network is available (use satellite diversity using double coverage?). 

As for the PEPs satellite networks are not the only ones using them. But with the new transports and e-2-e encryption this becomes potentially not feasible. Which is also an issue when you want to add computing in the network hence reliance on “trusted nodes” or even other distributed network concepts (blockchains?). 

Lots to think about…

mjm
 
Marie-Jose Montpetit, Ph.D.
[email-masked]
[email-masked]
+1-781-526-2661
@SocialTVMIT

> On Feb 7, 2019, at 12:50 PM, Border, John <[email-masked]> wrote:
> 
>  
> You wrote:
>  
> From: Carsten Bormann <[email-masked] <mailto:[email-masked]>> 
> To: [email-masked] <mailto:[email-masked]>
> Cc: Marie-Jose Montpetit <[email-masked] <mailto:[email-masked]>>; Border, John <[email-masked] <mailto:[email-masked]>>
> On Jan 3, 2019, at 18:48, Border, John <[email-masked] <mailto:[email-masked]>> wrote:
> >

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 7 Feb 2019 20:03:10 +0100

On Feb 7, 2019, at 19:37, Marie-Jose Montpetit <[email-masked]> wrote:
> 
> I think this work my loop back (yes pun intended) in some of the work we are doing on adding FEC/Network Coding to QUIC which would help the error recovery without waiting for the long retransmission.

Right, when the transport actually helps with this you can get even better results.
But then, LOOPS can send the FEC just along the problematic sub-paths, so the ones without the equivalent of rain-fade are not burdened.

> And of couse the work we are proposing in COIN where we want to add processing on network paths. So stay tuned.

Exactly, COIN would be one of the ways to get the in-network processing for this (for now, this is all about manually set-up nodes).

> As for the PEPs satellite networks are not the only ones using them. But with the new transports and e-2-e encryption this becomes potentially not feasible. Which is also an issue when you want to…

Indeed, pretty much a lifetime of research could go into this…
The aim of LOOPS is to recast and limit the problem in such a way that we can address a good chunk of it with low-hanging fruit that can be deployed promptly.

I’m pretty sure that nwcrg has some of this low-hanging fruit already plucked, so that’s why I’m going to ask there to contribute here.

Grüße, Carsten

> _______________________________________________
> EToSat mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/etosat

---

From: Kuhn Nicolas <[email-masked]>
Date: Fri, 8 Feb 2019 14:26:26 +0000

Dear all,

Thanks for putting this up - this indeed could help structure many activities related to the management and optimization of tunnels. 
I just wanted to point out that losses due to rain-fade do not happen much when ACM is correctly tuned. 

That being said, we may have non neglectable Wi-Fi losses for an end user on a phone. We can expect important QoE degradations in this context. 
At the moment, the low-hanging fruits are still to be defined, but IMHO trying to adapt QUIC recovery to this use-case is important (for QUIC v2?).

I would be happy to further discuss that and want to contribute (with presentations and/or drafts). 

Cheers,

Nico

-----Message d'origine-----
De : EToSat <[email-masked]> De la part de Carsten Bormann
Envoyé : jeudi 7 février 2019 20:03
À : Marie-Jose Montpetit <[email-masked]>
Cc : [email-masked]; Border, John <[email-masked]>
Objet : Re: [EToSat] about LOOPS(Localized Optimization of Path Segments) and etosat

> As for the PEPs satellite networks are not the only ones using them. But with the new transports and e-2-e encryption this becomes potentially not feasible. Which is also an issue when you want to…

Indeed, pretty much a lifetime of research could go into this… The aim of LOOPS is to recast and limit the problem in such a way that we can address a good chunk of it with low-hanging fruit that can be deployed promptly.

Grüße, Carsten

>> Traditional approaches at enhancing end-to-end performance by 
>> deploying nodes on the way (performance enhancing proxies, PEPs) have 
>> employed heavy interactions with the end-to-end transport protocols, 
>> usually without their explicit consent.  With end-to-end encryption 
>> going under the transport (QUIC), this approach is no longer useful.  
>> This is the underlying theme of the ETOSAT mailing list, but also an 
>> important motivator for the more general LOOPS approach.  (Please use 
>> the slides previously sent as a reference to node terminology.)
>>

---

From: "Border, John" <[email-masked]>
Date: Fri, 8 Feb 2019 17:21:43 +0000

I think we want to look at FEC both within the transport and under the transport.  FEC under the transport is easier to apply because it is tied to the specific links which need it.  But, it does not catch all of the losses.  Within the transport can catch all of the losses but its use all of the time is too much overhead when it is not needed.  So, there needs to be learning for when to apply it.  (RTT is a good measure to use, of course.)  As I understand Ian's current FEC for QUIC proposal, its use would be negotiated.  I am hoping we can find a way to start using it in the middle when the need for it is detected.  Using it only on the paths that need it (with multi-path QUIC) would also be good...

John

-----Original Message-----
From: Kuhn Nicolas <[email-masked]> 
Sent: Friday, February 08, 2019 9:26 AM
To: 'Carsten Bormann' <[email-masked]>; Marie-Jose Montpetit <[email-masked]>
Cc: [email-masked]; Border, John <[email-masked]>; [email-masked]' <[email-masked]>
Subject: RE: [EToSat] about LOOPS(Localized Optimization of Path Segments) and etosat

Dear all,

Cheers,

Nico

-----Message d'origine-----
De : EToSat <[email-masked]> De la part de Carsten Bormann Envoyé : jeudi 7 février 2019 20:03 À : Marie-Jose Montpetit <[email-masked]> Cc : [email-masked]; Border, John <[email-masked]> Objet : Re: [EToSat] about LOOPS(Localized Optimization of Path Segments) and etosat

Grüße, Carsten

> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> man_listinfo_etosat&d=DwIGaQ&c=dIKa1mMv92xhhFzVXv5A3Q&r=9F44ji63_2hvW5
> HufmlpP-DFKXuFy4jDtL5PXwKlTqg&m=_Q1nPO2pJ10BBlxW3l-906jg0TjJC88HgcGQHz
> z5fGU&s=b-A2yE8Y5rqn_3ZLDrkuQY_9JWf6OaNwvZ3P74LlYxY&e=

_______________________________________________
EToSat mailing list
[email-masked]
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_etosat&d=DwIGaQ&c=dIKa1mMv92xhhFzVXv5A3Q&r=9F44ji63_2hvW5HufmlpP-DFKXuFy4jDtL5PXwKlTqg&m=_Q1nPO2pJ10BBlxW3l-906jg0TjJC88HgcGQHzz5fGU&s=b-A2yE8Y5rqn_3ZLDrkuQY_9JWf6OaNwvZ3P74LlYxY&e=

---

From: Ludovic THOMAS <[email-masked]>
Date: Mon, 11 Feb 2019 08:48:14 +0100

Maybe the research should include a topic on the throughput 
initialization, namely the exponential probing of the link capacity 
(shall it be called "slow start" in NewReno, "hystart" in Cubic or 
"binary search" in BBR).

FWIU, deciding to increase the throughput is today a sole decision of 
the transport protocol at the endpoint.
The exponential probing of a link with high RTT is yet long and 
drastically reduces the web browsing performances (page load time, time 
to paint, ...)

With TCP, the gateway PEP was able to generate ACKs, allowing the server 
to only probe the (low RTT) link connecting it to the gateway. With 
QUIC, the decision to increase the throughput is performed only based on 
(encrypted) ACKs generated by the remote client (at a high RTT distance 
in a SATCOM context). LOOPS overlay can't today participate in that 
decision.

To put it in a nutshell, re-transmissions and FECs mechanisms under the 
transport layer are about how to maintain the best (and fair) use of the 
resources even if transmission capacity is suddenly reduced by weather 
conditions. But we may ask how to initially reach this best use in an 
efficient manner, starting from scratch.

FWIU, solutions to improve the slow start without cutting a QUIC 
connection will require cross-layer interactions.
I'm thinking maybe some overlay nodes can participate in the negotiation 
of the transport initial parameters during the handshake, so that 
end-points know they communicate over a high-BDP yet LOOPS-enabled path.

It would raises lot of questions, such as how a QUIC endpoint can trust 
a LOOPS overlay node...

Ludovic

Le 08/02/2019 à 15:26, Kuhn Nicolas a écrit :
> Dear all,
>
> Thanks for putting this up - this indeed could help structure many
> activities related to the management and optimization of tunnels.
> That being said, we may have non neglectable Wi-Fi losses for an end user on
> a phone. We can expect important QoE degradations in this context.
> At the moment, the low-hanging fruits are still to be defined, but IMHO
> trying to adapt QUIC recovery to this use-case is important (for QUIC v2?).
>
>> I think this work my loop back (yes pun intended) in some of the work we
>> are doing on adding FEC/Network Coding to QUIC which would help the error
>> recovery without waiting for the long retransmission.
> But then, LOOPS can send the FEC just along the problematic sub-paths, so
> the ones without the equivalent of rain-fade are not burdened.
>
>> And of couse the work we are proposing in COIN where we want to add
>> processing on network paths. So stay tuned.
> Exactly, COIN would be one of the ways to get the in-network processing for
> this (for now, this is all about manually set-up nodes).
>
>> And of course when we were doing TCP over Satellite and other fu-over
>> satellite work way back when we touched a lot of the topics Carsten
>> raised: how to design a network that will survive the yes sometimes
>> adverse conditions of high frequency satellite systems (Ka band for those
>> of you who want to know) where the usual random packet loss hypothesis in
>> the Internet meet deep fades, in fact beyond what a typical error recovery
>> can deal with. Hence yes multi-path is a good idea but not sure we can
>> always count on the 2nd path when only a satellite network is available
>> (use satellite diversity using double coverage?).
>>
>> As for the PEPs satellite networks are not the only ones using them. But
>> with the new transports and e-2-e encryption this becomes potentially not
>> feasible. Which is also an issue when you want to add computing in the
>> network hence reliance on “trusted nodes” or even other distributed
>> network concepts (blockchains?).
>>
>> Lots to think about…
> Indeed, pretty much a lifetime of research could go into this… The aim of
> LOOPS is to recast and limit the problem in such a way that we can address a
> good chunk of it with low-hanging fruit that can be deployed promptly.
>
> I’m pretty sure that nwcrg has some of this low-hanging fruit already
> plucked, so that’s why I’m going to ask there to contribute here.
>
> Grüße, Carsten
>
>
>> mjm
>>
>>> "Traditionally, satellite PEPs have treated the whole satellite section
>>> (uplink, downlink) as a single sub-path (hop) for optimization.  LOOPS
>>> would benefit from an active LOOPS node on the satellite.  Is that
>>> something that is on the table these days?"
>>>
>>> Do you mean on the satellite itself?   There have been a few satellites
>>> built with processing capability on board (mostly for government/military
>>> use) but that is not currently happening for the high speed satellites
>>> being used for Internet over satellite.
>>>
>>> Some of the elements of LOOPS already exist for satellite links but they
>>> need to be beefed up now because transport layer solutions are
>>> disappearing.  One area that concerns me a lot is loss recovery.  While
>>> the error rate on the satellite paths is generally very low most of the
>>> time, but gets higher during link condition fades (e.g. heavy rain).
>>> Satellite technology has come a long way in using adaptive techniques to
>>> compensate for fade but there is always some residual error rate.  And,
>>> from the outside, the mitigation techniques will look like a reduction in
>>> link capacity which ties in to the other LOOPS aspects of dealing with
>>> congestion management.
>>>
>>>
>>> John
>>>
>>>
>>>> Clearly LOOPS should not focus only on satellite.  But, I think LOOPS
>>>> is directly relevant to ETOSAT.  Techniques like LOOPS are likely to be
>>>> a key part of the solution to the performance over satellite problem.
>>>>
>>> So what kind of performance enhancement can we do even without
>>> aggressively reaching into transport layers?
>>>
>>> * Active queue management: AQM is certainly a good idea.  A potential
>>> benefit of LOOPS for AQM is that it can run measurements between sub-path
>>> ingress and sub-path egress nodes, as well as between overlay ingress and
>>> overlay egress.  So more data can be made available inside the system of
>>> LOOPS nodes, and this may help in active queue management.
>>>
>>> * Loss recovery.  Packets can be recovered between sub-path ingress and
>>> sub-path egress nodes.  This can be based on local retransmissions and/or
>>> on adding and using FEC.  By cooperation of the sub-path ingress and
>>> egress nodes, the parameters for these (RTO, FEC rate, choice between
>>> retransmission and FEC) can be chosen wisely.  Overlay ingress and egress
>>> nodes can provide additional information, such as end-to-end RTT
>>> estimates, which might also help in choosing these parameters and
>>> avoiding continuing with sub-path retransmission when an end-to-end
>>> retransmission is likely to kick in.  The total load on the network
>>> decreases with replacing end-to-end by subpath retransmissions/FEC.
>>> Tail-loss detection can benefit from mixing multiple end-to-end flows
>>> into one error-controlled subpath flow.
>>>
>>> * Multipathing.  Again, both the overlay ingress/egress nodes and the
>>> sub-path ingress/egress nodes can help with detecting multiple paths and
>>> using them in the best possible way.
>>>
>>> * Reflecting congestion information up from the sub-paths to the
>>> end-to-end paths.  Sub-path ingress/egress nodes may employ additional
>>> congestion indicators such as sojourn time to sort loss events into
>>> congestion or non-congestion losses.
>>>
>>> * Reordering may reduce end-to-end transport performance.  RACK may help
>>> us here, as may cwnd undo on spurious retransmission detection in current
>>> TCPs.  Or we could use resequencing in egress nodes, but this is
>>> generally not a favored approach.
>>>
>>> * MTU.
>>>
>>> Traditionally, satellite PEPs have treated the whole satellite section
>>> (uplink, downlink) as a single sub-path (hop) for optimization.  LOOPS
>>> would benefit from an active LOOPS node on the satellite.  Is that
>>> something that is on the table these days?
>>>
>>> * Sub-path “transport-layer style“ protocols, for retransmission and FEC.
>>> This includes designing/selecting good FEC schemes for sub-path usage, as
>>> well as some basic encapsulation formats.

---

From: Ludovic THOMAS <[email-masked]>
Date: Fri, 8 Feb 2019 11:23:27 +0100

FWIU, solutions to improve the slow start without cutting a QUIC 
connection will require cross-layer interactions.
I'm thinking maybe some overlay nodes can participate in the negotiation 
of the transport initial parameters during the handshake, so that 
end-points know they communicate over a high-BDP yet LOOPS-enabled path, 
such as proposed in https://github.com/quicwg/base-drafts/issues/2091

Ludovic

---

From: Jeremy Harris <[email-masked]>
Date: Mon, 11 Feb 2019 16:20:44 +0000

> connection will require cross-layer interactions.

Strawman: an advisory notification from the LOOPS endpoint,
on connection startup, with bandwidth and delay data, intended
for use by the TCP endpoint in setting an early value for CWND
(or the bbr equivalent).  Possibly an ICMP?
-- 
Cheers,
  Jeremy

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 12 Feb 2019 08:50:04 +0100

> (or the bbr equivalent).  Possibly an ICMP?

Well, initial CWND must be chosen so that it doesn’t cause damage on any node on the entire end-to-end path.  A single LOOPS midpoint node can’t know all the information needed to provide such an assurance.

Getting ahead information from the path was thought through in RFC 4872 Quick-Start.
This spec is not very deployable today because it relies on IP options.
But even if that were not a problem, getting consensus from all the nodes on the path would still be hard, and would violate the deployment economy maxim of providing incremental benefit.
(LOOPS could actually help here by taking sub-paths managed by LOOPS out of the picture, but still the other nodes would have to deploy.)

TCP splitting for satellites works because the ramp-up in the low-RTT first (pre-satellite) path segment is independent from the ramp-up in the following ones; the splitting node simply promises to provide any buffer space needed (which, worst case, can cause buffer bloat).  We would need to provide LOOPS mid-point nodes with ways to manage congestion up to the receiving endpoint to achieve the same effect, and that is not possible without a close interaction with the end-to-end transport (or global deployment of mid-to-end LOOPS, which is not even on the agenda currently).

Grüße, Carsten