Subject: [EToSat] Packet lost on last mile network?

From: Nitay Argov <[email-masked]>
Date: Wed, 13 May 2020 03:27:48 +0000

Hi,
I'm looking for discussions on issues related to a satellite network that is used with a lossy end-user delivery network - e.g. WiFi. As far as I understand, once QUIC is used, any packet drop by the user network will need to be retransmitted over the satellite? Doesn't this pose a real problem for using QUIC over such networks?
Am I missing something? I am sure this issue was discussed in the past but I could not find where.
Thanks!
IMPORTANT - This email and any attachments is intended for the above named addressee(s), and may contain information which is confidential or privileged. If you are not the intended recipient, please inform the sender immediately and delete this email: you should not copy or use this e-mail for any purpose nor disclose its contents to any person.

---

From: Kuhn Nicolas <[email-masked]>
Date: Wed, 13 May 2020 06:44:43 +0000

Hi,

IMHO, you are not missing much, there is an issue when you have a lossy end-user delivery network (unless QUIC supports an unreliable transmission).
Moreover, you may also have congestion losses in the aggregation network of the satellite system.
To consider this issue, we have proposed a specific use-case in the QUIC4SAT draft : https://tools.ietf.org/html/draft-kuhn-quic-4-sat-04
There are multiple ways of solving this issue: adding coding in QUIC (draft-swett-nwcrg-coding-for-quic-04), enabling local retransmissions (e.g. with MASQUE approach) or having an "aggressive" congestion control that ignore loss as a congestion signal, etc.
There are also some IETF presentations on this issue:
https://datatracker.ietf.org/meeting/105/materials/slides-105-panrg-quic-over-in-sequence-paths-with-different-characteristics-00
https://datatracker.ietf.org/meeting/105/materials/slides-105-panrg-quic-over-satellite-00

I hope this helps,

Nico

De : EToSat <[email-masked]> De la part de Nitay Argov
Envoyé : mercredi 13 mai 2020 05:28
À : [email-masked]
Objet : [EToSat] Packet lost on last mile network?

Hi,

---

From: Nitay Argov <[email-masked]>
Date: Wed, 13 May 2020 06:52:47 +0000

Thanks!!
I will study the suggested resources.

From: Kuhn Nicolas [mailto:[email-masked]]
Sent: יום ד 13 מאי 2020 09:45
To: Nitay Argov <[email-masked]>; [email-masked]
Subject: RE: Packet lost on last mile network?

Hi,

IMHO, you are not missing much, there is an issue when you have a lossy end-user delivery network (unless QUIC supports an unreliable transmission).
Moreover, you may also have congestion losses in the aggregation network of the satellite system.
To consider this issue, we have proposed a specific use-case in the QUIC4SAT draft : https://tools.ietf.org/html/draft-kuhn-quic-4-sat-04<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-kuhn-quic-4-sat-04&data=02%7C01%7CNitay%40gilat.com%7C1a1932285f114bf8688308d7f70922c6%7C7300b1a3573a401092a61c65cd85e927%7C0%7C0%7C637249490926514767&sdata=KCAaxn7ksxIBuMPDzAvSMJ0WIZKpPk%2FdUa5Sz15P3Vo%3D&reserved=0>
There are multiple ways of solving this issue: adding coding in QUIC (draft-swett-nwcrg-coding-for-quic-04), enabling local retransmissions (e.g. with MASQUE approach) or having an “aggressive” congestion control that ignore loss as a congestion signal, etc.
There are also some IETF presentations on this issue:
https://datatracker.ietf.org/meeting/105/materials/slides-105-panrg-quic-over-in-sequence-paths-with-different-characteristics-00<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F105%2Fmaterials%2Fslides-105-panrg-quic-over-in-sequence-paths-with-different-characteristics-00&data=02%7C01%7CNitay%40gilat.com%7C1a1932285f114bf8688308d7f70922c6%7C7300b1a3573a401092a61c65cd85e927%7C0%7C0%7C637249490926514767&sdata=aHwqMqos3IwCuw3bCHBjV8ZgeOYbz2XcHHs7XbN9f1g%3D&reserved=0>
https://datatracker.ietf.org/meeting/105/materials/slides-105-panrg-quic-over-satellite-00<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fmeeting%2F105%2Fmaterials%2Fslides-105-panrg-quic-over-satellite-00&data=02%7C01%7CNitay%40gilat.com%7C1a1932285f114bf8688308d7f70922c6%7C7300b1a3573a401092a61c65cd85e927%7C0%7C0%7C637249490926524767&sdata=9v71nkMR%2FvpY%2BJi8zyaX8H6nixVnhx8yfKjWz7HYG2E%3D&reserved=0>

I hope this helps,

Nico

De : EToSat <[email-masked]<mailto:[email-masked]>> De la part de Nitay Argov
Envoyé : mercredi 13 mai 2020 05:28
À : [email-masked]<mailto:[email-masked]>
Objet : [EToSat] Packet lost on last mile network?

Hi,
I'm looking for discussions on issues related to a satellite network that is used with a lossy end-user delivery network – e.g. WiFi. As far as I understand, once QUIC is used, any packet drop by the user network will need to be retransmitted over the satellite? Doesn't this pose a real problem for using QUIC over such networks?
Am I missing something? I am sure this issue was discussed in the past but I could not find where.
Thanks!
IMPORTANT - This email and any attachments is intended for the above named addressee(s), and may contain information which is confidential or privileged. If you are not the intended recipient, please inform the sender immediately and delete this email: you should not copy or use this e-mail for any purpose nor disclose its contents to any person.
IMPORTANT - This email and any attachments is intended for the above named addressee(s), and may contain information which is confidential or privileged. If you are not the intended recipient, please inform the sender immediately and delete this email: you should not copy or use this e-mail for any purpose nor disclose its contents to any person.

---

From: Christian Huitema <[email-masked]>
Date: Wed, 13 May 2020 08:49:49 -0700

I am running simulations of that lossy configuration as part of the Picoquic test suite. When using BBR, I observe that the large transfer over a high speed link takes a couple seconds more than in the absence of losses. The issue is not congestion, and not even the 1.6% bandwidth consumed by error correction. The problem is the cascade: some of the corrections of lost packets will be lost again, which causes several rounds of retransmission at the end of the transfer.

-- Christian Huitema 

> On May 12, 2020, at 11:52 PM, Nitay Argov <[email-masked]> wrote:
> 
> 
> Thanks!!
> IMPORTANT - This email and any attachments is intended for the above named addressee(s), and may contain information which is confidential or privileged. If you are not the intended recipient, please inform the sender immediately and delete this email: you should not copy or use this e-mail for any purpose nor disclose its contents to any person. _______________________________________________
> EToSat mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/etosat

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 20 May 2020 11:27:11 +0200

> I am running simulations of that lossy configuration as part of the Picoquic test suite. When using BBR, I observe that the large transfer over a high speed link takes a couple seconds more than in…

Hi Christian,

Since you have data about these cascades, I’ll ask you my curious question:  What is the timing of these?  How long would the WiFi segment had time to do a local retransmission?

Of course, I’m asking this in the context of LOOPS ([email-masked]), which is meeting next week in order to nail down its initial charter proposal; support for WiFi “last miles” has always been proposed as a possible subject.

Grüße, Carsten

>> IMPORTANT - This email and any attachments is intended for the above named addressee(s), and may contain information which is confidential or privileged. If you are not the intended recipient, please inform the sender immediately and delete this email: you should not copy or use this e-mail for any purpose nor disclose its contents to any person._______________________________________________
>> EToSat mailing list
>> [email-masked]

---

From: Christian Huitema <[email-masked]>
Date: Wed, 20 May 2020 09:05:37 -0700

> Since you have data about these cascades, I’ll ask you my curious question:  What is the timing of these?  How long would the WiFi segment had time to do a local retransmission?

The simulation sets a satellite link, 250 Mbps down, 3 Mbps up, 300ms
latency each way. The packet size is 1500 bytes. The 600ms RTT
translates to a 1sec timer, approximately. In 1 second, the link carries
more than 20,000 packets. If 1.6% are lost, that means 320 need to be
retransmitted. Out of those 320, 5 or 6 will need to be retransmitted a
second time, incurring an additional 2 seconds delay.

Note that this does not consume much bandwidth. The link is available
for transmission of further documents. It is strictly a "latency of
document transfer" issue. If you are piping a large set of large files
over the network, that's hardly a problem, but in some cases it is
important.

> Of course, I’m asking this in the context of LOOPS ([email-masked]), which is meeting next week in order to nail down its initial charter proposal; support for WiFi “last miles” has always been pro…

A lot could be done in improving the behavior of Wi-Fi, that's sure. But
I really don't like the idea of putting proxies everywhere -- the cost
in management and decreased reliability would be enormous. Besides, the
deployment issues would be daunting -- who is going to set up al these
proxies in all these Wi-Fi networks? And, if they are only deployed in a
fraction of the Wi-Fi network, then the transport folks will still need
an end-to-end solution.

I have not yet explored the end-to-end solutions to the latency problem,
but I could. FEC would cure the latency issue, but FEC trades bandwidth
for latency, so you don't want that in the general case. If I had to do
something, I would explore "selective FEC". Detect when the document is
almost done, when there is only 1 second of transmission left. During
that 1 second, apply something like 1/8 FEC for the remaining stream
frames. That would make the last 1 second 12.5% less efficient, but
would limit the "cascade' effect to 125ms.

-- Christian Huitema

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 20 May 2020 18:56:51 +0200

Hi Christian,

Thank you for the numbers.

> second time, incurring an additional 2 seconds delay.

Right.  You don’t say what the RTT is on the WiFi part, but I’d assume that is lower than those 1 s.  So local recovery would have had ample time.

> in management and decreased reliability would be enormous.

I’m not sure where the proxies would come in — a last-mile deployment of LOOPS would look more like a VPN.  (And I don’t want to talk down the deployment work needed here, but we do know how to deploy VPNs.)

> Besides, the
> proxies in all these Wi-Fi networks?

(People who want to have good WiFi networks?)

> And, if they are only deployed in a

They sure will.  But that never can be as good as local recovery.

> would limit the "cascade' effect to 125ms.

FEC packets as a replacement for tail probe.  Makes a lot of sense to me.

Grüße, Carsten

---

From: Roland Zink <[email-masked]>
Date: Wed, 20 May 2020 21:07:42 +0200

See inline

> I’m not sure where the proxies would come in — a last-mile deployment of LOOPS would look more like a VPN.  (And I don’t want to talk down the deployment work needed here, but we do know how to dep…
It can be part of the lower layers like mobile networks (LTE, 5G) are 
doing this. Doesn't WiFi also do ARQ?
> They sure will.  But that never can be as good as local recovery.
There are also potential issues from reordering packets from local 
retransmissions or FEC triggering end-to-end retransmission, although 
QUIC shouldn't react too fast on it.
> FEC packets as a replacement for tail probe.  Makes a lot of sense to me.
Local FEC will not apply the overhead end-to-end.
>
> Grüße, Carsten
>
> _______________________________________________
> EToSat mailing list
> [email-masked]

Grüße, Roland

---

From: Christian Huitema <[email-masked]>
Date: Wed, 20 May 2020 11:09:51 -0700

> They sure will.  But that never can be as good as local recovery.

In a nutshell, that's the entire discussion of end-to-end vs middle-boxes.

It is always possible to do a local optimization by adding some
intelligence in the network. That's the case for pretty much all middle
boxes, and it seems that the proxies envisaged in Loop fall into that
category. But there are always downsides, such as management,
flexibility, and most importantly ossification by old proxies prone to
bit rot.

Given effort, it is almost always possible to find end-to-end solutions
that are "good enough", getting you within 90% of the optimum or some
such. FEC as tail probe would be one example of that. The end-to-end
solutions also tend to get better over time, as implementers devise more
clever algorithms and update the code. All it takes is some motivation
that the effort is important.

An example of that would be support for cellular networks. Twenty years
ago, this was a niche, and TCP was not optimized for it. Hence the
deployment of middle-boxes in the cellular networks. But as this became
an ever more important scenario, TCP improved, and QUIC will improve
over that. The cellular middle-boxes become harder to justify, and will
start moving from the "optimization" to the "hindrance" category.

Satellite networks are a peculiar case. They remain very much a niche
application to this day, something like a communication media of last
resort in places that fiber or cellular cannot reach. Given the niche
status, it is unclear whether end-to-end transports will bother
implementing optimized algorithms, even when these are available. But it
is also unclear whether "mainline" networks will bother deploying
middle-boxes solely for the benefit of the satellite niche, which kinds
of rules out solutions like 'deploy a box in every Wi-Fi network'.
'Deploy a proxy at the ground station' might work, because the satellite
operator has the right business incentives, but of course that requires
nodes to still do something special for satellites, i.e., discover the
proxy and use it. Not obvious.

Hence my effort to look at end-to-end solutions covering not just the
satellite case, but also other long distance networks such as
intercontinental connections. These are still niches today, given VPN,
but they are still useful for P2P applications. Given enough niches,
there might be enough motivation for doing the effort.

-- Christian Huitema

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 20 May 2020 21:52:38 +0200

On 2020-05-20, at 20:09, Christian Huitema <[email-masked]> wrote:
> middle boxes,

> proxies

I’m not sure we are talking about the same things here, as LOOPS uses neither (it uses routers and tunnels).

But I agree that if there is a 80 % solution in the end-to-end protocol, go for it.

It just isn’t something a network provider can do on their own.

Grüße, Carsten

---

From: Roland Zink <[email-masked]>
Date: Wed, 20 May 2020 22:03:20 +0200

Am 20.05.2020 um 20:09 schrieb Christian Huitema:
> intelligence in the network. That's the case for pretty much all 
> middle boxes, and it seems that the proxies envisaged in Loop fall 
> into that category. But there are always downsides, such as 
> management, flexibility, and most importantly ossification by old 
> proxies prone to bit rot.
>
yes
>
> Given effort, it is almost always possible to find end-to-end 
> solutions that are "good enough", getting you within 90% of the 
> optimum or some such. FEC as tail probe would be one example of that. 
> The end-to-end solutions also tend to get better over time, as 
> implementers devise more clever algorithms and update the code. All it 
> takes is some motivation that the effort is important.
>
This is the question. Is there enough motivation for the 0.x% case?
>
> An example of that would be support for cellular networks. Twenty 
> years ago, this was a niche, and TCP was not optimized for it. Hence 
> the deployment of middle-boxes in the cellular networks. But as this 
> became an ever more important scenario, TCP improved, and QUIC will 
> improve over that. The cellular middle-boxes become harder to justify, 
> and will start moving from the "optimization" to the "hindrance" category.
>
The cellular boxes are handling radio, mobility and things like local 
retransmits and FEC. So is the situation now better because of improved 
TCP or because of improved mobile network boxes? Probably both, but you 
are probably only talking about PEPs. Those may go away but they are 
also built into those mobile network boxes doing the other stuff like 
eNodeBs for example.
> implementing optimized algorithms, even when these are available. But 
> it is also unclear whether "mainline" networks will bother deploying 
> middle-boxes solely for the benefit of the satellite niche, which 
> kinds of rules out solutions like 'deploy a box in every Wi-Fi 
> network'. 'Deploy a proxy at the ground station' might work, because 
> the satellite operator has the right business incentives, but of 
> course that requires nodes to still do something special for 
> satellites, i.e., discover the proxy and use it. Not obvious.
>
My fear is that niches are served late or not at all. The local network 
operator has a high interest in improving the situations but at global 
scale the end-2-end solution for it may not provide enough motivation. 
Assuming not all nices faces the same problem which can be solved then 
for all niches. Also solutions may need to be applied to all 
applications, e.g. if it is fixed in application a it still may need 
attention by application b. Applications may bit rot as well.

Anyway if you manage to build an end-to-end solution which does work for 
all sort of niches than that would be great. Please continue to work on 
an end-to-end solution.

> -- Christian Huitema
Regards,

Roland

---

From: "Border, John" <[email-masked]>
Date: Wed, 20 May 2020 21:38:40 +0000

Sigh!  The “niche” perception is a constant battle for satellite networks.  There are millions of users but that is small when compared to the overall size of the Internet.  (This could spin into a digital divide discussion but that doesn’t seem appropriate for an IETF mailing list.)  One of the advantages of a satellite network is the management aspects since the remote site terminals are right next to the customers.  On the other, avoiding ossification takes conscious effort.

One of the reasons why we look at in the network solutions is that we are in the network.  Tuning user devices is often possible to improve performance.  But, except for customers with experts, it is generally not feasible.  What we are looking for is end to end mechanisms that can adapt themselves.  I think QUIC has better hooks to do this than TCP did.  But, we have the problem of incentivizing the developers of the “ends” to do it.  I have been thinking about that problem for a while now but I still don’t have any good answers…

John

From: EToSat <[email-masked]> On Behalf Of Christian Huitema
Sent: Wednesday, May 20, 2020 2:10 PM
To: Carsten Bormann <[email-masked]>
Cc: [email-masked]; Nitay Argov <[email-masked]>; Kuhn Nicolas <[email-masked]>
Subject: Re: [EToSat] Packet lost on last mile network?

**EXTERNAL EMAIL**

an end-to-end solution.

It is always possible to do a local optimization by adding some intelligence in the network. That's the case for pretty much all middle boxes, and it seems that the proxies envisaged in Loop fall into that category. But there are always downsides, such as management, flexibility, and most importantly ossification by old proxies prone to bit rot.

Given effort, it is almost always possible to find end-to-end solutions that are "good enough", getting you within 90% of the optimum or some such. FEC as tail probe would be one example of that. The end-to-end solutions also tend to get better over time, as implementers devise more clever algorithms and update the code. All it takes is some motivation that the effort is important.

An example of that would be support for cellular networks. Twenty years ago, this was a niche, and TCP was not optimized for it. Hence the deployment of middle-boxes in the cellular networks. But as this became an ever more important scenario, TCP improved, and QUIC will improve over that. The cellular middle-boxes become harder to justify, and will start moving from the "optimization" to the "hindrance" category.

Satellite networks are a peculiar case. They remain very much a niche application to this day, something like a communication media of last resort in places that fiber or cellular cannot reach. Given the niche status, it is unclear whether end-to-end transports will bother implementing optimized algorithms, even when these are available. But it is also unclear whether "mainline" networks will bother deploying middle-boxes solely for the benefit of the satellite niche, which kinds of rules out solutions like 'deploy a box in every Wi-Fi network'. 'Deploy a proxy at the ground station' might work, because the satellite operator has the right business incentives, but of course that requires nodes to still do something special for satellites, i.e., discover the proxy and use it. Not obvious.

Hence my effort to look at end-to-end solutions covering not just the satellite case, but also other long distance networks such as intercontinental connections. These are still niches today, given VPN, but they are still useful for P2P applications. Given enough niches, there might be enough motivation for doing the effort.

-- Christian Huitema