Subject: Re: "Proxying" multipath usecases and application scheduling

From: "Gorry (erg)" <[email-masked]>
Date: Mon, 24 Jan 2022 17:57:59 +0000

> On 24 Oct 2020, at 09:03, Mirja Kuehlewind <mirja.kuehlewind=[email-masked]> wrote:
> 
> Hi David,
>  
> see below.
>  
> From: QUIC <[email-masked]> on behalf of David Schinazi <[email-masked]>
> Date: Saturday, 24. October 2020 at 03:20
> To: Mirja Kuehlewind <mirja.kuehlewind=[email-masked]>
> Cc: IETF QUIC WG <[email-masked]>, Matt Joras <[email-masked]>
> Subject: Re: "Proxying" multipath usecases and application scheduling
>  
>  
>  
> On Fri, Oct 23, 2020 at 5:54 PM Mirja Kuehlewind <mirja.kuehlewind=[email-masked]> wrote:
> Hi Matt,
>  
> I think this discussion rather would belong on the MASQUE list and I guess we already had some of this before chartering MASQUE but let me reply to a couple of comments below.
>  
> As per the MASQUE charter, multipath is out of scope for MASQUE. I'd personally suggest we keep all multipath QUIC conversations on the QUIC list for now.
>  
> The point I was making is that most of the discussion below (so far) is not about multipath but about use of QUIC as a tunneling protocol with a proxy function sitting in the network which is what MASQUE is looking at.
>  
> From: QUIC <[email-masked]> on behalf of Matt Joras <[email-masked]>
> Date: Friday, 23. October 2020 at 22:10
> To: IETF QUIC WG <[email-masked]>
> Subject: "Proxying" multipath usecases and application scheduling
>  
> Following up from Martin Thomson's excellent summary of the BoF-esque multipath QUIC interim, I thought I'd start a discussion extracting a couple things I saw as important takeaways from the presentation and subsequent questions.
>  
> As David noted, there were essentially two categories of usecases presented. The first, embodied by the Apple and Alibaba presentations, were essentially existing applications that can benefit from a transport with the capability to simultaneously utilize multiple paths. For these it seemed everyone agreed that in order for multiple paths to be useful the application has to have input into the scheduling decisions on these paths. For example, the application knows how it wants to trade off latency of delivery and overall throughput of delivery for application data.
>  
> Application input is needed if and only if there is actually a tradeoff to make. However, there are many scenarios where one can actually chose between a link that is better than the other, regarding both delay and latency, and then the decision is easy and the same for all applications.
>  
> Could you elaborate? Choosing between links applies to connection migration, multipath is a much more complex beast.
>  
> Yes, that is definitely also something to consider with connection migration and as I said below making the right choice when to switch to a new path is not easy (also with migration when used for handovers). Having the possibility to use paths simultaneous for a while and get measurement data for both paths can help a lot. However, just because you are able to use multiple paths simultaneously that does not mean that you have to do that all the time in all scenarios. There are many scenarios where you rather pick the best one (until you have reached threshold indicating performance degradation). Again sending some probing traffic on the other path is beneficial to make the handover decision.
>  
> The second set of usecases utilized multipath "in the network". ATSSS and the two variants of Hybrid Access Networks presented fall into this category. In this scheme application data is transparently proxied over multiple paths. This would include things like TCP flows, which could be intercepted in the traditional way TCP PEPs operate, as well as UDP (including QUIC) datagrams.
>  
> Intercepting TCP and tunneling is far not the same thing. There are many functions today that tunnel your traffic in some way in some network. Traffic is rerouted or EMCP is used to choose a path. These functions are needed for network management and network resilience and it’s explicitly the role of the end-to-end transport to adapt to changing network conditions and  handle these situations.
>  
>  
> Both types of traffic would be scheduled along the multiple paths towards the "other end" of the proxy, where it presumably re-enters the general Internet towards the destination endpoint.
>  
> Prior to the meeting this latter set of usecases always felt concerning to me, and the meeting did an excellent job of crystallizing my concerns. As others have said, a great benefit of QUIC has been its ability to "cut through" the meddling of things like TCP PEPs, which are often making transport-level decisions that harm the goals of the endpoint application. While the in-network multipath schemes cannot meddle with QUIC flows as much as TCP PEPs, I am still very concerned about the consequences of utilizing them.
>  
> Again for me this is by far not the same and threating everything that is done with your flows in the network the same seems just oversimplifying it.
>  
> We seem to have established that application input is necessary to achieve the benefits from utilizing multiple paths, as there is not one "best" way to schedule data.
>  
> There are cases where there is actually a best way, whenever one path is better than the other without having to make tradeoff, and as I said in my other mail its usually not only application input that is needed to make a decision. E.g. Christoph brought this example that mobile data should be avoided and of course it also depends on the network condition, and that’s where congestion control comes into play.
>  
>  
> These proxyng solutions have no way to act on application concerns, and rather have to devise scheduling policies solely from network-level information, ignorant of signals from the application. Does this not inevitably lead to pathological scheduling decisions and applications which are helpless to do anything about it?
>  
> The obvious counterpoint is that these proxies, through advent of being embedded "in the network", have access to information which the endpoint does not, e.g. about the state of the paths. An argument can be made that because of this they can make proactive decisions, rather than relying on the endpoint to react to changes. This is of course true, but completely ignores the fact that without the application context, it is impossible for the proxy to make an optimal scheduling decision.
>  
> Again there are scenarios where there is clearly a better path, or where there is a strong desire for the network to avoid usage of a certain. E.g. in the hybrid access case, some operators lease one of the links from another operator. I think it’s understandable that they only offer dual connectivity if they have a way to minimize usage of the more expensive line. There are many scenario where network management decision need to be made independent of application characteristics.
>  
> And regarding ATSSS is already exists (based on MPTCP for TCP and ATSSS-LL for other traffic) and will be used, similar as other techniques that impact routing. I think it’s understandable that operator try to utilize all available resources and usually that actually means that more capacity and a better is available for the user/the operator’s costumer. We can argue about the best technical way to make use of these resources but a) the final decision about ATSSS will be made in 3GPP and not the IETF and b) having an explicit proxy setup where the client can decide to use as it or not, as it is done also in other masque use case, seems to me actually a good approach (or at minimum better than what we have right now).
>  
> When you say "ATSSS will be used", do you have references? I'm not aware of any smartphone provider planning to support ATSSS but I might have missed it.
>  
> ATSSS is 5G, however, the MPTCP/TCP converter based solution is deployed for hybrid access as also standardized by BBF.
>  
> If the only benefit to running such proxies "in the network" is their extended knowledge of network conditions, wouldn't a better path forward to have endpoints engaging in "native" multipath? To solve the information gap, i.e. the fact that the "network knows more", could we not develop mechanisms for signalling this information proactively to endpoints?
>  
> I guess you can make the same proposal in the other direction: why cannot the endpoints tell the network what the application needs? I think this is a discussion to have in the APN BoF, however, we had this discussion many times and the short answer is trust. Might probably is a longer answer as well and there is a lot of disagreement.
>  
> End-user privacy?
>  
> I only meant to say I don’t think we should have this discussion here and now.
>  
> Now these concerns may not sound QUIC-specific, but I think it is still relevant to the working group for the following reason. Usecases are very important towards driving design of something like a multipath protocol. It seems to me that one of the two major categories of usecases for a multipath QUIC is, by design, incompatible with some of the core tenets of QUIC as an application-enhancing transport protocol.
>  
> I think that is a way to broad statement and far too simplified. There have been so many discussions in the IETF including in the QUIC working and we clearly don’t have agreement about what the right level of cooperation is because there is just not a simple solution. I fully understand the pain we had with TCP. I’ve been working so long on ECN and am still sad how this good idea git killed in its infancy because of a buggy home router implementation that badly interfered. However, I personally also still believe that there is a way to further optimize and better utilize the resource we have if there is some cooperation between the network and endpoints. Anyway that’s my view on a different topic and not the point of this discussion.
>  
> As Martin said, we obviously can't stop anyone from utilizing multipath in this way, and it of course does not serve as a reason to not design a multipath QUIC, but it seems to me something we should all keep in mind as we consider such designs.
>  
> I think we should not consider this to make a decision on multipath support in QUIC because as you said you can’t stop it. The only thing you can achieve for ATSSS is that 3GPP may design some proprietary way to realize this function which will further increase complexity in an already complex system. However, that also an different discussion to have.
>  
> I’m been trying here to mainly explain what 3GPP is doing on ATSSS in order to get this group some understand of what the requirements are. However, I mainly just think we should simply have multipath support in QUIC because a modern protocol should be able to utilize the fact that more and more devices have connectivity to multiple network paths. I think using path migration for handovers is a bit a hack and not the real solution because it’s actually not easy for an endpoint without further network knowledge to detect when it is the right time to switch. Having a more complete multipath feature were you have two congestion controllers and can get measurements from both paths does help that problem and we do have good experience with that for MPTCP.
>  
> I'll point out that "we should simply have multipath support in QUIC" is very optimistic, I really do not expect this to be simple.
>  
> By “simple” I meant the decision to add multipath support to QUIC. Still, adding multipath support to QUIC is actually much simpler than is was for MPTCP as QUIC is already halve way there. When we talk about adding multipath support, that means the signaling to setup and use multiple paths simultaneous. As Olivier explained already many times, there is no desire to standardize path managers and schedulers. However, these are also parts when we can actually directly leverage the  experience gained with MPTCP and what I see from actual deployments is that the path manager will be simply as most scenarios only have one interface at the server and a client with two interfaces which usually opens the second subpath as soon as the interface comes available; and there is a small set of schedulers that actually serve most use cases well, as shown in Christoph’s presentation.
>  
> We seems to disagree in this group also about the complexity multipath support would add. Yes very feature you add, adds new lines to your code (somethings even features your remove can add new code…), however, if people think this is an important argument for the decisions we should further discuss that and more objectively assess the proposed approaches to that respect, rather than exchanging options or guesses. So far the discussion has focusd on use cases and the question if someone would use the feature if it would be there (I believe the answer to that question is yes) and also if there  are enough people who would actively work on that (I also believe the answer to that question is also yes but there again seems to be some disagreement). I definitely would welcome more discussion about the proposed approaches at this point!
>  
> Mirja
>  
>  
> David
>  
> Mirja
>  
>  
> I am very curious to hear others' opinions in this direction and thanks for sitting through this wall of text.
>  
> Matt Joras

---

From: Christian Huitema <[email-masked]>
Date: Mon, 24 Jan 2022 10:15:07 -1000

I have a very hard time understanding who is saying what in this thread. 
The replies come from different agents using different conventions, and 
the result is a bit of a mess. I went back to the mail archive 
(https://mailarchive.ietf.org/arch/browse/quic/) to try to understand 
who said what, but I am still not sure. Anyhow, that will notstop me 
from adding my two cents...

Like others, I have mixed feelings about the kind of proxying proposed 
in the 5G design. It does look like a power grab by the telecom 
companies, force all user traffic through a telco managed proxy,  
getting an observation point to see all the user traffic, do all the 
kind of "statistics" we could expect in these days of surveillance 
capitalism, and be in a position to control how much bandwidth is 
allocated to specific content providers. The only good effect is that 
proxying will hide the actual user location from the content providers, 
which removes a bit of data from the surveillance capitalism dragnet. 
But overall, that's not great, and I would rather not have a feature 
like that on my phone. But hey, that's my opinion, people may differ. 
And I wonder whether that has much relevance to IETF work.

For the QUIC WG, we have two relevant works in progress: Masque, and 
multipath. It is pretty clear that if both are defined, then they can be 
combined to deliver something like the 5G proxying service. It is also 
pretty clear that multipath has use cases that do not involve Masque, 
and that availability of QUIC multipath diminishes the need to use 
proxies in order to benefit from multiple paths. I would very much like 
to focus on these "end to end" use cases for now, and only worry about 
combining Masque and multipath once Masque is done.

-- Christian Huitema

---

From: Paul Vixie <[email-masked]>
Date: Mon, 24 Jan 2022 12:33:38 -0800

Christian Huitema wrote on 2022-01-24 12:15:
> ...
> companies, force all user traffic through a telco managed proxy, getting 
> an observation point to see all the user traffic, do all the kind of 
> "statistics" we could expect in these days of surveillance capitalism, 
> and be in a position to control how much bandwidth is allocated to 
> specific content providers.

i think calling it a power grab when it's done by an internet transit 
connectivity provider (such as "the telecom companies") makes sense, but 
that without clarification, could connote untenable meanings.

managed private networks, such as enterprise, government, university, 
small office / home office, and home / family networks, already have the 
power you describe, and will preserve the status quo at "whatever cost" 
the ietf may impose.

i pray that we will consistently disambiguate. there will be no wide 
area UDP on most managed private networks. we can either pressure these 
networks with endpoint failures (to strong arm them into abandoning 
their historic powers), or we can rely on fallback (which for new 
protocols may be more fragile than for the existing WWW), or we can 
negotiate, in ways approximately alike to the "proxying proposed in the 
5G design". those are our choices -- there is no fourth way.

if the ietf wishes to disintermediate on-path actors, then we ought to 
consistently and carefully identify where the power is (managed private 
edge networks, and endpoints), and avoid antagonizing those power holders.

> The only good effect is that proxying will 
> hide the actual user location from the content providers, which removes 
> a bit of data from the surveillance capitalism dragnet. But overall, 
> that's not great, and I would rather not have a feature like that on my 
> phone. But hey, that's my opinion, people may differ. And I wonder 
> whether that has much relevance to IETF work.

if the ietf's mission is to impose societal change, then explicit 
negotiated proxy service may not be relevant. however if the ietf's 
mission is to create technology that supports generally desirable work 
flows, then proxy discovery/negotiation is vitally relevant to that.

several networks i operate and many that i'm aware of will have to do 
content inspection. there will be no free lunch for IoT (which is an 
instance of "surveillance capitalism dragnet"). the ietf has to decide 
whether to oppose, or ignore, or cooperate with that reality.

-- 
P Vixie

---

From: David Schinazi <[email-masked]>
Date: Mon, 24 Jan 2022 15:34:28 -0800

This thread dates back to 15 months ago, I'm not entirely sure why Gorry
resurrected it without adding any text, but perhaps my email client is
confused.

Without going into the vices and virtues of content inspection, let's not
increase
the scope of many working groups too soon. The MASQUE WG already has its
plate pretty full getting proxying to work, so let's not add multipath into
the mix
until we have proxying working. Similarly, the QUIC WG is going to be
solving
the already hard problem of end-to-end multipath without adding proxying to
complicate things. If folks want to work on multipath proxying, I strongly
suggest
waiting for both multipath and proxying to exist before trying to combine
them.

David, MASQUE and QUIC enthusiast

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 25 Jan 2022 09:59:23 +0000

So...

I've added something at the end, to note agree on the priority...

On 24/01/2022 23:34, David Schinazi wrote:
> Without going into the vices and virtues of content inspection, let's 
> not increase
> plate pretty full getting proxying to work, so let's not add multipath 
> into the mix
> complicate things. If folks want to work on multipath proxying, I 
> strongly suggest
> waiting for both multipath and proxying to exist before trying to 
> combine them.
>
> On Mon, Jan 24, 2022 at 12:34 PM Paul Vixie 
> <paul=[email-masked]> wrote:
>
>
>
>     getting
I really wanted to be sure the QUIC WG didn't forget that there are 
other use-cases for the MASQUE work.  When I paused to think about what 
to ask in my email, my email client "helpfully" just "sent" my pending 
email.... actually on this occasion that wasn't such a bad thing as it 
turns out, because it generated some responses!

Other people have now mentioned enterprise networks; 5G (using a variety 
of sub-IP technologies) and IoT, there are probably more. I expect the 
future to not be as simple as a transfer of power/trust from operators 
to content providers for all uses of QUIC. Really, this has potential to 
help with QUIC performance for some specific deployment scenarios - if 
the IETF chooses "ignore these use cases" or "design to prevent these 
use cases" - then my guess is that this will not stop those who operate 
networks from doing things, things that might be worse - the example of 
NAT seems a relevant previous use of IETF thinking that if we ignore, it 
will disappear.

And I agee with David: the plan to first do MASQUE (for the clear use 
case accepted by the group); and also to better understand "end to end" 
multipath do seem like a good step in the correct direction. That's all.

Gorry

---

From: Watson Ladd <[email-masked]>
Date: Tue, 25 Jan 2022 08:44:26 -0500

> I've added something at the end, to note agree on the priority...
I'm really not sure I understand how these usecases are indended to work
and will complicate an already complicated enough protocol. In particular
discovering the indended proxy endpoint seems like a ball of wax. Then we
have the issue of ossification as well as the privacy impacts of at least
some of the proposals.

> of sub-IP technologies) and IoT, there are probably more.
>

I'm not really sure that these are the same problem. Advertising multiple
subIP paths isn't the same as network based management, which really should
be done at the host level. And ECMP in the network is a thing. It's
possible to imagine opportunistic multipath that changes the 5 tuple hash
to obtain multiple potential paths.

In the case of satcom where FEC and retransmissions due to consistent
packet loss together with bad latency are challenges for loss based
congestion control the case for a proxy is a bit better assuming we can't
succeed in having better congestion control algorithms.

Sincerely,
Watson