Subject: Re: [QUIC] Network Path Requirements for QUIC

From: "DRUTA, DAN" <[email-masked]>
Date: Wed, 29 Jun 2016 15:48:58 +0000

I have been following the discussion and while it is probably premature to agree on specific requirements I do believe the scope of the work must include the deployment, management and troubleshooting aspects.
The enterprise use cases are also extremely important and we need to make sure we're addressing them from the beginning.
With this being said, I support Brian's proposed wording in the pull request #14. I think it covers the high level objectives that should be in scope for the work.
This is at least a good start.

Regards,
Dan

---

From: "Diego R. Lopez" <[email-masked]>
Date: Wed, 29 Jun 2016 21:29:49 +0000

Agree with Dan on the support of Brian’s pull request, and on Joe’s concerns on the availability of data supporting some of the statements made during the discussion, especially when they impy the risk of ignoring a significant number of users.

Be goode,

On 29 Jun 2016, at 18:48 , DRUTA, DAN <[email-masked]<mailto:[email-masked]>> wrote:

I have been following the discussion and while it is probably premature to agree on specific requirements I do believe the scope of the work must include the deployment, management and troubleshooting aspects.
The enterprise use cases are also extremely important and we need to make sure we’re addressing them from the beginning.
With this being said, I support Brian’s proposed wording in the pull request #14. I think it covers the high level objectives that should be in scope for the work.
This is at least a good start.

Regards,
Dan
_______________________________________________
QUIC mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/quic

--
"Esta vez no fallaremos, Doctor Infierno"

Dr Diego R. Lopez
Telefonica I+D
http://people.tid.es/diego.lopez/

e-mail: [email-masked]
Tel:    +34 913 129 041
Mobile: +34 682 051 091
----------------------------------

________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 30 Jun 2016 07:26:26 +0000

Hi,

> The enterprise use cases are also extremely important and we need to make sure we’re addressing them from the beginning.

why are they extremely important?

First, HTTP over TCP will continue to function as well as it does today in such use cases. So if QUIC doesn't work, whatever happy-eyeballs mechanism the user agent is using should make it fall back to TCP gracefully.

Second, limiting or blocking UDP in enterprise environments is a management decision by the IT department, and not something that is due to the technology used. Normal NATs in enterprise environments will pass QUIC just fine. If it doesn't pass, it's because it has been administratively disabled.

So we're really talking about what kind of protocol features would make an admin want to unblock UDP for QUIC sessions. I'm pretty sure there's not going to be an easy or even uniform answer to that question. For that reason, I believe that saying "you should open up UDP port 443" is likely going to be as difficult a discussion to have with IT staff than any discussion about yet undefined protocol features that we add in the hopes of making QUIC appeal to them.

We're talking about less than 10% of Internet paths here, and since HTTP over TCP continues to be available and work as well as it does today, I don't think that complicating QUIC for these use cases - especially since we really don't know what features may make an admin want to administratively unblock UDP for QUIC - is worthwhile.

Lars

PS: All hats off.

> [email-masked]

---

From: Brian Trammell <[email-masked]>
Date: Thu, 30 Jun 2016 11:57:34 +0200

Hi Lars, all,

One tiny quibble (that turned into a short essay), inline:

Sent from my iPhone

> On 30 Jun 2016, at 09:26, Eggert, Lars <[email-masked]> wrote:
> 
> Hi,
> 
> Second, limiting or blocking UDP in enterprise environments is a management decision by the IT department, and not something that is due to the technology used. Normal NATs in enterprise environments will pass QUIC just fine.

I think subtly different definitions of "just fine" here is where this discussion is getting stuck. I want to unpack this part a bit:

"UDP blocking is a policy decision" is a statement I'll accept as true: it's hard to measure, but it seems reasonable.

"UDP limits are not due to the technology": all the measurement we've done, and all I've seen results from others, indicates this is also true. The lack of middleboxes that Just Want To Help UDP traffic even seems to make UDP work better on some paths.

If by "just fine" you mean "in the default configuration UDP traffic passes with the same probability as TCP traffic", yes, that certainly seems to be the case.

The order of magnitude difference between default UDP and TCP idle timeouts on NATs, however, remains a cause for concern; for persistent connections with unlucky delays, this will work less "just fine". This is a particular instance of the general problems caused by making transport state opaque.

For H2 over QUIC, sending PING frames every thirty seconds "fixes" this problem, for nonzero cost; there are I'm sure smarter ways of doing this. Or maybe QUIC decides to mark any path on which it has seen state loss as bad for a while and falls back to TCP. But the WG will have to make a decision how to handle this specific state.

> If it doesn't pass, it's because it has been administratively disabled.

We can make that assumption and most of the time it holds. 

But, larger than and separate from the blocking/timeout issue, the reason I suggested the word "diagnosability" in my PR is "what happens when it doesn't". Right now, the state of the art in figuring out what's happening with a given TCP connection is, in most environments I know of, peering at it for a while in Wireshark. A network admin should be able to do at least that well when trying to figure out what's happening with a QUIC connection.

Cheers,

Brian (no hats either)

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 30 Jun 2016 14:12:29 +0000

Hi,

> If by "just fine" you mean "in the default configuration UDP traffic passes with the same probability as TCP traffic", yes, that certainly seems to be the case.

yep, that is what I meant. Thanks for putting it into clearer words.

> For H2 over QUIC, sending PING frames every thirty seconds "fixes" this problem, for nonzero cost; there are I'm sure smarter ways of doing this. Or maybe QUIC decides to mark any path on which it…

Correct; we measured this for the IMC study and I'd be shocked if it has gotten better.

My belief (since I don't have data) is that such middleboxes incorrectly assumed that UDP was just being used for DNS, and that shorter timeout are therefore OK. This is just another example of middlebox vendors making wrong assumptions.

(And - pet peeve - there is really no need for binding timeouts for TCP or UDP to begin with. No binding needs to be timed out until the middlebox runs out of memory, because as long as there are sufficient resources, why bother managing all these timeouts? If a middlebox runs out of resources, it needs a mechanism - LRU, etc. - to free resources, i.e., remove bindings, anyway. So enforcing timeouts just makes things more complicated without actually making anything better. Compare this with processes on operating systems, they aren't being killed either just because they happen to sleep a lot; at least until the OS runs out of resources, and then all bets are off anyway in terms of what gets killed and what doesn't.)

There isn't anything QUIC can do to better pass through these existing middleboxes - they need to be upgraded.

> But, larger than and separate from the blocking/timeout issue, the reason I suggested the word "diagnosability" in my PR is "what happens when it doesn't". Right now, the state of the art in figuri…

Sure. But that sounds more like that we need a wireshark dissector for QUIC, which actually already exists: https://www.wireshark.org/docs/dfref/q/quic.html

Lars

---

From: Brian Trammell <[email-masked]>
Date: Thu, 30 Jun 2016 16:43:14 +0200

hi Lars,

> On 30 Jun 2016, at 16:12, Eggert, Lars <[email-masked]> wrote:
> 
> Hi,
> 

Would have cited the paper by name, but was stuck on an iPhone. :)

> (And - pet peeve - there is really no need for binding timeouts for TCP or UDP to begin with. No binding needs to be timed out until the middlebox runs out of memory, because as long as there are sufficient resources, why bother managing all these timeouts? If a middlebox runs out of resources, it needs a mechanism - LRU, etc. - to free resources, i.e., remove bindings, anyway. So enforcing timeouts just makes things more complicated without actually making anything better.

FWIW, I agree.

Lots of little stupid reasons collude here, I suspect: the operating systems these things run on making it easier to fire a timer than to figure out when their process(es) might be out of memory, a (possibly misguided) desire to make (opaque) behavior predictable instead of unpredictable, a (equally misguided) desire to make this predictable thing admin-configurable.

There's also the fact that it's not just memory that NATs need to conserve: they might also be implemented to manage public address and port scarcity. But LRU mechanisms can handle this scarcity at least as well as timers can.

> Compare this with processes on operating systems, they aren't being killed either just because they happen to sleep a lot; at least until the OS runs out of resources, and then all bets are off anyway in terms of what gets killed and what doesn't.)
> 

This only helps if they get upgraded to a something that actually has better timeout/mapping expiry properties.

Maybe I'm a little less optimistic than you are about the potential impact of encouraging better engineering practice among middlebox vendors, and maybe I'm more optimistic about the potential impact of making this expiry behavior explicit.

I'm not saying QUIC needs a PLUS-style endpoint-exposed desired-timeout indicator; I think that's a discussion for PLUS, actually. I'm saying that discussion of possible protocol design mitigations to the timeout-and-spurious-keepalive problem (as a subpart of the general NAT traversal problem) should be in scope for the WG. The WG might well decide that it explicitly doesn't care, that middleboxes that timeout UDP mappings on a ludicrously short scale are a reason to have TCP fallback. But that decision should appear in the WG's output.

> Sure. But that sounds more like that we need a wireshark dissector for QUIC, which actually already exists: https://www.wireshark.org/docs/dfref/q/quic.html

Yep. There's a lot more to using Wireshark for TCP debugging that just the dissector, though. Some of these things need the protocol to radiate information that QUIC doesn't. Again, not saying that QUIC should expose ACKs, just saying that the discussion of the tradeoff between information radiated and how much pain it inflicts on existing practice needs to happen in the WG.

Cheers,

Brian (still no hats)

---

From: "DRUTA, DAN" <[email-masked]>
Date: Thu, 30 Jun 2016 14:46:10 +0000

Lars,

Brian covered some the reasoning to explore enterprise use cases in detail.
What's important is to understand and capture the reasons why IT Administrators are blocking UDP. Some are as you pointed out not related to the technology but we need to make sure they get the right level of manageability with QUIC in order troubleshooting legitimate traffic and prevent attacks.

Regards,
Dan

-----Original Message-----
From: Eggert, Lars [mailto:[email-masked]] 
Sent: Thursday, June 30, 2016 7:12 AM
To: Brian Trammell
Cc: DRUTA, DAN; [email-masked]
Subject: Re: [QUIC] Network Path Requirements for QUIC

Hi,

Lars

---

From: Watson Ladd <[email-masked]>
Date: Thu, 30 Jun 2016 07:54:25 -0700

On Thu, Jun 30, 2016 at 7:43 AM, Brian Trammell <[email-masked]> wrote:
<chop>
> Yep. There's a lot more to using Wireshark for TCP debugging that just the dissector, though. Some of these things need the protocol to radiate information that QUIC doesn't. Again, not saying that…

What sorts of things are we talking about? I have zero network
operations experience but have investigated stuff with Wireshark
before.

-- 
"Man is born free, but everywhere he is in chains".
--Rousseau.

---

From: Watson Ladd <[email-masked]>
Date: Thu, 30 Jun 2016 08:05:39 -0700

> What's important is to understand and capture the reasons why IT Administrators are blocking UDP. Some are as you pointed out not related to the technology but we need to make sure they get the rig…

So initial emails argued that we need an exposed consent mechanism to
enable connection tracking for DOS prevention. But QUIC already has
DOS prevention techniques: the server can tell clients to "GO AWAY AND
WHEN YOU COME BACK QUOTE ME".  Of course middlebox vendors have an
incentive to pretend this isn't the case, and load balancer vendors
can insist that connection tracking is essential (instead of something
much more clever you can do with QUIC and not TLS), etc. But the
reality is QUIC is working today.

---

From: Brian Trammell <[email-masked]>
Date: Thu, 30 Jun 2016 17:13:08 +0200

> On 30 Jun 2016, at 16:54, Watson Ladd <[email-masked]> wrote:
> 

The simplest ones are the stream graphs (Statistics -> TCP Stream Graphs), which give you a nice tcptrace-like visualization of what's going on with the flow, but need to match sequence and ack numbers. Following a flow (Analyze -> Follow -> TCP Stream) is also useful, but QUIC gives enough information to do this already.

Cheers,

Brian

---

From: "DRUTA, DAN" <[email-masked]>
Date: Thu, 30 Jun 2016 15:14:00 +0000

-----Original Message-----
From: Watson Ladd [mailto:[email-masked]] 
Sent: Thursday, June 30, 2016 8:06 AM
To: DRUTA, DAN
Cc: Eggert, Lars; Brian Trammell; [email-masked]
Subject: Re: [QUIC] Network Path Requirements for QUIC

I'm not arguing if it does or it doesn't. I'm just saying that use cases drive requirements and requirements map to capabilities that are implemented in the protocol.
I'm sure clever things can be done. How to make clever things that work in an interoperable way for a majority of the stakeholders is the hard part.

Regards,
Dan

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 30 Jun 2016 15:21:21 +0000

Hi,

> What's important is to understand and capture the reasons why IT Administrators are blocking UDP. Some are as you pointed out not related to the technology but we need to make sure they get the rig…

again: why? We're talking about less than 10% of the paths, and HTTPS over TCP readily available as an alternative on those paths, supposedly with all the manageability that IT admins need.

My personal viewpoint is that I wouldn't want to complicate the protocol or even slightly weaken its security for such a small subset of paths (some of which may end up unblocking UDP port 443 anyway.) I'd be open to the WG reviewing concrete proposals someone may have in this space once the WG is underway, but I wouldn't want to see charter text making it a requirement for QUIC to do anything about such paths, since we really don't know what fulfilling this requirement may actually entail.

Lars

---

From: Ted Hardie <[email-masked]>
Date: Thu, 30 Jun 2016 09:34:32 -0700

On Thu, Jun 30, 2016 at 7:12 AM, Eggert, Lars <[email-masked]> wrote:

> (And - pet peeve - there is really no need for binding timeouts for TCP or
> UDP to begin with. No binding needs to be timed out until the middlebox
> runs out of memory, because as long as there are sufficient resources, why
> bother managing all these timeouts? <snip>)
>

If I understand this correctly, this is largely because many of  these are
NAT+FW boxes which combine the binding and the firewall permission; leaving
them up for long periods of time leaves the pinhole up.  If you really
relied on state exhaustion, it might never happen at all for small
networks.  Eliminating the firewall permission to receive traffic and
leaving the binding to send would be odd, so they don't.

Joe H. might well be able to get you better and fersher data, but that's
why I think the timers were put in in the first place.

Ted

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 30 Jun 2016 16:42:58 +0000

> On Jun 30, 2016, at 9:21 AM, Eggert, Lars <[email-masked]> wrote:
> 
> Hi,
> 

We're talking about less than 10% of the paths that were measured by one experiment.  I don't yet know how reproducible that is.  Again, it's 100% of the flows on my network, and on that of many other enterprise networks where I have deployed software in the past.

I guarantee that we will deploy new software for our firewall if it's possible to do so in a way that we don't have to update for every newly-popular version number that someone in the QUIC world decides to deploy.

> My personal viewpoint is that I wouldn't want to complicate the protocol or even slightly weaken its security for such a small subset of paths (some of which may end up unblocking UDP port 443 anyway.)

If QUIC was only ever going to be used on 443/udp, some of the detection issues would be easier.  Right now, the prototype code I've seen for the QUIC state tracker hardcodes individual version numbers to look for the traffic that needs to be tracked - but the developer is just trying to intuit what he's supposed to do from what has been written so far.

I don't think anyone has proposed weakening QUIC's security yet.  If they did, the working group should have a thoughtful discussion of the tradeoffs involved, and I would expect security to win out almost every time.  It may be that certain security measures could be implemented in different ways to solve multiple problems, for instance.

Mostly, I would like to see the assumptions that folks are making about how the path might interact with the bits that are already explicitly exposed to the path by the current protocol be discussed, then written down.

> I'd be open to the WG reviewing concrete proposals someone may have in this space once the WG is underway, but I wouldn't want to see charter text making it a requirement for QUIC to do anything about such paths, since we really don't know what fulfilling this requirement may actually entail.

Brian's proposed text here:

https://github.com/IETF-QUIC/Charter/pull/14/files

says "This work will consider deployability, manageability and diagnosability of the wire format, including how the wire format will be processed by devices on path".  I don't think that makes doing anything for the path required, but it does make it possible to have the conversation.  If we're worried about this language requiring changes to the protocol, rather than allowing for the conversation, we can of course continue to tweak the language.

From a previous message of yours:

> And - pet peeve - there is really no need for binding timeouts for TCP or UDP to begin with. No binding needs to be timed out until the middlebox runs out of memory, because as long as there are sufficient resources, why bother managing all these timeouts?

One of the reasons people give for this is the concern that traffic will be allowed in from the outside to boxes and applications that are no longer expecting the traffic.  Since they believe that many older UDP protocols don't have adequate security measures in place to authenticate, authorize, or sessionize traffic, they just block it all except for the protocols where they do deeper inspection, such as DNS.  It is not because these administrators are idiots, or because they have misconfigured their boxes unintentionally.

Many of the folks that block UDP *do* update their firewall software often, and do review their security stance on blocked protocols periodically.  They will want their users to be able to take advantage of the benefits of QUIC, so they will have incentives to do the review.  We need to at least provide them with the tools to do this review, such as security considerations that tell an administrator how to continue to protect their network from unknown UDP protocols while allowing QUIC.

--
Joe Hildebrand

---

From: "DRUTA, DAN" <[email-masked]>
Date: Thu, 30 Jun 2016 15:29:53 +0000

> Sent: Thursday, June 30, 2016 8:21 AM
> To: DRUTA, DAN
> Cc: Brian Trammell; [email-masked]
> > What's important is to understand and capture the reasons why IT
> Administrators are blocking UDP. Some are as you pointed out not related to
> the technology but we need to make sure they get the right level of
> manageability with QUIC in order troubleshooting legitimate traffic and
> prevent attacks.
> 
> again: why? We're talking about less than 10% of the paths, and HTTPS over
> TCP readily available as an alternative on those paths, supposedly with all the
> manageability that IT admins need.
> 
> My personal viewpoint is that I wouldn't want to complicate the protocol or
> even slightly weaken its security for such a small subset of paths (some of
> which may end up unblocking UDP port 443 anyway.) I'd be open to the WG
> reviewing concrete proposals someone may have in this space once the WG
> is underway, but I wouldn't want to see charter text making it a requirement
> for QUIC to do anything about such paths, since we really don't know what
> fulfilling this requirement may actually entail.
> 

I already said that for the charter I'm fine with the proposed text from Brian.

Regards,
Dan

---

From: Brian Trammell <[email-masked]>
Date: Thu, 30 Jun 2016 22:31:34 +0200

> On 30 Jun 2016, at 18:34, Ted Hardie <[email-masked]> wrote:
> 
> (And - pet peeve - there is really no need for binding timeouts for TCP or UDP to begin with. No binding needs to be timed out until the middlebox runs out of memory, because as long as there are sufficient resources, why bother managing all these timeouts? <snip>)
> 
> If I understand this correctly, this is largely because many of  these are NAT+FW boxes which combine the binding and the firewall permission; leaving them up for long periods of time leaves the pinhole up.  If you really relied on state exhaustion, it might never happen at all for small networks.  Eliminating the firewall permission to receive traffic and leaving the binding to send would be odd, so they don't.

That makes sense, too. We're talking about "NATs" and "firewalls" as if you could actually still buy or build a device that is capable of one set of functionality but not the other.

> Joe H. might well be able to get you better and fersher data, but that's why I think the timers were put in in the first place.