Subject: Should the specifications hard-refer to UDP?

From: "John Ericson" <[email-masked]>
Date: Mon, 17 Aug 2020 23:47:46 -0400

To be clear, I have no issue with the pragmatics that QUIC will be layered atop UDP in practice. I am *not*, for example, trying to dredge up some years-old discussion about whether QUIC should be assigned an IP protocol number!

What I am questioning is the editorial/rhetorical choice of whether the specifications should refer to UDP in particular as the underlying protocol, or abstractly speak of an underlying protocol that merely must meet certain criteria, such as preserving message boundaries and (at least ephemerally) addressing endpoints.

Here are the benefits I think of doing so:

* By speaking about the underlying protocol in abstract terms, the specification can narrow in on just the qualities of UDP pertinent to the aspect of QUIC being specified, bringing clarity to the spec.

* Implementations that do use other underlying transports can still provide a fully-conformant QUIC implementation, rather than a protocol that is merely QUIC "in spirit". I am thinking of research or experimental implementations that are coupled with real-world applications for evaluation, and also of educational projects---perhaps ones that do so merely to teach students about protocol layering!

* If the underlying protocol is swapped in educational projects, students would retain the ability to follow the RFC to the letter while doing their work, rather than having to carefully take creative license. [I might remark that it's great that the layering **within* *QUIC is already so wonderful, and that ought to yield educational dividends.]

* Implementations that rely on the OS's or other external implementation of UDP (which I'd guess is at least a plurality of them weighted by usage) are already abstracting over exact details of the underlying transport mechanism. Even as abstracting over the underlying transport bring the specification further from how QUIC is **used* *in practice, one could argue it brings it closer to how it is **implemented* *in practice.

If there is interest in this, I would be happy to take to the time to author pull requests replacing references to UDP and port numbers with more abstract language.

Finally, my sincerest apologies if this has been discussed before. It certainly seems like the sort of thing that would have been, but I couldn't find a prior discussion searching the GitHub issue tracker and mailing list. (The closest I saw was [1], but that was focused on avoiding the use of 5-tuple altogether, rather than having QUIC continue to rely on the underlying protocol's addressing/disambiguation and merely avoid requiring that it be 5-tuple in particular.)

Thanks for your consideration,

John

[1]: https://mailarchive.ietf.org/arch/msg/quic/dPyWlsrhQGyAUae4_Ht5wYbX_vk

---

From: "Martin Thomson" <[email-masked]>
Date: Tue, 18 Aug 2020 14:36:31 +1000

Hi John,

Thanks for your email.  This is something I recall discussing, but it might not have made it onto a record (it might have been in an in-person meeting), so it's good that someone raises it.

There are reasons that we might not want too much abstraction here.  To take the most obvious example, if you were to seek an IP protocol number for QUIC, there are a few things that need attention.  You lose the UDP port number, so references to those in the protocol are invalid.  We encode those in just a few places, so maybe you can just pretend that any value is OK, or set them to zero always.  Maybe you tweak the MTU to account for the lower overhead.  But we mostly just talk about "address" in the abstract, and we already deal with MTU variation between IPv4 and IPv6.  After that, there isn't much more to say.  So then you have a protocol that interoperates.

Whatever you call this new thing, it won't interoperate with QUIC.  

This also carries a risk, because this "porting" exercise is different based on the target.  If you were to overlay the abstract protocol on a different medium entirely, you need to consider how other IP services are used by QUIC.  A protocol defined directly atop IP still benefits from ICMP and ECN and the other facilities provided by IP, but if you wanted to layer this in some other way those facilities might not be available.  Maybe you don't miss those facilities, and it is almost certainly not work of the same magnitude as a full protocol design, but it requires careful analysis.

Now, I'm not saying that there isn't value in understanding the exact shape of the protocol's dependency on UDP.  That's a great exercise.  Personally, I would want to see that this didn't adversely affect the ability of a new reader to comprehend what is already a complicated mess of a protocol (300+ pages of old-fashioned text).

Getting a split right is challenging.  It's more than just understanding the structure of specification text.  You have to discover all the places that the abstraction leaks through.  That could be a useful contribution, but we haven't prioritized it in this working group.  That doesn't mean it's not possible, only that changes on that magnitude would need to be careful.  That's a non-trivial exercise in a document on this scale.

The split for TLS was originally much cleaner, but reality set in and I'm not sure that we'd be able to unwind the resulting tangle (though we have an issue on file).  That interface is probably more involved than the interface to UDP though; maybe a clean abstraction is more achievable in this case.

I don't want to discourage you from trying if you are so inclined, but I do want to set expectations appropriately.  We regard this documentation to be almost done, so our tolerance for risk or delays is pretty low.  All I can promise is that we'll give any contributions a fair assessment.

Cheers,
Just one of the editors

On Tue, Aug 18, 2020, at 13:47, John Ericson wrote:
> To be clear, I have no issue with the pragmatics that QUIC will be 
> layered atop UDP in practice. I am *not*, for example, trying to dredge 
> up some years-old discussion about whether QUIC should be assigned an 
> IP protocol number!
> 
> What I am questioning is the editorial/rhetorical choice of whether the 
> specifications should refer to UDP in particular as the underlying 
> protocol, or abstractly speak of an underlying protocol that merely 
> must meet certain criteria, such as preserving message boundaries and 
> (at least ephemerally) addressing endpoints.
> 
> * By speaking about the underlying protocol in abstract terms, the 
> specification can narrow in on just the qualities of UDP pertinent to 
> the aspect of QUIC being specified, bringing clarity to the spec.
> 
> * Implementations that do use other underlying transports can still 
> provide a fully-conformant QUIC implementation, rather than a protocol 
> that is merely QUIC "in spirit". I am thinking of research or 
> experimental implementations that are coupled with real-world 
> applications for evaluation, and also of educational projects---perhaps 
> ones that do so merely to teach students about protocol layering!
> 
> * If the underlying protocol is swapped in educational projects, 
> students would retain the ability to follow the RFC to the letter while 
> doing their work, rather than having to carefully take creative 
> license. [I might remark that it's great that the layering **within* 
> *QUIC is already so wonderful, and that ought to yield educational 
> dividends.]
> 
> * Implementations that rely on the OS's or other external 
> implementation of UDP (which I'd guess is at least a plurality of them 
> weighted by usage) are already abstracting over exact details of the 
> underlying transport mechanism. Even as abstracting over the underlying 
> transport bring the specification further from how QUIC is **used* *in 
> practice, one could argue it brings it closer to how it is 
> **implemented* *in practice.
> 
> If there is interest in this, I would be happy to take to the time to 
> author pull requests replacing references to UDP and port numbers with 
> more abstract language.
> 
> Finally, my sincerest apologies if this has been discussed before. It 
> certainly seems like the sort of thing that would have been, but I 
> couldn't find a prior discussion searching the GitHub issue tracker and 
> mailing list. (The closest I saw was [1], but that was focused on 
> avoiding the use of 5-tuple altogether, rather than having QUIC 
> continue to rely on the underlying protocol's addressing/disambiguation 
> and merely avoid requiring that it be 5-tuple in particular.)
>

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 18 Aug 2020 14:52:14 +1000

Just to add to what Martin has said - 

This reminds me of the HTTP/1.1 specification very much. A fair amount of effort has been put in over the years to preserve some independence from TCP, so that it can be layered on top of other protocols. Apparently that is sometimes done, but I don't think at Internet scale (i.e., between parties who don't have some  previous agreement).

So it *is* of some value to have this sort of abstraction in protocol specification. However, it comes at a price -- both in interoperability (as Martin mentions), and in clarity. For HTTP, we're still working through the consequences of this early documentation choice (e.g., with bugs like 'what does a TCP half-closed connection mean?').

Also, as Martin mentions all abstractions are leaky; we like to think of them as barriers that we can build on top of without knowledge of anything beneath, but in practice they're rarely so absolute. If we were to factor UDP out, we'd have to replace it with an abstraction; if that abstraction wasn't 'exactly what UDP offers', the delta would essentially be a set of assumptions begging to be violated.

Cheers,

> On 18 Aug 2020, at 2:36 pm, Martin Thomson <[email-masked]> wrote:
> 
> Hi John,
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 18 Aug 2020 12:02:38 +0200

On Tue, Aug 18, 2020 at 02:52:14PM +1000, Mark Nottingham wrote:
> This reminds me of the HTTP/1.1 specification very much. A fair amount of
> effort has been put in over the years to preserve some independence from TCP,
> so that it can be layered on top of other protocols. Apparently that is
> sometimes done, but I don't think at Internet scale (i.e., between parties
> who don't have some  previous agreement).

In fact it's used quite a lot inside hosts between proxies and servers
for example, as UNIX sockets. And I predict the exact same will happen
for QUIC for the exact same reasons (no resource shortage on ports,
easier security, lower system cost etc). But each time it's done, it's
up to the implementation to know how much they deviate from the standard
and in my opinion this is what limits the risks of seeing ugly monsters
appear, because everyone tries to stick as closed as possible to the
initially chosen protocol.

If we start to be vague about what can be used underneath, while the risk
of seeing ugly things was pretty low for HTTP/1 given that it required a
stream protocol, we could expect to see really ugly stuff pretending that
anything is a datagram and could carry a QUIC packet (including ICMP or
infrared beams). I'm not saying that it's not interesting to explore these
areas, just that they are far outside of the scope of what's being done
here, and I'd rather see this explored only by those having a real interest
there.

And FWIW, yes, once we have a working QUIC in haproxy, I really hope that
we'll also support it on UNIX dgram sockets just for completeness, just like
H1 and H2 work pretty fine on UNIX stream sockets. It's just not a goal
right now, and I wouldn't want the draft to tell us how we should implement
OS-specific sockets for internal usage.

Just my two cents,
Willy

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 18 Aug 2020 11:19:46 +0100

(Chair hat off)

+1 to Martin's points.

I think there is a risk that trying to piecemeal abstract language into the
core transport document will reduce the document comprehension, and tie up
time with reviewers etc. I'm of the opinion that this challenge, at this
late stage in the document lifecycle, is too large.

However, there is a concrete use case in the IETF for exchanging QUIC
packets not in plain old UDP datagrams - MASQUE [1]. Perhaps that could be
a suitable venue for some of the things you have in mind?

Cheers
Lucas

[1] - https://datatracker.ietf.org/wg/masque/about/

On Tue, Aug 18, 2020 at 5:37 AM Martin Thomson <[email-masked]> wrote:

> Hi John,
>
> Thanks for your email.  This is something I recall discussing, but it
> might not have made it onto a record (it might have been in an in-person
> meeting), so it's good that someone raises it.
>
> There are reasons that we might not want too much abstraction here.  To
> take the most obvious example, if you were to seek an IP protocol number
> for QUIC, there are a few things that need attention.  You lose the UDP
> port number, so references to those in the protocol are invalid.  We encode
> those in just a few places, so maybe you can just pretend that any value is
> OK, or set them to zero always.  Maybe you tweak the MTU to account for the
> lower overhead.  But we mostly just talk about "address" in the abstract,
> and we already deal with MTU variation between IPv4 and IPv6.  After that,
> there isn't much more to say.  So then you have a protocol that
> interoperates.
>
> This also carries a risk, because this "porting" exercise is different
> based on the target.  If you were to overlay the abstract protocol on a
> different medium entirely, you need to consider how other IP services are
> used by QUIC.  A protocol defined directly atop IP still benefits from ICMP
> and ECN and the other facilities provided by IP, but if you wanted to layer
> this in some other way those facilities might not be available.  Maybe you
> don't miss those facilities, and it is almost certainly not work of the
> same magnitude as a full protocol design, but it requires careful analysis.
>
> Now, I'm not saying that there isn't value in understanding the exact
> shape of the protocol's dependency on UDP.  That's a great exercise.
> Personally, I would want to see that this didn't adversely affect the
> ability of a new reader to comprehend what is already a complicated mess of
> a protocol (300+ pages of old-fashioned text).
>
> Getting a split right is challenging.  It's more than just understanding
> the structure of specification text.  You have to discover all the places
> that the abstraction leaks through.  That could be a useful contribution,
> but we haven't prioritized it in this working group.  That doesn't mean
> it's not possible, only that changes on that magnitude would need to be
> careful.  That's a non-trivial exercise in a document on this scale.
>
> The split for TLS was originally much cleaner, but reality set in and I'm
> not sure that we'd be able to unwind the resulting tangle (though we have
> an issue on file).  That interface is probably more involved than the
> interface to UDP though; maybe a clean abstraction is more achievable in
> this case.
>
> I don't want to discourage you from trying if you are so inclined, but I
> do want to set expectations appropriately.  We regard this documentation to
> be almost done, so our tolerance for risk or delays is pretty low.  All I
> can promise is that we'll give any contributions a fair assessment.
>
> Cheers,
> Just one of the editors
>
> > [1]:
> https://mailarchive.ietf.org/arch/msg/quic/dPyWlsrhQGyAUae4_Ht5wYbX_vk
> >
> >
>
>

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 18 Aug 2020 11:47:12 +0000

*   I think there is a risk that trying to piecemeal abstract language into the core transport document
Please don’t do this.  Maybe don’t do it ever.  QUIC is not an abstraction, it is bits on the wire with specific syntax and semantics. Turning IETF specifications into a semantic-only document, with all the attendant language issues, really hurts interop.

---

From: Ian Swett <[email-masked]>
Date: Tue, 18 Aug 2020 09:04:34 -0400

I don't believe anyone has implemented QUIC over anything besides UDP,
which makes me think it's too early to specify it.

I'm also pretty risk averse and would agree with most of Martin's points.
I'd support editorial changes that make the text a bit less tightly coupled
to UDP, and identification of design changes which could be put into
potential QUICv2 work, but I doubt I'd support any design changes specific
to this at this stage.

Willy, I think it'd be very cool to see this run over UNIX dgram sockets.

On Tue, Aug 18, 2020 at 6:20 AM Lucas Pardue <[email-masked]>
wrote:

> (Chair hat off)
>
> +1 to Martin's points.
>
> I think there is a risk that trying to piecemeal abstract language into
> the core transport document will reduce the document comprehension, and tie
> up time with reviewers etc. I'm of the opinion that this challenge, at this
> late stage in the document lifecycle, is too large..
>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 18 Aug 2020 09:55:15 -0700

On 8/18/2020 6:04 AM, Ian Swett wrote:

> which makes me think it's too early to specify it..

+1

>
> I'm also pretty risk averse and would agree with most of Martin's
> points.  I'd support editorial changes that make the text a bit less
> tightly coupled to UDP, and identification of design changes which
> could be put into potential QUICv2 work, but I doubt I'd support any
> design changes specific to this at this stage.

Please leave the spec as Quic over UDP, because that's what we have
developed and tested.

If people want to run Quic over IPv6, ATM, or MPLS, or whatever else,
they should first run an experiment, learn what works and what does not,
and publish drafts describing problems and solutions. That is, running
code instead of some theoretical exercise.

> Willy, I think it'd be very cool to see this run over UNIX dgram sockets.

Sure. I also run QUIC over a simulated network as part of the picoquic
test suite, without any actual UDP layer. Of course you can do that. But
that's not the design goal of the WG. Let's not pollute the main spec.

-- Christian Huitema

---

From: Roberto Peon <[email-masked]>
Date: Tue, 18 Aug 2020 19:09:33 +0000

Agreed-- at the point where someone has implemented QUIC over something else and wish to interoperate, then they can document what they did differently, and we could all learn from it!

-=R

On 8/18/20, 10:16 AM, "QUIC on behalf of Christian Huitema" <[email-masked] on behalf of [email-masked]> wrote:

    +1

    -- Christian Huitema

---

From: Lars Eggert <[email-masked]>
Date: Wed, 19 Aug 2020 10:11:22 +0300

+1

Lars

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 19 Aug 2020 09:33:16 -0500

I've read this thread, so I'm current, but wanted to chime in on one of the
encouraging things Martin said ...

> shape of the protocol's dependency on UDP.  That's a great exercise.

I think that's a great place to start. ISTM that we're used to thinking of
UDP as "IP plus ports for path sharing", but that's probably not nearly
enough to base implementation over another protocol on. Documenting that
dependency is probably useful, thought-provoking, and (best of all) not
something that will slow QUICv1 down at all.

As pointed out, running directly over IP and running through a sockets
interface are two obvious possibilities (and I didn't see "running directly
over IP" coming), and they aren't particularly similar, so a document that
describes what QUIC expects from an underlying protocol would be helpful.

John, thank you for starting this conversation.

Best,

Spencer

---

From: "John Ericson" <[email-masked]>
Date: Fri, 21 Aug 2020 01:22:58 -0400

Well, this is about the response I've expected. I'm fully aware I'm finally showing up for this sort of thing at an inconvenient time. Nevertheless, I thought there was just enough positivity, especially in the initial response from Martin, and latest one from Spencer, so I started to give it ago in https://github.com/quicwg/base-drafts/pull/4043.

In it's current state, this WIP PR is just the sort of "piecemeal" introduction of abstract language that Lucas, rightly, warns against. Nevertheless, I think that I could continue that PR, actually covering the interactions of the spec comprehensively, without being substantially more invasive, so I think the WIP PR does have some virtue in indicating the sort of changes I am interested in.

If this attempt looks promising, I'll continue with it, but if this does not escape the pitfalls you all warned about, I'll abandon it and trouble the WG no further.

John

On Wed, Aug 19, 2020, at 10:33 AM, Spencer Dawkins at IETF wrote:
> I've read this thread, so I'm current, but wanted to chime in on one of the encouraging things Martin said .... 
> 
>> Now, I'm not saying that there isn't value in understanding the exact shape of the protocol's dependency on UDP.  That's a great exercise. 
> 
> I think that's a great place to start. ISTM that we're used to thinking of UDP as "IP plus ports for path sharing", but that's probably not nearly enough to base implementation over another protocol on. Documenting that dependency is probably useful, thought-provoking, and (best of all) not something that will slow QUICv1 down at all. 
> 
> As pointed out, running directly over IP and running through a sockets interface are two obvious possibilities (and I didn't see "running directly over IP" coming), and they aren't particularly similar, so a document that describes what QUIC expects from an underlying protocol would be helpful. 
> 
>> Personally, I would want to see that this didn't adversely affect the ability of a new reader to comprehend what is already a complicated mess of a protocol (300+ pages of old-fashioned text).
>>