Subject: Re: [Potential Spoof] Re: Side meeting @ IETF 104 for QUIC Datagrams

From: Roberto Peon <[email-masked]>
Date: Wed, 27 Mar 2019 17:49:44 +0000

From a base requirements perspective, I imagine that  a datagram needs to:
- delimit the start/end of the message.
- have a unique ID attached so that it is not corrupted by mixing with some other message for the cases where doesn’t fit inside a packet.
- indicate where they need to go (especially in any case where we’re not just doing p2p, but are worried about e2e routing)

It is nice if they also:
- are efficient

If we do ‘message streams’, we waste a single var-int per packet. That is pretty low overhead.
That var-int provides/allows-for e2e (not just p2p) mapping which is useful in many topologies. It allows for larger-than-packet size messages. It enables flow control.

What am I missing?
-=R

From: QUIC <[email-masked]> on behalf of Roberto Peon <[email-masked]>
Date: Wednesday, March 27, 2019 at 10:41 AM
To: Lucas Pardue <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>
Cc: Eric Kinnear <[email-masked]>, David Schinazi <[email-masked]>, Mirja Kuehlewind <[email-masked]>, QUIC WG <[email-masked]>, Colin Perkins <[email-masked]>, Tommy Pauly <tpauly=[email-masked]>
Subject: [Potential Spoof] Re: Side meeting @ IETF 104 for QUIC Datagrams

I remain super un-enthused about datagrams. I think they’re a major step backwards.
Message-streams, or some other partially reliable thing, sure, but datagrams?

-=R

From: QUIC <[email-masked]> on behalf of Lucas Pardue <[email-masked]>
Date: Wednesday, March 27, 2019 at 10:37 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]>
Cc: Eric Kinnear <[email-masked]>, David Schinazi <[email-masked]>, Tommy Pauly <tpauly=[email-masked]>, QUIC WG <[email-masked]>, Colin Perkins <[email-masked]>, Mirja Kuehlewind <[email-masked]>
Subject: Re: Side meeting @ IETF 104 for QUIC Datagrams

Interesting Mikkel. I think also that certain pub/sub models would like reliable datagram. For example, in the past I used DDS (Data Distribution Service) where the user defined topics definition expressed the form of delivery as a requirement, including reliability.

On Wed, 27 Mar 2019, 18:30 Mikkel Fahnøe Jørgensen, <[email-masked]<mailto:[email-masked]>> wrote:
I’d just like to add a comment I also med on the slack datagram channel:

A use case for reliable datagrams:

A consensus protocol like PAXOS for use with databases or block chain could would care about timely delivery, but not about strict ordering. The application endpoint would introduce a total ordering from multiple sources. Here it is important to not loose an event and that it happens ASAP, but the relative order is less important.

The game slide also touches upon this because if two plays pick up the same item, it can be random which event arrives first, but both events are important. And application endpoint makes the call on total ordering.

I think unreliable datagrams are at least as important as reliable datagrams, but both are important.

Mikkel

On 27 March 2019 at 17.35.49, Tommy Pauly (tpauly=[email-masked]<mailto:tpauly=[email-masked]>) wrote:
Thanks to everyone who joined our side meeting! It was a productive time—both for bashing the document and for all of us learning more about the use cases we have.

I've posted the minutes and slides to GitHub:

Minutes (thanks to Tom Jones for taking notes!):
https://github.com/tfpauly/draft-pauly-quic-datagram/blob/master/Side%20Meetings/IETF%20104/minutes.txt

Slides:
https://github.com/tfpauly/draft-pauly-quic-datagram/tree/master/Side%20Meetings/IETF%20104

If you'd like to make any edits to the minutes, please send me a PR...

Best,
Tommy

> On Mar 25, 2019, at 11:42 PM, Tommy Pauly <tpauly=[email-masked]<mailto:[email-masked]>> wrote:
>
> Hello QUIC,
>
> I wanted to share our informal agenda for our side meeting. Please feel free to bring any thoughts that relate to these topics in the forms of a slide or two, or just comments you can share with the group!
>
> Best,
> Tommy
>
> ======
>
> QUIC DATAGRAM protocol details, 20 minutes - Tommy Pauly moderating
> - Datagrams + connection flow control
> - Datagrams + congestion control
> - Datagram flow IDs
> - Negotiating support for datagrams
>
> Voice/Video use cases and requirements, 15 minutes - Colin Perkins moderating
>
> Tunneling use cases and requirements, 15 minutes - David Schinazi moderating
>
> Next steps, 10 minutes - Tommy Pauly moderating
>
>> On Mar 19, 2019, at 6:18 PM, Tommy Pauly <tpauly=[email-masked]<mailto:[email-masked]>> wrote:
>>
>> I've reserved a room for a side meeting on Wednesday to discuss progressing work on unreliable QUIC Datagrams, and other non-HTTP use cases related to QUIC. If you're working on using QUIC for WebRTC, VPN tunneling, unreliable streaming for voice/video, or games, please come along and join the discussion!
>>
>> What: QUIC Datagram Side Meeting
>> When: Wednesday, March 27, 15:00-16:00
>> Where: Congress Hall 3
>>
>> Best,
>> Tommy
>>
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 27 Mar 2019 11:05:09 -0700

Datagrams differ from partial streams in atomicity. Even if you can
reassemble, the chance of loss grows exponentially (or something) with
fragments and it can also tie up a lot of resources, especially in lossy
environments.

I do agree that partially reliable streams are also important, and perhaps
they can be constructed on top of other primitives.

Regardless of the low-level implementation, I believe there is need to have
a language to name the concept of a (un)reliable datagram - and either map
 it to something else or provide a frame for it.

Use of one stream per datagram is not necessarily a good replacement for
datagrams as you churn through these at a fast pace - as has been mentioned
earlier, and the id / offset could take up much space if a datagram is one
byte large, which is natural for many kinds of events.

Mikkel

- delimit the start/end of the message.
- have a unique ID attached so that it is not corrupted by mixing with some
other message for the cases where doesn’t fit inside a packet.
- indicate where they need to go (especially in any case where we’re not
just doing p2p, but are worried about e2e routing)

It is nice if they also:
- are efficient

If we do ‘message streams’, we waste a single var-int per packet. That is
pretty low overhead.
That var-int provides/allows-for e2e (not just p2p) mapping which is useful
in many topologies. It allows for larger-than-packet size messages. It
enables flow control.

What am I missing?
-=R

*From: *QUIC <[email-masked]> on behalf of Roberto Peon <[email-masked]
*Date: *Wednesday, March 27, 2019 at 10:41 AM
*To: *Lucas Pardue <[email-masked]>, Mikkel Fahnøe Jørgensen <
[email-masked]>
*Cc: *Eric Kinnear <[email-masked]>, David Schinazi <
[email-masked]>, Mirja Kuehlewind <[email-masked]>,
QUIC WG <[email-masked]>, Colin Perkins <[email-masked]>, Tommy Pauly
<tpauly=[email-masked]>
*Subject: *[Potential Spoof] Re: Side meeting @ IETF 104 for QUIC Datagrams

I remain super un-enthused about datagrams. I think they’re a major step
backwards.
Message-streams, or some other partially reliable thing, sure, but
datagrams?

-=R

*From: *QUIC <[email-masked]> on behalf of Lucas Pardue <
[email-masked]>
*Date: *Wednesday, March 27, 2019 at 10:37 AM
*To: *Mikkel Fahnøe Jørgensen <[email-masked]>
*Cc: *Eric Kinnear <[email-masked]>, David Schinazi <
[email-masked]>, Tommy Pauly <tpauly=[email-masked]>,
QUIC WG <[email-masked]>, Colin Perkins <[email-masked]>, Mirja
Kuehlewind <[email-masked]>
*Subject: *Re: Side meeting @ IETF 104 for QUIC Datagrams

Interesting Mikkel. I think also that certain pub/sub models would like
reliable datagram. For example, in the past I used DDS (Data Distribution
Service) where the user defined topics definition expressed the form of
delivery as a requirement, including reliability.

On Wed, 27 Mar 2019, 18:30 Mikkel Fahnøe Jørgensen, <[email-masked]>
wrote:

A consensus protocol like PAXOS for use with databases or block chain could
would care about timely delivery, but not about strict ordering. The
application endpoint would introduce a total ordering from multiple
sources. Here it is important to not loose an event and that it happens
ASAP, but the relative order is less important.

The game slide also touches upon this because if two plays pick up the same
item, it can be random which event arrives first, but both events are
important. And application endpoint makes the call on total ordering.

I think unreliable datagrams are at least as important as reliable
datagrams, but both are important.

Mikkel

On 27 March 2019 at 17.35.49, Tommy Pauly (tpauly=[email-masked])
wrote:

Thanks to everyone who joined our side meeting! It was a productive
time—both for bashing the document and for all of us learning more about
the use cases we have.

Slides:

Best,
Tommy

> On Mar 25, 2019, at 11:42 PM, Tommy Pauly <tpauly=
[email-masked]> wrote:
>
> Hello QUIC,
>
> I wanted to share our informal agenda for our side meeting. Please feel
free to bring any thoughts that relate to these topics in the forms of a
slide or two, or just comments you can share with the group!
>> On Mar 19, 2019, at 6:18 PM, Tommy Pauly <tpauly=
[email-masked]> wrote:
>>
>> I've reserved a room for a side meeting on Wednesday to discuss
progressing work on unreliable QUIC Datagrams, and other non-HTTP use cases
related to QUIC. If you're working on using QUIC for WebRTC, VPN tunneling,
unreliable streaming for voice/video, or games, please come along and join
the discussion!

---

From: Ryan Hamilton <[email-masked]>
Date: Wed, 27 Mar 2019 11:09:42 -0700

For messages/datagrams which fit in a single packet a unique ID is not
needed by the transport and consequently should probably not be part of the
base requirements. It can be left up to the application to serialize these
messages in whatever form they prefer. This might or might not include an
ID internally, but QUIC has no need for this information.

On Wed, Mar 27, 2019 at 10:52 AM Roberto Peon <[email-masked]> wrote:

> - have a unique ID attached so that it is not corrupted by mixing with
> some other message for the cases where doesn’t fit inside a packet.
> That var-int provides/allows-for e2e (not just p2p) mapping which is
> useful in many topologies. It allows for larger-than-packet size messages.
> It enables flow control.
>
> What am I missing?
> -=R
>
>
>
>
> *From: *QUIC <[email-masked]> on behalf of Roberto Peon <
> [email-masked]>
> A consensus protocol like PAXOS for use with databases or block chain
> could would care about timely delivery, but not about strict ordering. The
> The game slide also touches upon this because if two plays pick up the
> same item, it can be random which event arrives first, but both events are
> On 27 March 2019 at 17.35.49, Tommy Pauly (
> tpauly=[email-masked]) wrote:
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 27 Mar 2019 11:25:05 -0700

I almost agree that the ID is not needed for single MTU datagrams, but you
don’t know if the packet you didn’t receive contained a datagram. Hence for
applications where you need a sort of timestamping with gap detection, you
do need something. It may be that this something is better contained with
the user part of a datagram - however - if there are hundreds of datagrams
in one packet, this is also not ideal. So - the special use case with gap
detection is separate from the general use case of atomic single MTU
(un)reliable datagram.

On 27 March 2019 at 19.09.55, Ryan Hamilton ([email-masked]) wrote:

On Wed, Mar 27, 2019 at 10:52 AM Roberto Peon <[email-masked]> wrote:

> *From:* QUIC <[email-masked]> on behalf of Roberto Peon <
> [email-masked]>
> *Date:* Wednesday, March 27, 2019 at 10:41 AM
> *To:* Lucas Pardue <[email-masked]>, Mikkel Fahnøe Jørgensen <
> [email-masked]>
> *Cc:* Eric Kinnear <[email-masked]>, David Schinazi <
> *Subject:* [Potential Spoof] Re: Side meeting @ IETF 104 for QUIC
> Datagrams
>
>
>
> *From:* QUIC <[email-masked]> on behalf of Lucas Pardue <
> *Date:* Wednesday, March 27, 2019 at 10:37 AM
> *To:* Mikkel Fahnøe Jørgensen <[email-masked]>
> *Cc:* Eric Kinnear <[email-masked]>, David Schinazi <
> *Subject:* Re: Side meeting @ IETF 104 for QUIC Datagrams
>
>
>

---

From: Roberto Peon <[email-masked]>
Date: Wed, 27 Mar 2019 19:46:44 +0000

Thanks, Mikkel, Ryan!

Agreed that an ID is not required for sub-MTU messages within an already ID’d packet (else it can’t be reliable) for a protocol which is guaranteed to only be p2p.

If you have messages > MTU, this won’t work. If you need to route e2e (i.e. via proxies), then this will be insufficient. If you have to proxy (even with 1:1 mapping of connections) onto a network with a different (smaller) MTU, this will be… interesting.

I suspect that we don’t need an ID per message—we just need an ID for the totality of the messages in a packet. If we wanted both ordering and routing, we want an ID and an offset once per packet.
If we didn’t care about ordering, then we could have a ‘channel’ instead of a ‘stream’ (where streams require offsets).

In any e2e case, using a ‘channel’ would result in higher efficiency as you state the routing ID once per packet instead of needing to state the routing ID per message.

I don’t think that the channel solves all the problems either—in the changing MTU circumstance, things get messy without offsets/message-IDs.

If we didn’t try to be reliable, this wouldn’t be an issue (just drop the message). With reliable (ordered or unordered) messages, it is an issue.

Assuming that there was no efficiency difference between ordered/unordered messages, why would we ever want unordered messages? I can see how we might want out-of-order/scatter-gather sending and receiving, but that is not the same as unordered!
-=R

From: QUIC <[email-masked]> on behalf of Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wednesday, March 27, 2019 at 11:25 AM
To: Roberto Peon <[email-masked]>, Ryan Hamilton <[email-masked]>
Cc: Eric Kinnear <[email-masked]>, David Schinazi <[email-masked]>, Tommy Pauly <tpauly=[email-masked]>, Lucas Pardue <[email-masked]>, QUIC WG <[email-masked]>, Colin Perkins <[email-masked]>, Mirja Kuehlewind <[email-masked]>
Subject: Re: [Potential Spoof] Re: Side meeting @ IETF 104 for QUIC Datagrams

I almost agree that the ID is not needed for single MTU datagrams, but you don’t know if the packet you didn’t receive contained a datagram. Hence for applications where you need a sort of timestamping with gap detection, you do need something. It may be that this something is better contained with the user part of a datagram - however - if there are hundreds of datagrams in one packet, this is also not ideal. So - the special use case with gap detection is separate from the general use case of atomic single MTU (un)reliable datagram.

On 27 March 2019 at 19.09.55, Ryan Hamilton ([email-masked]<mailto:[email-masked]>) wrote:
For messages/datagrams which fit in a single packet a unique ID is not needed by the transport and consequently should probably not be part of the base requirements. It can be left up to the application to serialize these messages in whatever form they prefer. This might or might not include an ID internally, but QUIC has no need for this information.

It is nice if they also:
- are efficient

What am I missing?
-=R

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Roberto Peon <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, March 27, 2019 at 10:41 AM
To: Lucas Pardue <[email-masked]<mailto:[email-masked]>>, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>
Cc: Eric Kinnear <[email-masked]<mailto:[email-masked]>>, David Schinazi <[email-masked]<mailto:[email-masked]>>, Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>, QUIC WG <[email-masked]<mailto:[email-masked]>>, Colin Perkins <[email-masked]<mailto:[email-masked]>>, Tommy Pauly <tpauly=[email-masked]<mailto:[email-masked]>>
Subject: [Potential Spoof] Re: Side meeting @ IETF 104 for QUIC Datagrams

-=R

From: QUIC <[email-masked]<mailto:[email-masked]>> on behalf of Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, March 27, 2019 at 10:37 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>
Cc: Eric Kinnear <[email-masked]<mailto:[email-masked]>>, David Schinazi <[email-masked]<mailto:[email-masked]>>, Tommy Pauly <tpauly=[email-masked]<mailto:[email-masked]>>, QUIC WG <[email-masked]<mailto:[email-masked]>>, Colin Perkins <[email-masked]<mailto:[email-masked]>>, Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>
Subject: Re: Side meeting @ IETF 104 for QUIC Datagrams

Mikkel

Slides:

Best,
Tommy

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 27 Mar 2019 13:02:12 -0700

On 27 March 2019 at 20.46.58, Roberto Peon ([email-masked]) wrote:

Assuming that there was no efficiency difference between ordered/unordered
messages, why would we ever want unordered messages? I can see how we might
want out-of-order/scatter-gather sending and receiving, but that is not the
same as unordered!

It depends on what you mean by efficiency. But you want unordered to avoid
head of line blocking regardless of bandwidth, cpu and memory (buffering)
overhead.

In some more obscure cases perhaps also to ensure independence of events in
sampling applications such CERN collider sensors or Monte Carlo simulation.

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 27 Mar 2019 21:11:30 +0100

On Wed, 27 Mar 2019, 20:46 Roberto Peon, <[email-masked]> wrote:

> Thanks, Mikkel, Ryan!
>
> Agreed that an ID is not required for sub-MTU messages within an already
> ID’d packet (else it can’t be reliable) for a protocol which is guaranteed
> to only be p2p.
>
> If you have messages > MTU, this won’t work. If you need to route e2e
> (i.e. via proxies), then this will be insufficient. If you have to proxy
> (even with 1:1 mapping of connections) onto a network with a different
> (smaller) MTU, this will be… interesting.
>
FWIW DATAGRAM fragmentation was discussed in this session but there
appeared to be consensus on punting this off in to future.

Lucas

---

From: Ryan Hamilton <[email-masked]>
Date: Wed, 27 Mar 2019 13:44:19 -0700

> appeared to be consensus on punting this off in to future.

This seems utterly sensible to me.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 27 Mar 2019 21:07:12 +0000

By efficiency, I meant to address number-of-messages-per-second as a function of bytes/second of network throughput.

I think what you’re saying is that you don’t want HoL blocking, i.e. you want data received out of order to be forwarded to the application (and we want the application to be able to interpret it).

In-order delivery is orthogonal to the data being ordered—the data can always be delivered without HoL blocking, but we know for a sequence AB that when we receive B (without A), A should have preceded it.

Part of the reason I’m saying that this distinction is important is that the lack of this can cause many (most?) applications that deal with messages to have to invent their own sequencing/loss-discovery semantics. These will likely be more costly than what we can do at transport where we’re aware of packetization/MTU, not to mention it is sad to push that complexity into all such applications when it is something we already must solve (in order to provide reliability) in QUIC transport today.

-=R

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wednesday, March 27, 2019 at 1:02 PM
To: Roberto Peon <[email-masked]>, Ryan Hamilton <[email-masked]>
Cc: Tommy Pauly <tpauly=[email-masked]>, QUIC WG <[email-masked]>, Colin Perkins <[email-masked]>, Eric Kinnear <[email-masked]>, David Schinazi <[email-masked]>, Lucas Pardue <[email-masked]>, Mirja Kuehlewind <[email-masked]>
Subject: Re: [Potential Spoof] Re: Side meeting @ IETF 104 for QUIC Datagrams

On 27 March 2019 at 20.46.58, Roberto Peon ([email-masked]<mailto:[email-masked]>) wrote:
Assuming that there was no efficiency difference between ordered/unordered messages, why would we ever want unordered messages? I can see how we might want out-of-order/scatter-gather sending and receiving, but that is not the same as unordered!

It depends on what you mean by efficiency. But you want unordered to avoid head of line blocking regardless of bandwidth, cpu and memory (buffering) overhead.

In some more obscure cases perhaps also to ensure independence of events in sampling applications such CERN collider sensors or Monte Carlo simulation.