Subject: Re: [QUIC] Requirements

From: Jana Iyengar <[email-masked]>
Date: Tue, 7 Jun 2016 23:14:43 -0700

Michael,

> Perhaps a list of features that can be added to TCP can be discussed on
> TCPM, and similarly to SCTP on TSVWG. I'd be happy to chime in on those
> threads.
> I'm currently trying to figure out how such a list might look like.
>

I think the list you have is a fairly good one to start with. I would add
Versioning and Version Negotiation to the list -- this is fundamental to be
able to evolve the wire format.

- jana

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 8 Jun 2016 00:08:09 -0700

Randy,

Is this activity to:
>
> (a) document what Google as done and produce an RFC showing here is “Quic"
>          <or>
> (b) solving a specific problem
>
> If it is (b) it would make sense to figure out what the problem is and how
> it could be solved most easily. In this case one could consider evolving
> TCP (or even
> SCTP … but hey I do TCP right now :D). In case of (a) then this is not
> even something
> we need to discuss..
>

I think it's somewhere in between the two. IMO, the goal is to solve the
specific problem of developing a transport that minimizes latency for
applications; specifically, HTTP. QUIC was built expressly for this
purpose, and there is fairly extensive deployment experience with using
QUIC for HTTP. The BoF's charter is not clean-slate design -- QUIC exists,
it is being used to solve precisely the problem of being a low-latency HTTP
transport, and the charter says as much that the proposed goal is to start
with QUIC's current design as the initial seed and generalize it. There's
active interest from various entities in building and using QUIC, which is
the primary motivation for this standardization effort.

There's also the additional opportunity to jumpstart a next-generation
transport for modern applications, which should be a close second
motivation for this work at the IETF. We can usually find rough
translations from one transport to another, if we are willing to squint
hard enough and live with the resulting complexity. Ryan Bickhart's work on
mapping TCP semantics onto SCTP and my work with Bryan Ford on mapping
SCTP-like semantics onto TCP are both examples of how with enough work it
is feasible to retrofit a transport on top of or inside another. That said,
a clean-slate design has substantial benefits if the costs of building and
deploying it can be balanced out. QUIC shows an initial deployable (and
deployed) clean-slate design, and arguably, the costs of building and
deploying it have been considered reasonable by folks, at Google and
elsewhere, currently working on developing and deploying QUIC.

- jana

---

From: Brian Trammell <[email-masked]>
Date: Wed, 8 Jun 2016 09:11:30 +0200

hi Jana, all,

> On 08 Jun 2016, at 08:14, Jana Iyengar <[email-masked]> wrote:
> 
> Michael,
> 
> > Perhaps a list of features that can be added to TCP can be discussed on TCPM, and similarly to SCTP on TSVWG. I'd be happy to chime in on those threads.
> I think the list you have is a fairly good one to start with. I would add Versioning and Version Negotiation to the list -- this is fundamental to be able to evolve the wire format.

+1. (It is of course tricky to add versioning to a protocol that doesn't have it yet. But that doesn't make it not a requirement. :) )

Cheers,

Brian

> - jana
> _______________________________________________
> QUIC mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/quic

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 08 Jun 2016 09:29:56 +0200

Versioning is almost always the wrong way to hand evolution.

(One recent article that made this point in passing:

https://www.imperialviolet.org/2016/05/16/agility.html
)

In any case, the requirement is not for versioning, but for supporting
evolution.

Grüße, Carsten

Jana Iyengar wrote:
> I would add Versioning and Version Negotiation to the list -- this is
> fundamental to be able to evolve the wire format.

---

From: Brian Trammell <[email-masked]>
Date: Wed, 8 Jun 2016 09:41:23 +0200

hi Carsten,

> On 08 Jun 2016, at 09:29, Carsten Bormann <[email-masked]> wrote:
> 

Better put.

Fully flexible capability exposure and negotiation is the other end of this continuum, and can turn into a somewhat different nightmare than that described if not carefully managed. To stretch the article's central metaphor, it is very difficult to fully oil a joint that has degrees of freedom in dimensions you can't see.

I think we agree on the end goal, though, and perhaps phrasing it as "support evolution of the protocol, including stable and interoperable coexistence of protocol instances at different phases of this evolution" captures the goal without bias toward one or another end of this continuum of possible designs.

Cheers,

Brian

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 8 Jun 2016 09:46:52 +0200

> I think the list you have is a fairly good one to start with. I would add Versioning and Version Negotiation to the list -- this is fundamental to be able to evolve the wire format.
I think we agree, that the protocol must provide a way to evolve it.
Using version numbers and version negotiation is one way. Negotiating
features is another one. That would allow for a backwards compatible
way of evolution.

Best regards
Michael

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Wed, 8 Jun 2016 08:57:15 +0000

Given what is written below, I would prefer that an IETF protocol does not use the name of a proprietary protocol.

Also, given that not everybody seems to be able to reproduce the claimed performance benefits, given the lack of hardware segmentation offload etc., I fear that the proposed BoF name could raise wrong expectations.

However, given that there is running code, I could imagine to use e.g. “q” in an acronym of the IETF version of a protocol.

Michael

From: QUIC [mailto:[email-masked]] On Behalf Of Jana Iyengar
Sent: Wednesday, June 08, 2016 9:08 AM
To: Randall Stewart
Cc: Michael Tüxen; [email-masked]
Subject: Re: [QUIC] Requirements

Randy,

Is this activity to:

If it is (b) it would make sense to figure out what the problem is and how
it could be solved most easily. In this case one could consider evolving TCP (or even
SCTP … but hey I do TCP right now :D). In case of (a) then this is not even something
we need to discuss..

I think it's somewhere in between the two. IMO, the goal is to solve the specific problem of developing a transport that minimizes latency for applications; specifically, HTTP. QUIC was built expressly for this purpose, and there is fairly extensive deployment experience with using QUIC for HTTP. The BoF's charter is not clean-slate design -- QUIC exists, it is being used to solve precisely the problem of being a low-latency HTTP transport, and the charter says as much that the proposed goal is to start with QUIC's current design as the initial seed and generalize it. There's active interest from various entities in building and using QUIC, which is the primary motivation for this standardization effort.

There's also the additional opportunity to jumpstart a next-generation transport for modern applications, which should be a close second motivation for this work at the IETF. We can usually find rough translations from one transport to another, if we are willing to squint hard enough and live with the resulting complexity. Ryan Bickhart's work on mapping TCP semantics onto SCTP and my work with Bryan Ford on mapping SCTP-like semantics onto TCP are both examples of how with enough work it is feasible to retrofit a transport on top of or inside another. That said, a clean-slate design has substantial benefits if the costs of building and deploying it can be balanced out. QUIC shows an initial deployable (and deployed) clean-slate design, and arguably, the costs of building and deploying it have been considered reasonable by folks, at Google and elsewhere, currently working on developing and deploying QUIC.

- jana

---

From: Brian Trammell <[email-masked]>
Date: Wed, 8 Jun 2016 11:05:49 +0200

The obvious candidate is QUAC: QUIC-based UDP Application-focused Connection-oriented transport (or similar)

Cheers,

Brian

> On 08 Jun 2016, at 10:57, Scharf, Michael (Nokia - DE) <[email-masked]> wrote:
>

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Wed, 8 Jun 2016 09:07:14 +0000

Quick User Data Protection

Michael

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Brian Trammell
Sent: Wednesday, June 08, 2016 11:06 AM
To: Scharf, Michael (Nokia - DE)
Cc: Jana Iyengar; [email-masked]; Randall Stewart; Michael Tüxen
Subject: Re: [QUIC] Requirements

Cheers,

Brian

> If it is (b) it would make sense to figure out what the problem is and 
> how it could be solved most easily. In this case one could consider 
> evolving TCP (or even SCTP … but hey I do TCP right now :D). In case 
> of (a) then this is not even something we need to discuss..
>

---

From: Randall Stewart <[email-masked]>
Date: Wed, 8 Jun 2016 10:01:36 -0400

Carsten:

+1 on the idea its transport evolution not necessarily versioning.

A side note to your weblink on TLS, I can say the same thing with experience in SCTP’s compatibility
bits.. in all of the many SCTP interop’s the proper application of the upper two bits in chunks and
parameters was always a sore point that was done wrong. It never failed to cause us issues.. hopefully
most implementations of SCTP have it right now (but I doubt it :-D)

R
> On Jun 8, 2016, at 3:29 AM, Carsten Bormann <[email-masked]> wrote:
> 

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 8 Jun 2016 16:46:01 +0200

> On 08 Jun 2016, at 16:01, Randall Stewart <[email-masked]> wrote:
> 
> Carsten:
> 
My experience with testing shows that implementations have it right now... Both on the chunk and
parameter level.

Part of the confusion was about a copy and paste error in RFC 2960. After this got fixed in the
Implementor's Guide, things got better.

Best regards
Michael

---

From: Randall Stewart <[email-masked]>
Date: Wed, 8 Jun 2016 11:28:52 -0400

> On Jun 8, 2016, at 10:46 AM, Michael Tuexen <[email-masked]> wrote:
> 

Michael:

I agree that.. all we have seen the last few interop’s confirm that.. but my “doubt/:-D” above was
that if someone builds a new implementation of SCT there would be problems.. since that
did tend to be one of the principle problem spots that early interop’s saw :)

> Implementor's Guide, things got better.

Maybe you are right and its all due to the mis-wordings in RFC 2960, or could it be just
be time, experience and inter-ops :-)

R

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 8 Jun 2016 14:57:07 -0700

On Wed, Jun 8, 2016 at 12:46 AM, Michael Tuexen <
[email-masked]> wrote:

> > > Perhaps a list of features that can be added to TCP can be discussed
> on TCPM, and similarly to SCTP on TSVWG. I'd be happy to chime in on those
> threads.
> > I think the list you have is a fairly good one to start with. I would
> add Versioning and Version Negotiation to the list -- this is fundamental
> to be able to evolve the wire format.

Yup, agreed.

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 8 Jun 2016 15:26:50 -0700

Michael,

On Wed, Jun 8, 2016 at 1:57 AM, Scharf, Michael (Nokia - DE) <
[email-masked]> wrote:

> Given what is written below, I would prefer that an IETF protocol does not
> use the name of a proprietary protocol
>
Also, given that not everybody seems to be able to reproduce the claimed
> performance benefits, given the lack of hardware segmentation offload etc.,
> I fear that the proposed BoF name could raise wrong expectations.
>
Unless it's miscommunication, you seem to be suggesting that others have
tried and failed to realize some/any of the claimed benefits of QUIC. I'm
not aware of any such work. I also don't understand what you mean by "lack
of hardware segmentation offload". Can you clarify?

I don't understand what wrong expectations the BoF name raises. To be
clear, there may well be, and if there are any such, I think we definitely
ought to at least clarify them in the charter text. Can you be more
specific about the wrong expectations you see?

- jana

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Thu, 9 Jun 2016 08:03:46 +0000

My concern is that the current charter suggestion markets the protocol by “minimizing (…) overall transport latency for applications”. IMHO this can have rebound effects.

Here is what I mean by wrong expectations: Whether a protocol is indeed fast or not depends on many parameters including aspects such as hardware acceleration/offload (which matters at 10G/40G line speed, to the best of my knowledge). Some of these effects of application latency are entirely independent of the protocol specification. But if the IETF creates the expectation that speed is the main differentiator of a protocol, this could have effects such as encouraging app developers to disable or “tune” the congestion control logic, to actually get the “promised” minimum latency... And if a protocol targets the Web, it will probably be used by diverse user communities, and not everybody may have the same understanding of what “congestion control agility” means to him...

Obviously, there is not much the IETF can do to prevent users from disabling or messing up congestion control, and I am fine with excluding congestion control in the charter.

But I believe the IETF could pick a protocol acronym and use charter wording that focuses more on the functional properties of protocol, to make clear that in the Internet not every transfer can be quick.

Michael

From: Jana Iyengar [mailto:[email-masked]]
Sent: Thursday, June 09, 2016 12:27 AM
To: Scharf, Michael (Nokia - DE)
Cc: Randall Stewart; Michael Tüxen; [email-masked]
Subject: Re: [QUIC] Requirements

Michael,

On Wed, Jun 8, 2016 at 1:57 AM, Scharf, Michael (Nokia - DE) <[email-masked]<mailto:[email-masked]>> wrote:

Given what is written below, I would prefer that an IETF protocol does not use the name of a proprietary protocol

Also, given that not everybody seems to be able to reproduce the claimed performance benefits, given the lack of hardware segmentation offload etc., I fear that the proposed BoF name could raise wrong expectations.
Unless it's miscommunication, you seem to be suggesting that others have tried and failed to realize some/any of the claimed benefits of QUIC. I'm not aware of any such work. I also don't understand what you mean by "lack of hardware segmentation offload". Can you clarify?

I don't understand what wrong expectations the BoF name raises. To be clear, there may well be, and if there are any such, I think we definitely ought to at least clarify them in the charter text. Can you be more specific about the wrong expectations you see?

- jana

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 9 Jun 2016 11:06:07 +0200

> On 09 Jun 2016, at 10:03, Scharf, Michael (Nokia - DE) <[email-masked]> wrote:
> 
> Here is what I mean by wrong expectations: Whether a protocol is indeed fast or not depends on many parameters including aspects such as hardware acceleration/offload (which matters at 10G/40G line speed, to the best of my knowledge). Some of these effects of 
I think this refers to high throughput. For TCP, offloading TCP processing to a NIC improves performance
of the TCP processing. I'm not sure if this still helps if encryption/decryption comes into the game.
> application latency are entirely independent of the protocol specification. But if the IETF creates the expectation that speed is the main differentiator of a protocol, this could have effects such as encouraging app developers to disable or “tune” the
I think term speed is too generic. Depending on whom you talk this can mean:
* high bandwidth
* low latency
* something else.
So I do think the term "minimising the transport latency" is more precisely.

Best regards
Michael
> congestion control logic, to actually get the “promised” minimum latency... And if a protocol targets the Web, it will probably be used by diverse user communities, and not everybody may have the same understanding of what “congestion control agility” means to him...
>  
> On Wed, Jun 8, 2016 at 1:57 AM, Scharf, Michael (Nokia - DE) <[email-masked]> wrote:

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 9 Jun 2016 11:08:39 +0200

> On 08 Jun 2016, at 17:28, Randall Stewart <[email-masked]> wrote:
> 
> 
The point you are referring to is the handling of unknown chunk / parameters.
Newer SCTP extensions use the SUPPORTED_CHUNK_TYPES parameter to do the "feature negotiation".
This works pretty well and I haven't seen problems with that. It allow a pretty fine granular
negotiation.

Best regards
Michael

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Thu, 9 Jun 2016 09:32:18 +0000

Whether "transport latency" is dominate by throughput or by protocol handshakes may depend on the use case. The HTTP communication pattern of a Web browser at a residential Internet user may be different to, say, the HTTP communication pattern of a real-time big data analytics application inside a data center.

And, for what it is worth, I believe that people who really care about latency, but not about throughput, currently e.g. terminate TCP inside FPGAs directly on the NIC. In that use case, I have heard that the end-to-end application "transport latency", over a WAN, targets at 10 microseconds. (Disclaimer: I don't know whether TLS and HTTP is used in this use case).

"Transport latency" can mean quite different things to different people. In the current charter wording I don't find statements to limit the scope to Web browsing for users with (narrow-band) Internet access.

Michael

> From: QUIC [mailto:[email-masked]] On Behalf Of Michael Tuexen
> Sent: Thursday, June 09, 2016 11:06 AM
> Cc: Jana Iyengar; [email-masked]; Randall Stewart
> > My concern is that the current charter suggestion markets the
> protocol by “minimizing (…) overall transport latency for
> applications”. IMHO this can have rebound effects.
> >
> > Here is what I mean by wrong expectations: Whether a protocol is
> indeed fast or not depends on many parameters including aspects such as
> hardware acceleration/offload (which matters at 10G/40G line speed, to
> the best of my knowledge). Some of these effects of
> of the TCP processing. I'm not sure if this still helps if
> encryption/decryption comes into the game.
> > application latency are entirely independent of the protocol
> specification. But if the IETF creates the expectation that speed is
> the main differentiator of a protocol, this could have effects such as
> encouraging app developers to disable or “tune” the
> > congestion control logic, to actually get the “promised” minimum
> latency... And if a protocol targets the Web, it will probably be used
> by diverse user communities, and not everybody may have the same
> understanding of what “congestion control agility” means to him...
> >
> > Obviously, there is not much the IETF can do to prevent users from
> disabling or messing up congestion control, and I am fine with
> excluding congestion control in the charter.
> >
> > But I believe the IETF could pick a protocol acronym and use charter
> wording that focuses more on the functional properties of protocol, to
> make clear that in the Internet not every transfer can be quick.
> >
> > Michael
> >
> >
> >
> > Given what is written below, I would prefer that an IETF protocol
> does not use the name of a proprietary protocol
> >
> > Also, given that not everybody seems to be able to reproduce the
> claimed performance benefits, given the lack of hardware segmentation
> offload etc., I fear that the proposed BoF name could raise wrong
> expectations.
> >
> > Unless it's miscommunication, you seem to be suggesting that others
> have tried and failed to realize some/any of the claimed benefits of
> QUIC. I'm not aware of any such work. I also don't understand what you
> mean by "lack of hardware segmentation offload". Can you clarify?
> >
> clear, there may well be, and if there are any such, I think we
> definitely ought to at least clarify them in the charter text. Can you
> be more specific about the wrong expectations you see?
> >
> > - jana
>

---

From: Randall Stewart <[email-masked]>
Date: Thu, 9 Jun 2016 06:30:34 -0400

> On Jun 9, 2016, at 5:06 AM, Michael Tuexen <[email-masked]> wrote:
> 
> 

I too was thinking Michael meant offloading such as TSO and LRO. I know on most of the caches
I develop on if I turn off TSO and LRO performance of the box drops by roughly 50%… and I think
this will be a consequence of encrypting everything since I don’t think you can do TSO/LRO unless
the hardware/lower-layers has access to things equivalent to TCP sequence numbers and ACK values,
which is what I believe the encryption is meant to hide (from middle boxes). 

That will be an extreme price to pay the question is, is it worth it /and or/ is there some other way
to get the benefits of the TSO/LRO even with encryption… :-)

R

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Brian Trammell <[email-masked]>
Date: Thu, 9 Jun 2016 12:41:29 +0200

hi Randall,

> On 09 Jun 2016, at 12:30, Randall Stewart <[email-masked]> wrote:
> 
> 

(adding [email-masked], as this is a relevant question for PLUS, too):

The question is what is the minimum information xSO/LRO needs to work. If I understand how generic segment offload works, if QUIC(/PLUS) exposes:

(1) a header field that can be used to order packets and find gaps that the kernel can find and interpret,

(2) a defined and easily-found boundary between header and payload, such that the payload retains any framing/message boundaries needed to meet API contracts, and

(3) rules about which header to choose for the "big" segment on receive

then adding QUIC(/PLUS) aware SO/LRO to the kernel should be reasonably simple.  I don't see why you need ACK-equivalents to be exposed to make this work, but I might be missing something.

Hardware offload takes longer to deploy, obviously, but runs on equivalent information.

Cheers,

Brian

---

From: Randall Stewart <[email-masked]>
Date: Thu, 9 Jun 2016 07:02:52 -0400

> On Jun 9, 2016, at 6:41 AM, Brian Trammell <[email-masked]> wrote:
> 
> hi Randall,
> 

My understanding of TSO (which is where we find the most benefit since its hardware) is the hardware gets a *large* segment (say 64k) and
cuts it up into MTU sized pieces duplicating the IP header and doing the right things with the tcp-sequence number.

If the sequence number is encrypted and not exposed that would be rather difficult with Quic (or any such protocol for
that matter even TCP if it was wrapped in UDP and encrypted).

The other benefit is of course comes from LRO and thats really not done in hardware but in software (at least in the
implementation on which I work that is) .. where Acks <or> Segments are combined into bigger chunks. 

For my work the Acks are what I care about having to process less Acks is a  win (but not nearly as much as TSO). 
Both Seq and Ack can be compiled by LRO into a “larger segment” so you need again the exposure of the Ack value as well
as the sequence number..

R

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Christian Huitema <[email-masked]>
Date: Thu, 9 Jun 2016 19:17:43 +0000

On Thursday, June 9, 2016 3:31 AM, Randall Stewart wrote:
> 
> > On Jun 9, 2016, at 5:06 AM, Michael Tuexen
> <[email-masked]> wrote:
> >
> >
> >>...
> >> indeed fast or not depends on many parameters including aspects such
> >> as hardware acceleration/offload (which matters at 10G/40G line
> >> speed, to the best of my knowledge). Some of these effects of
> > I think this refers to high throughput. For TCP, offloading TCP
> > processing to a NIC improves performance of the TCP processing. I'm not sure
> if this still helps if encryption/decryption comes into the game.
> 
> I too was thinking Michael meant offloading such as TSO and LRO. I know on
> most of the caches I develop on if I turn off TSO and LRO performance of the
> box drops by roughly 50%… and I think this will be a consequence of encrypting
> everything since I don’t think you can do TSO/LRO unless the hardware/lower-
> layers has access to things equivalent to TCP sequence numbers and ACK
> values, which is what I believe the encryption is meant to hide (from middle
> boxes).

Sure. But then, which part of RFC 793 describes hardware offload? None, of course. It was developed over time, without involving the IETF. Basically, some innovators observed the installed based, looked at all these TCP connections, and saw an opportunity for a new product. Clearly, if you compare TCP to any new protocol, you will find lots of innovations for TCP, including hardware offload, and zilch for the new protocol.

On the other hand, if the new protocol is actually deployed in large numbers, we can surely expect some innovators to see a business opportunity and develop solutions. Yes, it will take some time. But it will happen eventually.

-- Christian Huitema

---

From: Randall Stewart <[email-masked]>
Date: Fri, 17 Jun 2016 06:38:28 -0700

All:

So I think the final list of features that have so far been mentioned as requirements are:

1. Legacy NAT traversal (doable via UDP encapsulation)
2. Pacing
3. FEC
4. Path MTU limit
5. Fast connection setup
6. Improved loss recovery
7. CC agility
8. Encryption of transport layer information
9. Multi-Streaming
10. Implementable in User Space
11. 0-RTT connection setup^^

Now from the perspective of my employer and our current network, I would like to see 
a couple of other things added:

12. It *must* be implementable in the kernel (i.e. no floating point, or at least integer equivalents to 
      any floating point operations defined).
13. The ability to do LRO/TSO in some fashion, this does *not* exclude the possibility of
      enhanced hardware (i.e. we are not limited to the current set of hardware).. but we should
      have some provision to gain TSO especially. I realize this is in opposition (somewhat) to
      the encrypt every part but a bare minimum seen by middle boxes.. but I think its needed :-)

^^ In cases where two endpoints have never talked to each other 0 RTT would not be required.
--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Brian Trammell <[email-masked]>
Date: Fri, 17 Jun 2016 16:00:07 +0200

> On 17 Jun 2016, at 15:38, Randall Stewart <[email-masked]> wrote:
> 
> All:
> 

One clarifying question:

Is the requirement here really "segmentation offload," or is the requirement "minimize context switching"?

Cheers,

Brian

---

From: Randall Stewart <[email-masked]>
Date: Fri, 17 Jun 2016 07:03:38 -0700

> On Jun 17, 2016, at 7:00 AM, Brian Trammell <[email-masked]> wrote:
> 
>> 
> Is the requirement here really "segmentation offload," or is the requirement "minimize context switching”?

Its not minimizing context switching :-)

We really need Segmentation offload.. when that is turned off the CPU performance goes into the
dumper… 

Without TSO/LRO (and especially TSO) our performance is cut by at least 50%

R

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 17 Jun 2016 16:49:10 +0200

> On 17 Jun 2016, at 16:03, Randall Stewart <[email-masked]> wrote:
> 
> 
Would it be a way to say

13. Ability for offload protocol processing to hardware, for example by future
    NICs.

This would allow things like LRO/TSO (which means some sort of packet batching),
checksumming, hashing, encryption, decryption offload, or whatever makes sense
for the particular protocol.

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 17 Jun 2016 16:51:56 +0200

> 1. Legacy NAT traversal (doable via UDP encapsulation)
I think FEC is at a different requirements level as the other requirements. At least this is
not considered as a core feature according to
https://github.com/IETF-QUIC/Charter/blob/master/quic-charter.md

Best regards
Michael

---

From: Randall Stewart <[email-masked]>
Date: Mon, 20 Jun 2016 15:01:33 -0700

sure that would be fine :-)

One other thing that should be added.. and I say this with my little bit of
Quic implémentation experience…. (not much).. 

* ) Older Quic implementations MUST be able to interoperate with newer ones..

That may be a bit unclear so let  me explain.

A while ago (last Sept?) we had someone hack up nginx and force fit Googles Chrome/Quic into place.
This got shelved before the person finished and left for a sabbatical.

Now I was asked in Dec to work on this to see if I could get it to “work”. I found a few bugs in the
way it was done fixed those and was still having problems.

Jana was nice enough to offer to help me out (thanks!) but when we connected my older implementation
with his, they would *not* talk.. they would just give an error msg “that guys too old for me to talk to”.

Now when building a transport prototyping system (which is what googles Quic is) thats probably ok, since
you can fall back to TCP. But for the *new* IETF-Quic that to me is unacceptable. Just like an older version
of TCP can talk to a newer version that supports SACK and new features, or SCTP implementations can
talk to each other with diverse feature sets, any IETF-Quic will need to talk to older version number (if we
even keep the version number.. after all everything will be on the table for change here.. and we may
want to go with just a capabilities negotiation instead).. but I digress.. my point is we should have a requirement
that makes in mandatory that older and newer implementations of the protocol should be inter-operable..

That may be “obvious” to some, but after my experience I would like to have that noted down :-)

R

> On Jun 17, 2016, at 7:49 AM, Michael Tuexen <[email-masked]> wrote:
> 
>> 

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Ian Swett <[email-masked]>
Date: Tue, 21 Jun 2016 15:16:53 -0400

I would agree with you Brian that QUIC/PLUS could be offloaded if #2 was
present.  #3 is not an issue if the header is encrypted, since it's just
part of the encrypted segment.  If it wasn't encrypted, I would suggest it
either be omitted from the non-first packets or identical across all
packets.  But either way, it's a solvable problem.

That being said, I think we should think hard about whether we want to
solve that problem, because it involves a lot of complexity and I believe
we'd be better served in ensuring there are simple send/receive paths that
are very fast.

In regards to acks and LRO, some middleboxes/endpoints will drop all but
the most recent ack if multiple arrive in close proximity.  It's not clear
this is always a feature from a congestion control perspective, but it
likely increases throughput in some environments.  However, given QUIC/PLUS
can easily change the receiver's behavior, we can send fewer acks.  Ideally
the sender would be aware of that, so it can adjust it's congestion control
to expect long pauses.  I think this achieves the same goal end-to-end and
CPU benchmarks in QUIC indicate it performs very well.

I'm hugely concerned about the CPU performance of QUIC, but nothing has
convinced me LRO or TSO would help enough to be worth the work.  Instead,
sending fewer acks and faster send/receive APIs show a lot more promise, at
least on the host side.

On Thu, Jun 9, 2016 at 6:41 AM, Brian Trammell <[email-masked]> wrote:

> >>> My concern is that the current charter suggestion markets the protocol
> by “minimizing (…) overall transport latency for applications”. IMHO this
> can have rebound effects.
> >>>
> hardware acceleration/offload (which matters at 10G/40G line speed, to the
> best of my knowledge). Some of these effects of
> processing to a NIC improves performance
> The question is what is the minimum information xSO/LRO needs to work. If
> I understand how generic segment offload works, if QUIC(/PLUS) exposes:
>
> (2) a defined and easily-found boundary between header and payload, such
> that the payload retains any framing/message boundaries needed to meet API
> contracts, and
>
> then adding QUIC(/PLUS) aware SO/LRO to the kernel should be reasonably
> simple.  I don't see why you need ACK-equivalents to be exposed to make
> this work, but I might be missing something.
>