Subject: [tsvwg] Agenda requests for TSVWG@IETF101

From: Yingzhen Qu <[email-masked]>
Date: Sun, 4 Mar 2018 05:54:37 +0000

Dear Chairs,

A new draft (The draft was suggested by TSVWG @IETF100) was just submitted, and we’d like to request a time slot to present it @IETF101.

Title: A New Congestion Control in Bandwidth Guaranteed Network
Presenter: Yingzhen Qu (Huawei)
Time required (including Q/A): 10 mins
Draft: https://datatracker.ietf.org/doc/draft-han-tsvwg-cc/

If there is any question, please kindly let us know.

Thanks,
Yingzhen

---

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Sun, 4 Mar 2018 10:34:29 +0000

Hi all,

From the abstract: “… This draft proposes a new TCP congestion control algorithm used in bandwidth guaranteed networks.  It is an extension to the current TCP standards.”

In the IETF, I believe the expertise for this specific document would be in TCPM, which in CC. If the authors are interested in feedback on the proposed mechanism, I would recommend to ask TCPM.

Alternatively, corresponding research could perhaps be performed in the ICCRG. ICCRG has published RFC 6077 to document some of the open research issues in this space.

Michael

From: tsvwg [mailto:[email-masked]] On Behalf Of Yingzhen Qu
Sent: Sunday, March 04, 2018 6:55 AM
To: [email-masked]; [email-masked]
Cc: Thomas Nadeau <[email-masked]>
Subject: [tsvwg] Agenda requests for TSVWG@IETF101

Dear Chairs,

Thanks,
Yingzhen

---

From: Emmanuel Lochin <[email-masked]>
Date: Sun, 4 Mar 2018 12:45:04 +0100

Hi Yingzhen

Just for your information, you might be interested in gTFRC : 
https://tools.ietf.org/html/draft-lochin-ietf-tsvwg-gtfrc-02
which is a similar solution i.e. a CC for bandwidth guaranteed networks 
with TFRC.
Furthermore just a quick question : in your draft does AR/VR stands for 
Augmented/Real Virtuality ?

Regards

Emmanuel

On 04/03/2018 06:54, Yingzhen Qu wrote:
>
> Dear Chairs,
>
> A new draft (The draft was suggested by TSVWG @IETF100) was just 
> submitted, and we’d like to request a time slot to present it @IETF101.
>
> Title:A New Congestion Control in Bandwidth Guaranteed Network
>
> <https://datatracker.ietf.org/doc/draft-han-tsvwg-cc/>
>

-- 
Emmanuel LOCHIN
Professeur ISAE

ISAE SUPAERO - Institut Supérieur de l'Aéronautique et de l'Espace
10 avenue Edouard Belin - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE - http://www.isae-supaero.fr
Tel +33 5 61 33 84 85 - Fax (+33) 5 61 33 83 30

---

From: Gorry Fairhurst <[email-masked]>
Date: Sun, 04 Mar 2018 12:49:41 +0000

I am unsure yet what the correct group of people world be to explore a 
"Bandwidth Guaranteed Network". The presentation last IETF looked like 
the work could imply a need for changes proposed to the network layer 
(using OAM exchnages) to set the sending rate and make those bandwidth 
reservations.  In the end, it could result in a protocol quite different 
to TCP, I think this sort of change may possibly have a home in TSVWG  - 
but first I'd agree with Michaeland would encourage a presentation of 
the problem statement in ICCRG to explore the issues.

Gorry

On 04/03/2018, 10:34, Scharf, Michael (Nokia - DE/Stuttgart) wrote:
>
> Hi all,
>
> From the abstract: “…This draft proposes a new TCP congestion control 
> algorithm used in bandwidth guaranteed networks.  It is an extension 
> to the current TCP standards.”
>
> In the IETF, I believe the expertise for this specific document would 
> be in TCPM, which in CC. If the authors are interested in feedback on 
> the proposed mechanism, I would recommend to ask TCPM.
>
> Alternatively, corresponding research could perhaps be performed in 
> the ICCRG. ICCRG has published RFC 6077 to document some of the open 
> research issues in this space.
>
> Michael
>
> *From:*tsvwg [mailto:[email-masked]] *On Behalf Of *Yingzhen Qu
> *Sent:* Sunday, March 04, 2018 6:55 AM
> *To:* [email-masked]; [email-masked]
> *Cc:* Thomas Nadeau <[email-masked]>
> *Subject:* [tsvwg] Agenda requests for TSVWG@IETF101
>
> Dear Chairs,
>

---

From: Yingzhen Qu <[email-masked]>
Date: Sun, 4 Mar 2018 12:48:12 -0800

Hi Emmanuel,

Thanks for the pointer. I was not aware of the existence of this draft. I
agree they have some similarities with detail variations. Just curious, why
this draft didn't progress?

BTW, yes, AR/VR means Augmented Reality/Virtual Reality. I'll add the
acronyms to the next version.

Thanks,
Yingzhen

On Sun, Mar 4, 2018 at 3:45 AM, Emmanuel Lochin <
[email-masked]> wrote:

> ISAE SUPAERO - Institut Supérieur de l'Aéronautique et de l'Espace10 avenue Edouard Belin <https://maps.google.com/?q=10+avenue+Edouard+Belin&entry=gmail&source=g> - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE - http://www.isae-supaero.fr
> Tel +33 5 61 33 84 85 <+33%205%2061%2033%2084%2085> - Fax (+33) 5 61 33 83 30 <+33%205%2061%2033%2083%2030>
>
>

---

From: Yingzhen Qu <[email-masked]>
Date: Sun, 4 Mar 2018 12:58:58 -0800

Hi Gorry and Michael,

Thanks for the suggestion, I'll request a presentation in ICCRG. Meanwhile,
I think since the in-band signaling draft was presented in TSVWG, if time
allows it still makes sense to present this draft in TSVWG.

The in-band signaling draft covers lots of aspects, and the required
changes include network layer and transport layer. We're working on
updating the draft, and may break it into pieces to fit different WGs.

Your comments and help are very much appreciated.

Thanks,
Yingzhen

On Sun, Mar 4, 2018 at 4:49 AM, Gorry Fairhurst <[email-masked]>
wrote:

> "Bandwidth Guaranteed Network". The presentation last IETF looked like the
> work could imply a need for changes proposed to the network layer (using
> OAM exchnages) to set the sending rate and make those bandwidth
> reservations.  In the end, it could result in a protocol quite different to
> TCP, I think this sort of change may possibly have a home in TSVWG  - but
> first I'd agree with Michaeland would encourage a presentation of the
> problem statement in ICCRG to explore the issues.
>
> Gorry
>
>> algorithm used in bandwidth guaranteed networks.  It is an extension to the
>> current TCP standards.”
>>
>> In the IETF, I believe the expertise for this specific document would be
>> in TCPM, which in CC. If the authors are interested in feedback on the
>> proposed mechanism, I would recommend to ask TCPM.
>>
>> Alternatively, corresponding research could perhaps be performed in the
>> ICCRG. ICCRG has published RFC 6077 to document some of the open research
>> issues in this space.
>>
>> Michael
>>

---

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Mon, 5 Mar 2018 08:09:39 +0000

I believe that this proposal is similar to QuickStart TCP (RFC 4782), which is not cited in draft-han-tsvwg-cc, and the reference is also missing in draft-han-6man-in-band-signaling-for-transport-qos.

RFC 6077 explains some of the issues that an in-band signaling mechanism like Quick-Start has to solve. As far as I can tell, the fundamental challenge is neither the protocol specification nor a prototype implementation. For instance, it has been proven that QuickStart TCP can be implemented e.g. in network processors (see http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf).

So, when updating the documents, I suggest to add a discussion of how the open research issues explained in RFC 6077 are addressed.

Michael

From: Yingzhen Qu [mailto:[email-masked]]
Sent: Sunday, March 04, 2018 9:59 PM
To: [email-masked]
Cc: Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>; Thomas Nadeau <[email-masked]>; [email-masked]; Yingzhen Qu <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [tsvwg] Agenda requests for TSVWG@IETF101

Hi Gorry and Michael,

Thanks for the suggestion, I'll request a presentation in ICCRG. Meanwhile, I think since the in-band signaling draft was presented in TSVWG, if time allows it still makes sense to present this draft in TSVWG.

The in-band signaling draft covers lots of aspects, and the required changes include network layer and transport layer. We're working on updating the draft, and may break it into pieces to fit different WGs.

Thanks,
Yingzhen

On Sun, Mar 4, 2018 at 4:49 AM, Gorry Fairhurst <[email-masked]<mailto:[email-masked]>> wrote:
I am unsure yet what the correct group of people world be to explore a "Bandwidth Guaranteed Network". The presentation last IETF looked like the work could imply a need for changes proposed to the network layer (using OAM exchnages) to set the sending rate and make those bandwidth reservations.  In the end, it could result in a protocol quite different to TCP, I think this sort of change may possibly have a home in TSVWG  - but first I'd agree with Michaeland would encourage a presentation of the problem statement in ICCRG to explore the issues.

Gorry

Hi all,

From the abstract: “…This draft proposes a new TCP congestion control algorithm used in bandwidth guaranteed networks.  It is an extension to the current TCP standards.”

Michael

*From:*tsvwg [mailto:[email-masked]<mailto:[email-masked]>] *On Behalf Of *Yingzhen Qu
*Sent:* Sunday, March 04, 2018 6:55 AM
*To:* [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
*Cc:* Thomas Nadeau <[email-masked]<mailto:[email-masked]>>
*Subject:* [tsvwg] Agenda requests for TSVWG@IETF101

Dear Chairs,

Thanks,

Yingzhen

---

From: Emmanuel Lochin <[email-masked]>
Date: Mon, 5 Mar 2018 10:16:19 +0100

Le 04/03/2018 à 21:48, Yingzhen Qu a écrit :
> Hi Emmanuel,
>
> Thanks for the pointer. I was not aware of the existence of this 
> draft. I agree they have some similarities with detail variations. 
> Just curious, why this draft didn't progress?

Hi Yingzhen,

What I remember (looking back IETF archives) is that Sally Floyd 
answered she had no objection concerning the proposal if it was clearly 
written that the gTFRC mechanism must be used only in the context of a 
QoS network. Sally Floyd also asked questions related to the algorithm 
and proposed enhancements about the slow-start behavior of the 
mechanism. We answered that we fully agreed with this hypothesis and 
justified the choice of our design concerning the slow-start method.

Therefore, a second discussion started with Mark Allman on potential 
security issues raised by this mechanism. Basically, Mark asked us to 
add a part concerning the reaction to the congestion in the guaranteed 
part (the “g” part) even if we are in a context of a QoS network. We 
have discussed this point and following this emails exchange, published 
a new version of the draft.

We also talked about the best way to publish this proposal, it seemed to 
be well-accepted to publish a new draft rather than inserting this 
contribution inside the new TFRC draft (now RFC 5348). As a result, we 
extended the gTFRC draft in order to detail a complete protocol for QoS 
networks.

Finally I left the laboratory I worked during the elaboration of this 
draft in 2007 and I assume that this work has not been pursued.

Considering your proposal, I reckon you'll be face to the same questions 
in particular the one on the security issue. But I'm quite interesting 
in discussing this proposal on the mailing-list while even doing 
something more generic about transport protocol extensions for QoS 
networks.

EL

> On Sun, Mar 4, 2018 at 3:45 AM, Emmanuel Lochin 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Yingzhen
>
>     <https://tools.ietf.org/html/draft-lochin-ietf-tsvwg-gtfrc-02>
>     which is a similar solution i.e. a CC for bandwidth guaranteed
>     networks with TFRC.
>     Furthermore just a quick question : in your draft does AR/VR
>     stands for Augmented/Real Virtuality ?
>
>     Regards
>
>     Emmanuel
>
>
>     10 avenue Edouard Belin
>     <https://maps.google.com/?q=10+avenue+Edouard+Belin&entry=gmail&source=g>  - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE -http://www.isae-supaero.fr
>     Tel+33 5 61 33 84 85 <tel:+33%205%2061%2033%2084%2085>  - Fax(+33) 5 61 33 83 30 <tel:+33%205%2061%2033%2083%2030>
>
>

-- 
Emmanuel LOCHIN
Professeur ISAE
ISAE SUPAERO - Institut Supérieur de l'Aéronautique et de l'Espace
10 avenue Edouard Belin - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE -
http://www.isae-supaero.fr
Tel +33 5 61 33 84 85 - Fax (+33) 5 61 33 83 45
Web : http://personnel.isae.fr/emmanuel-lochin/

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Mar 2018 21:28:40 +0100

Thanks, Michael,

> In the IETF, I believe the expertise for this specific document would be in TCPM, which in CC. If the authors are interested in feedback on the proposed mechanism, I would recommend to ask TCPM.

TCPM might be actually a good idea, and that WG's feedback would be highly welcome.
Let me send a separate email to the chairs and ask if they could
consider presenting this in TCPM.

> From the abstract: ?????? This draft proposes a new TCP congestion control algorithm used in bandwidth guaranteed networks.  It is an extension to the current TCP standards.???

The way i would describe it: This memo defines simple extensions to TCP-Reno CC
to support and leverage a pre-known network guaranteed (minimum) bandwidth (CIR) and
pre-known maximumum needed bandwidth (PIR).  

IMHO, this work is very much orthogonal to how you would determine PIR and CIR,
the draft mentions some IMHO cool new ideas how to do this, but this CC draft
by itself would be a great benefit to application developers today (assume 
CIR/PIR API extensions to transport service -> TAPS, etc. pp).

My most simple deployment scenario is simple GUI on video devices/apps, e.g.:
in home (STB) where i could select guaranted minimum qualiy (translated
by app into CIR), and (optional) maximum quality (PIR). On the parents living
room STB CIR would be set according to 4k, and maybe in one other room to FullHD,
and the rest has to start compete for available bandwidth all the way from *yuck*
QCIF).

Of course, with more and more high-bitrate apps like AR/VR and others, the
problem to support easy policy differentiation for different apps will increase,
but i always like to look first at the most common use case that could profit today.

I feel like we have discussed this problem forever, but i am not aware of any
actual progress that app developers could use. We really need to break
through the chicken & egg circle. IMHO we first need to enable the host
stacks to do something like this, and then folks will see a lot easier the
need to have better in-network mechanisms to actually guarantee CIR.

So, i wouldn't know something more pragmatic & simple than this proposal
based on TFP reno.

> Alternatively, corresponding research could perhaps be performed in the ICCRG. ICCRG has published RFC 6077 to document some of the open research issues in this space.

Nice overview for researchers. Can't quite find from browsing a section
related to this CIR/PIR knowledge issue, admission control is mentioned
only once in passing. Let me know what you think i should specifically
look into, and we'll be happy to relate to that RFC6077 work in the draft.

> A new draft (The draft was suggested by TSVWG @IETF100) was just submitted, and we???d like to request a time slot to present it @IETF101.
> 

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Mar 2018 21:58:18 +0100

On Mon, Mar 05, 2018 at 10:16:19AM +0100, Emmanuel Lochin wrote:
> answered she had no objection concerning the proposal if it was
> clearly written that the gTFRC mechanism must be used only in the
> context of a QoS network.

And if history serves me right, that draft of yours was before
the IETF figured out that there was and is a lot of traffic assuming
bandwidth guaranteees without sufficient network guarantees provided - and
ultimately, the concept of circuit breakers was born. I think for
these mechanisms we're proposing one could equally refine their
applicability to "badly QoS supported" networks by making them
automatically fall back to standard congestion control (in our drafts
case Reno without CIR).

> Sally Floyd also asked questions related
> to the algorithm and proposed enhancements about the slow-start
> behavior of the mechanism. We answered that we fully agreed with
> this hypothesis and justified the choice of our design concerning
> the slow-start method.

> security issues raised by this mechanism. Basically, Mark asked us
> to add a part concerning the reaction to the congestion in the
> guaranteed part (the ???g??? part) even if we are in a context of a
> QoS network. We have discussed this point and following this emails
> exchange, published a new version of the draft.

Right, your -02 section 3.4.2.

I told the authors of our draft that that was also a piece i
felt is missing in our draft. See above - i would today rephrase it
in terms of a "soft circuit breaker" - not breaking the flow at
all, but just breaking the assumption of have a CIR.

> We also talked about the best way to publish this proposal, it
> seemed to be well-accepted to publish a new draft rather than
> inserting this contribution inside the new TFRC draft (now RFC
> 5348). As a result, we extended the gTFRC draft in order to detail a
> complete protocol for QoS networks.
> 
> Finally I left the laboratory I worked during the elaboration of
> this draft in 2007 and I assume that this work has not been pursued.

Right.

> Considering your proposal, I reckon you'll be face to the same
> questions in particular the one on the security issue.
> But I'm quite
> interesting in discussing this proposal on the mailing-list while
> even doing something more generic about transport protocol
> extensions for QoS networks.

Great.

Toerless
> 
> EL
> 
> 
> 
> >
> >BTW, yes, AR/VR means Augmented Reality/Virtual Reality. I'll add
> >the acronyms to the next version.
> >
> >Thanks,
> >Yingzhen
> >
> >
> >>    submitted, and we???d like to request a time slot to present it
> >>    @IETF101.
> >>
> >    --     Emmanuel LOCHIN

---

From: Yingzhen Qu <[email-masked]>
Date: Mon, 12 Mar 2018 23:43:28 -0700

Hi Michael,

Sorry for the late response.

Thanks for pointing out that we missed this important reference. You're
right, Quick-Start and our proposal do have lots of similarities, for
example both of them require that end-points and routers to work together.
But they are also different in details. For example, in our proposal
in-band signaling proposal bandwidth is reserved on routers along the path.

In next version of this draft, We'll add discussions about RFC 4728 and
6077.

BTW, can I request a slot to present this draft in TCPM if time allows?

Thanks,
Yingzhen

On Mon, Mar 5, 2018 at 12:09 AM, Scharf, Michael (Nokia - DE/Stuttgart) <
[email-masked]> wrote:

> I believe that this proposal is similar to QuickStart TCP (RFC 4782),
> which is not cited in draft-han-tsvwg-cc, and the reference is also missing
> in draft-han-6man-in-band-signaling-for-transport-qos.
>
>
>
> RFC 6077 explains some of the issues that an in-band signaling mechanism
> like Quick-Start has to solve. As far as I can tell, the fundamental
> challenge is neither the protocol specification nor a prototype
> implementation. For instance, it has been proven that QuickStart TCP can be
> implemented e.g. in network processors (see http://www.ikr.uni-stuttgart.
> de/Content/Publications/Archive/Sf_Diss_40112.pdf).
>
>
>
> So, when updating the documents, I suggest to add a discussion of how the
> open research issues explained in RFC 6077 are addressed.
>
>
>
> Michael
>
>
>
>
>
> *From:* Yingzhen Qu [mailto:[email-masked]]
> *Sent:* Sunday, March 04, 2018 9:59 PM
> *To:* [email-masked]
> *Cc:* Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>;
> Thomas Nadeau <[email-masked]>; [email-masked]; Yingzhen Qu <
> [email-masked]>; [email-masked]; [email-masked]
> *Subject:* Re: [tsvwg] Agenda requests for TSVWG@IETF101
>
>
>
> Hi Gorry and Michael,
>
>
>
> Thanks for the suggestion, I'll request a presentation in ICCRG.
> Meanwhile, I think since the in-band signaling draft was presented in
> TSVWG, if time allows it still makes sense to present this draft in TSVWG.
>
>
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 13 Mar 2018 09:38:54 +0000

Maybe I'm  saying what is obvious, but to be sure:

The way I see QS is:
- The router tracks the unutilised link capacity at the current time, 
and uses this as some estimate of what may be available to new flows in 
the near future.
- The router responds to QS requests and can accept them if there is 
"unutilised capacity" remaining.
- When accepted router updates it's count of the total unutilised 
capacity reducing by the accepted QS requests. (i.e., Subsequent QS 
requests in the same interval will currently have a smaller pool of 
unutilised capacity available).
- After the end of the interval all "QS requests" expire and the router 
again makes a measure of the unutilised capacity at the current time.

None of this implies a "reservation" of link resource in my thinking. In 
QS, the whole link capacity is intended to be shared using congestion 
control, and if  flows start to send more using normal CC within the 
interval where QS is operating, they would simply receive a larger share 
of the capacity. That is: QS request didn't "allocate" any link resource.

Gorry

On 13/03/2018, 06:58, Katsushi Kobayashi wrote:
> Hi,
>
> I think TCP QS router on the path also reserves bandwidth resource,
> if it accepts a request.
>
> RFC4782 says:
>    If the router approves the Quick-Start Request, this approval SHOULD
>    be taken into account in the router's decision to accept or reject
>    subsequent Quick-Start Requests (e.g., using a variable that tracks
>    the recent aggregate of accepted Quick-Start Requests).
>
>
> ---
> Katsushi Kobayashi
>
>> 2018/03/13 午後3:43、Yingzhen Qu <[email-masked] 
>> <mailto:[email-masked]>> のメール:
>>
>> Hi Michael,
>> Thanks for pointing out that we missed this important reference. 
>> You're right, Quick-Start and our proposal do have lots of 
>> similarities, for example both of them require that end-points and 
>> routers to work together. But they are also different in details. For 
>> example, in our proposal in-band signaling proposal bandwidth is 
>> reserved on routers along the path.
>>
>> In next version of this draft, We'll add discussions about RFC 4728 
>> and 6077.
>>
>> On Mon, Mar 5, 2018 at 12:09 AM, Scharf, Michael (Nokia - 
>> DE/Stuttgart) <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>     I believe that this proposal is similar to QuickStart TCP (RFC
>>     4782), which is not cited in draft-han-tsvwg-cc, and the
>>     reference is also missing in
>>     draft-han-6man-in-band-signaling-for-transport-qos.
>>
>>     RFC 6077 explains some of the issues that an in-band signaling
>>     mechanism like Quick-Start has to solve. As far as I can tell,
>>     the fundamental challenge is neither the protocol specification
>>     nor a prototype implementation. For instance, it has been proven
>>     that QuickStart TCP can be implemented e.g. in network processors
>>     (see
>>     http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf
>>     <http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf>).
>>
>>     So, when updating the documents, I suggest to add a discussion of
>>     how the open research issues explained in RFC 6077 are addressed.
>>
>>     Michael
>>
>>     *From:* Yingzhen Qu [mailto:[email-masked]
>>     <mailto:[email-masked]>]
>>     *To:* [email-masked] <mailto:[email-masked]>
>>     *Cc:* Scharf, Michael (Nokia - DE/Stuttgart)
>>     <[email-masked] <mailto:[email-masked]>>;
>>     Thomas Nadeau <[email-masked]
>>     <mailto:[email-masked]>>; [email-masked]
>>     <mailto:[email-masked]>; Yingzhen Qu <[email-masked]
>>     <mailto:[email-masked]>>; [email-masked]
>>     <mailto:[email-masked]>; [email-masked]
>>     <mailto:[email-masked]>
>>     Meanwhile, I think since the in-band signaling draft was
>>     presented in TSVWG, if time allows it still makes sense to
>>     present this draft in TSVWG.
>>
>>     The in-band signaling draft covers lots of aspects, and the
>>     required changes include network layer and transport layer. We're
>>     working on updating the draft, and may break it into pieces to
>>     fit different WGs.
>>
>>     On Sun, Mar 4, 2018 at 4:49 AM, Gorry Fairhurst
>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>         I am unsure yet what the correct group of people world be to
>>         explore a "Bandwidth Guaranteed Network". The presentation
>>         last IETF looked like the work could imply a need for changes
>>         proposed to the network layer (using OAM exchnages) to set
>>         the sending rate and make those bandwidth reservations.  In
>>         the end, it could result in a protocol quite different to
>>         TCP, I think this sort of change may possibly have a home in
>>         TSVWG  - but first I'd agree with Michaeland would encourage
>>         a presentation of the problem statement in ICCRG to explore
>>         the issues.
>>
>>         Gorry
>>
>>             >From the abstract: “…This draft proposes a new TCP
>>             congestion control algorithm used in bandwidth guaranteed
>>             networks.  It is an extension to the current TCP standards.”
>>
>>             In the IETF, I believe the expertise for this specific
>>             document would be in TCPM, which in CC. If the authors
>>             are interested in feedback on the proposed mechanism, I
>>             would recommend to ask TCPM.
>>
>>             Alternatively, corresponding research could perhaps be
>>             performed in the ICCRG. ICCRG has published RFC 6077 to
>>             document some of the open research issues in this space.
>>
>>             Michael
>>
>>             *From:*tsvwg [mailto:[email-masked]
>>             <mailto:[email-masked]>] *On Behalf Of *Yingzhen Qu
>>             *To:* [email-masked] <mailto:[email-masked]>;
>>             [email-masked] <mailto:[email-masked]>
>>             A new draft (The draft was suggested by TSVWG @IETF100)
>>             was just submitted, and we’d like to request a time slot
>>             to present it @IETF101.
>>
>>             Draft:
>>             https://datatracker.ietf.org/doc/draft-han-tsvwg-cc/

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 13 Mar 2018 10:45:09 +0100

> On 13. Mar 2018, at 07:43, Yingzhen Qu <[email-masked]> wrote:
> 
> Hi Michael,
>  
> Thanks for pointing out that we missed this important reference. You're right, Quick-Start and our proposal do have lots of similarities, for example both of them require that end-points and routers to work together. But they are also different in details. For example, in our proposal in-band signaling proposal bandwidth is reserved on routers along the path.
> 
> In next version of this draft, We'll add discussions about RFC 4728 and 6077.
> 
How much time would you need?

Best regards
Michael
> 
> Thanks,
> Yingzhen
> 
> On Mon, Mar 5, 2018 at 12:09 AM, Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]> wrote:
> On Sun, Mar 4, 2018 at 4:49 AM, Gorry Fairhurst <[email-masked]> wrote:
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Katsushi Kobayashi <[email-masked]>
Date: Tue, 13 Mar 2018 19:22:21 +0900

Gorry, 

Thank you for clarification.
I completely agree with you in terns of bandwidth resource reservation.

---
Katsushi Kobayashi

> 2018/03/13 午後6:38、Gorry Fairhurst <[email-masked]>のメール:
> 
> - The router tracks the unutilised link capacity at the current time, and uses this as some estimate of what may be available to new flows in the near future.
> - The router responds to QS requests and can accept them if there is "unutilised capacity" remaining.
> - When accepted router updates it's count of the total unutilised capacity reducing by the accepted QS requests. (i.e., Subsequent QS requests in the same interval will currently have a smaller pool of unutilised capacity available).
> - After the end of the interval all "QS requests" expire and the router again makes a measure of the unutilised capacity at the current time.
> 
> None of this implies a "reservation" of link resource in my thinking. In QS, the whole link capacity is intended to be shared using congestion control, and if  flows start to send more using normal CC within the interval where QS is operating, they would simply receive a larger share of the capacity. That is: QS request didn't "allocate" any link resource.
> 
> Gorry
> 
> 
>>> 2018/03/13 午後3:43、Yingzhen Qu <[email-masked] <mailto:[email-masked]>> のメール:
>>> 
>>> Hi Michael,
>>> On Mon, Mar 5, 2018 at 12:09 AM, Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked] <mailto:[email-masked]>> wrote:
>>>

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 13 Mar 2018 17:49:28 +0100

Thanks, Gory, nice explanation

TCP Quickstart is rfc4782 btw. i think there was a buggy number
in the thread.

TCP quickstart really relates IMHO primarily to
draft-han-6man-in-band-signaling-for-transport-qos, but not to 
draft-han-tsvwg-cc. The latter one is really meant to modify TCP
assuming a known guaranteed CIR - whatever mechanism is used
to provide that guarantee.

TCP quickstart is an interesting example for inband signaling,
which is what Lin's in-band-signaling draft does too. The main
difference is that our draft focusses on high-value traffic where
per-flow state is feasible and beneficial, if not necessary. And
TCP quickstart seems more targeted to ANY TCP flow. Which raises
a complete different scalability challenge to TCP quickstart.

This type of comparison discussion will go into draft updates on
our side.

Would be interested in any more data points about the history
of TCP quickstart, eg: where it was observed in the wild in deployments.

Cheers
   Toerless

On Tue, Mar 13, 2018 at 09:38:54AM +0000, Gorry Fairhurst wrote:
> - The router tracks the unutilised link capacity at the current
> time, and uses this as some estimate of what may be available to new
> flows in the near future.
> - After the end of the interval all "QS requests" expire and the
> router again makes a measure of the unutilised capacity at the
> current time.
> 
> None of this implies a "reservation" of link resource in my
> thinking. In QS, the whole link capacity is intended to be shared
> using congestion control, and if  flows start to send more using
> normal CC within the interval where QS is operating, they would
> simply receive a larger share of the capacity. That is: QS request
> didn't "allocate" any link resource.
> 
> Gorry
> 
> 
> >>2018/03/13 ??????3:43???Yingzhen Qu <[email-masked]
> >><mailto:[email-masked]>> ????????????:
> >>
> >>Hi Michael,
> >>similarities, for example both of them require that end-points
> >>and routers to work together. But they are also different in
> >>details. For example, in our proposal in-band signaling proposal
> >>bandwidth is reserved on routers along the path.
> >>
> >>            >From the abstract: ??????This draft proposes a new TCP
> >>            networks.  It is an extension to the current TCP standards.???
> >>
> >>            was just submitted, and we???d like to request a time slot
> >>            to present it @IETF101.
> >>

-- 
---
[email-masked]

---

From: Lin Han <[email-masked]>
Date: Tue, 13 Mar 2018 19:27:18 +0000

Hi, Gorry/Toerless

Here is some explanation and clarification for two drafts (draft-han-tsvwg-cc and draft-han-6man-in-band-signaling-for-transport-qos)

1. About draft-han-tsvwg-cc compared with other CC for bandwidth guaranteed network, such as gTFRC.
draft-han-tsvwg-cc is a split draft from draft-han-6man-in-band-signaling-for-transport-qos . It proposes a new CC for a network if the TCP session's bandwidth is guaranteed. Here the guaranteed  bandwidth is per TCP session and described by PIR/CIR and given directly by TCP end user. 

2. About TCP-QS and draft-han-6man-in-band-signaling-for-transport-qos
>From my understanding, TCP-QS and draft-han-6man-in-band-signaling-for-transport-qos do have similarity, both are kind of using in-band signaling for the resource reservation. But they still have big difference.
1). draft-han-6man-in-band-signaling-for-transport-qos is more general in-band signaling method, it not only applies to TCP, but also UDP or other upper layer of protocol, even the draft only exemplifies the method to TCP.
2). draft-han-6man-in-band-signaling-for-transport-qos provides reserved resource for bandwidth and latency for the whole life of TCP session. It gives more flavors and more granularity for the QoS (unlike TCP-QS only has 16 values of starting bandwidth). Moreover, it can support the "worst case latency" at each hop; also, it can support the OAM to detect the congestion states, and support the "forwarding state" for host to be notified in real-time about if the resource reservation or QoS forwarding is interrupted by abnormal situation such as link/node failures.

Best regards

Lin

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Toerless Eckert
Sent: Tuesday, March 13, 2018 9:49 AM
To: Gorry Fairhurst <[email-masked]>
Cc: Thomas Nadeau <[email-masked]>; [email-masked]; [email-masked]; Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>; [email-masked]; Katsushi Kobayashi <[email-masked]>; Yingzhen Qu <[email-masked]>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

TCP Quickstart is rfc4782 btw. i think there was a buggy number in the thread.

TCP quickstart really relates IMHO primarily to draft-han-6man-in-band-signaling-for-transport-qos, but not to draft-han-tsvwg-cc. The latter one is really meant to modify TCP assuming a known guaranteed CIR - whatever mechanism is used to provide that guarantee.

TCP quickstart is an interesting example for inband signaling, which is what Lin's in-band-signaling draft does too. The main difference is that our draft focusses on high-value traffic where per-flow state is feasible and beneficial, if not necessary. And TCP quickstart seems more targeted to ANY TCP flow. Which raises a complete different scalability challenge to TCP quickstart.

This type of comparison discussion will go into draft updates on our side.

Would be interested in any more data points about the history of TCP quickstart, eg: where it was observed in the wild in deployments.

Cheers
   Toerless

On Tue, Mar 13, 2018 at 09:38:54AM +0000, Gorry Fairhurst wrote:
> and uses this as some estimate of what may be available to new flows 
> in the near future.
> router again makes a measure of the unutilised capacity at the current 
> time.
> 
> None of this implies a "reservation" of link resource in my thinking. 
> In QS, the whole link capacity is intended to be shared using 
> congestion control, and if  flows start to send more using normal CC 
> within the interval where QS is operating, they would simply receive a 
> larger share of the capacity. That is: QS request didn't "allocate" 
> any link resource.
> 
> Gorry
> 
> 
> >>routers to work together. But they are also different in details. 
> >>For example, in our proposal in-band signaling proposal bandwidth is 

--
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 13 Mar 2018 22:17:39 +0100

Thanks, Lin

Let me also try another angle of comparison: 

Having worked on RSVP for a long time, your inband signaling
draft is IMHO very easy to vet - lightweight inband signaling,
replacing/simplifying/scaling the Intserv QoS model model for
deterministic services.

I think inband signaling could really be the antidote for what
i would call the IETFs intserv "second system syndrom" trauma from NSIS
and many ever more complex RSVP extensions never deployed.

Compared to the work on inband signaling for DiffServ, i think we did
not really have good attempts to do it for intserv. And ultimately,
DiffServ inband signaling has been quite successful, and we also
understand the dynamics needed to deploy it in various type of networks.
last but not least ECN in the Internet.

In fact, i think we are seeing with DiffServ a lot more difficult
experiments, struggling with the complexities of control loops.
TCP-QS is a great example: How do you even try to optimize in the
real world a control loop with N-nodes on the path using their own
"likely-best-useable-free-bandwidth" prediction based on a
local algorithm ? I also did not see in the TCP-QS docs references
to the impact of TCP behavior, (bursts), but its clear that the
result will be different with most-bursty (classic TCP CC), DV,
or even rate predictive (BBR etc.). How long are we working on
bufferload reduction in our 'aqm' groups ?

Give intserv a chance ;-)

Cheers
   toerless

-- 
---
[email-masked]

---

From: Lin Han <[email-masked]>
Date: Tue, 13 Mar 2018 21:52:34 +0000

Right, Toerless. You gave a very good comparison. 
If we must use a simplest sentence to describe our work, it would be "Intserv Renaissance by in-band signaling"

Regards

Lin

-----Original Message-----
From: Toerless Eckert [mailto:[email-masked]] 
Sent: Tuesday, March 13, 2018 2:18 PM
To: Lin Han <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>; Thomas Nadeau <[email-masked]>; [email-masked]; [email-masked]; Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>; [email-masked]; Katsushi Kobayashi <[email-masked]>; Yingzhen Qu <[email-masked]>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Thanks, Lin

Having worked on RSVP for a long time, your inband signaling draft is IMHO very easy to vet - lightweight inband signaling, replacing/simplifying/scaling the Intserv QoS model model for deterministic services.

I think inband signaling could really be the antidote for what i would call the IETFs intserv "second system syndrom" trauma from NSIS and many ever more complex RSVP extensions never deployed.

Compared to the work on inband signaling for DiffServ, i think we did not really have good attempts to do it for intserv. And ultimately, DiffServ inband signaling has been quite successful, and we also understand the dynamics needed to deploy it in various type of networks.
last but not least ECN in the Internet.

In fact, i think we are seeing with DiffServ a lot more difficult experiments, struggling with the complexities of control loops.
TCP-QS is a great example: How do you even try to optimize in the real world a control loop with N-nodes on the path using their own "likely-best-useable-free-bandwidth" prediction based on a local algorithm ? I also did not see in the TCP-QS docs references to the impact of TCP behavior, (bursts), but its clear that the result will be different with most-bursty (classic TCP CC), DV, or even rate predictive (BBR etc.). How long are we working on bufferload reduction in our 'aqm' groups ?

Cheers
   toerless

On Tue, Mar 13, 2018 at 07:27:18PM +0000, Lin Han wrote:
> Hi, Gorry/Toerless
> 
> Here is some explanation and clarification for two drafts 
> (draft-han-tsvwg-cc and 
> draft-han-6man-in-band-signaling-for-transport-qos)
> 
> Cc: Thomas Nadeau <[email-masked]>; [email-masked]; 
> [email-masked]; Scharf, Michael (Nokia - DE/Stuttgart) 
> <[email-masked]>; [email-masked]; Katsushi Kobayashi 
> <[email-masked]>; Yingzhen Qu <[email-masked]>
> > within the interval where QS is operating, they would simply receive 
> > a larger share of the capacity. That is: QS request didn't "allocate"
> > any link resource.
> > 
> > Gorry
> > 
> > 
> > >>For example, in our proposal in-band signaling proposal bandwidth 
> > >>is reserved on routers along the path.
> > >>

--
---
[email-masked]

---

From: Pat Thaler <[email-masked]>
Date: Wed, 14 Mar 2018 09:41:55 -0700

There work underway in detnet WG in the routing area for " point-to-point
(unicast) and point-to-multipoint (multicast) flows which can be
characterized in a manner that allows the network to 1) reserve the
appropriate resources for the flows in advance, and 2) release/reuse the
resources when they are no longer required."  The purpose is to provide for
bounded latency, loss and delay variation along with high reliability. The
current charter is focused on defining the data plane aspects. I hope that
that will be followed by work to define the control plane.

I've only had a quick glance at the draft. It isn't clear to me that it
would be compatible with a network providing reservations for guaranteed
bandwidth because it allows for sending at a rate higher than the bandwidth
guarantee; (i.e. a peak rate higher than the committed rate). If flows are
allowed to consume more than their reserved resources, they may interfere
with the resources committed to other flows.

Pat

---

From: Yingzhen Qu <[email-masked]>
Date: Wed, 14 Mar 2018 19:03:15 +0000

Hi Pat,

Thanks for the comments.

Exactly what the QoS parameters to be used is TBD. In the draft, we defined a minimum set such as PIR/CIR/BS/latency, so hardware programming may use one or more parameters depending on the feature requirement. For example, for bandwidth sensitive application, customer can give PIR/CIR (even you can have PIR=CIR, it means device only guarantees the forwarding of the traffic less than CIR); for latency sensitive application, we can have CIR and/or bounded latency for the traffic like CBR in TSN.

Thanks,
Yingzhen

From: Pat Thaler <[email-masked]>
Date: Wednesday, March 14, 2018 at 9:42 AM
To: Gorry Fairhurst <[email-masked]>
Cc: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>, Thomas Nadeau <[email-masked]>, "[email-masked]" <[email-masked]>, Yingzhen Qu <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, DetNet Chairs <[email-masked]>
Subject: Re: [tcpm] Agenda requests for TSVWG@IETF101

There work underway in detnet WG in the routing area for " point-to-point (unicast) and point-to-multipoint (multicast) flows which can be characterized in a manner that allows the network to 1) reserve the appropriate resources for the flows in advance, and 2) release/reuse the resources when they are no longer required."  The purpose is to provide for bounded latency, loss and delay variation along with high reliability. The current charter is focused on defining the data plane aspects. I hope that that will be followed by work to define the control plane.

I've only had a quick glance at the draft. It isn't clear to me that it would be compatible with a network providing reservations for guaranteed bandwidth because it allows for sending at a rate higher than the bandwidth guarantee; (i.e. a peak rate higher than the committed rate). If flows are allowed to consume more than their reserved resources, they may interfere with the resources committed to other flows.

Pat

Gorry

Hi all,

Michael

Dear Chairs,

Thanks,

Yingzhen

_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 15 Mar 2018 08:38:42 +1300

Pat,

> If flows are allowed to consume more than their reserved resources, they may interfere with the resources committed to other flows.

No. They will simply use resources that are otherwise available for the best effort (or lower effort) class. Clearly, the total of all CIRs must be less than the total capacity, on any given path.

Bounded latency and jitter are definitely harder problems. I think it might be wise to wait until DETNET has made some progress before fixing the signalling parameters. There's no point in signalling a request that is physically impossible to satisfy. So Yingzhen is correct to say "TBD" below.

Regards
   Brian Carpenter

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 15 Mar 2018 00:32:11 +0100

Thanks, Pat - inline.

On Wed, Mar 14, 2018 at 09:41:55AM -0700, Pat Thaler wrote:
> current charter is focused on defining the data plane aspects.

Yes, well aware of that. We will also propose he inband signalling fr
that context. The draft already mentions in passing UDP and is really
not difficult to adopt to any transport. TCP was just the most widely
accepted protocol even with more challenges than UDP IMHO, thats why
our inband signaling draft focusses on it. 

> I hope that that will be followed by work to define the control plane.

+1.

> with the resources committed to other flows.

The enforcement of not intruding on reserved capacity is a matter of
the forwarding plane. In the most extreme case which we describe in
our inband singaling, you use per-flow reservation state (shaper/scheduler).

This can be optimized(cost reduced) by only doing per-flow state on the
edge and only admission control (calculate/signal inband) hop-by-hop, but
not establish per-flow packet processing state (shaper/scheduler).

This is all exactly like IMHO it would need to work in DetNet (e.g.:
forwarding plane. Signaling is obviously variable... inband
signaling is just the best IMHO ;-)).

The interesting part of course is that your overall latency and jitter
will go down the drain once you exceed your per-flow shaper/scheduler
CIR + burst size and start to compete based on congestion experience.

For the not really 'realtim' apps like he typical streaming with
guaranteed CIR requiremens but desire for upspeeding, this is a perfect
compromise to strike. For the typical realtime app coming from a CBR
world, thats a surprise that might want them to stick to just CBR = CIR = PIR
requests.

The more advanced video solutions btw would use the AF style forwarding
approach, whereby your <= CIR rate is marked with a hghier priority
AF code point and the CIR..PIR rate with a lower AF code point. And then
you put the more important video frames into your higher AF code point/
<= CIR rate, and the less important frames into the lower priority and
you get even better upspeeding experience (visually). This btw.
is something requiring UDP or SCTP or the like. TCP apps have no
control over message/packet boundaries to mark them accordingly.

Wrt. DetNet: I wam not quite sure about the PHBs defined in TSN.
I sometime have the fear that its a lot more limited than in IP.
I hope we manage to leverage the full extend of IP QoS in DetNet
and not only some L2 TSN QoS subset...

Toerless

> >> From the abstract: ??????This draft proposes a new TCP congestion control
> >> current TCP standards.???
> >>
> >> submitted, and we???d like to request a time slot to present it @IETF101.
> >>

-- 
---
[email-masked]

---

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Thu, 15 Mar 2018 23:42:08 +0000

I am responding since the e-mail was directed to me...

> TCPM might be actually a good idea, and that WG's feedback would be highly

See my sepatate e-mail.

> Let me send a separate email to the chairs and ask if they could consider
> presenting this in TCPM.

Please feel free to answer my questions on the TCPM list or in the presentation.  The list of questions is obviously not comprehensive.
 
> > From the abstract: ?????? This draft proposes a new TCP congestion control
> The way i would describe it: This memo defines simple extensions to TCP-
> Reno CC to support and leverage a pre-known network guaranteed
> (minimum) bandwidth (CIR) and pre-known maximumum needed
> bandwidth (PIR).
> 

I disagree. The work has to specify exactly what requirements need to be fulfilled to make the mechanism applicable, and these requirements would have to be explicitly listed in the document. 

For instance, it seems to be assumed that CIR and PIR is correctly known to the sender. So the document would e.g. have to discuss the implications if the sender has erroneous CIR or PIR information. 

How the requirements would indeed met would be irrelevant, but of course in the IETF context meeting all the requirements would have to be guaranteed for the Internet.

> the draft mentions some IMHO cool new ideas how to do this, but
> this CC draft by itself would be a great benefit to application developers
> today (assume CIR/PIR API extensions to transport service -> TAPS, etc. pp).

I don't see a lot of discussion in the document on how the application developers will get a benefit from this, but I assume this can be added.

> in home (STB) where i could select guaranted minimum qualiy (translated by
> app into CIR), and (optional) maximum quality (PIR). On the parents living
> room STB CIR would be set according to 4k, and maybe in one other room to
> FullHD, and the rest has to start compete for available bandwidth all the way
> from *yuck* QCIF).

Is video streaming the target application of what we discuss here?

> problem to support easy policy differentiation for different apps will
> increase, but i always like to look first at the most common use case that
> could profit today.

I may be wrong but I believe for AR/VR apps the application traffic pattern can depend on the application design, no?

So will AR/VR be impossible over the current Internet with the tools we have already?

> stacks to do something like this, and then folks will see a lot easier the need
> to have better in-network mechanisms to actually guarantee CIR.

Me, as well as some of my students, have implemented mechanisms like this in the Linux TCP stack 10 years ago. None of us had kernel programming experience. It wasn't super-complex to change the TCP stack to make this happen.

Given that is simple to implement the logic in the host stacks, what has been missing so far according to this chicken&egg logic?

And I assume that you are not suggesting to deploy TCP stacks that send CBR traffic into the current Internet without reacting to congestion signals, right?
 
> ICCRG. ICCRG has published RFC 6077 to document some of the open
> related to this CIR/PIR knowledge issue, admission control is mentioned only
> once in passing. Let me know what you think i should specifically look into,
> and we'll be happy to relate to that RFC6077 work in the draft.

Section 3.1 in RFC 6077 applies also to CIR and PIR signaling.

Thanks

Michael

---

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Fri, 16 Mar 2018 00:06:47 +0000

If somebody is interested in exploring this, I suggest to have a look at http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf (Section 4.5, and a lot of the rest of the document). This document discusses how to process Quick-Start in routers. I never found the time to publish some of the content elsewhere.

In a nutshell, there can be different ways how a router could process Quick-Start requests with tradeoffs between performance, fairness, and risk of congestion.

However, the fundamental problem is more how applications can indeed make a reasonable request for bandwidth. Quick-Start works relatively well if it is enabled for these applications that indeed leverage it (e.g., mid-sized request-response traffic patterns), and if the requested rates are reasonable.

The Quick-Start mechanism basically fails completely e.g. if Quick-Start requests are also included in short TCP data transfers that complete in one or two RTTs. In that case, if the routers keep track of the request they approve, a lot of capacity will be kept for flows that will actually never use the rate they have requested, at the cost of other TCP connections that may be able to use it. There is also a fairness issue then. This issue can be solved by a more optimistic approval scheme that assumes that not all requests will indeed use the full capacity. But then one risks congestion as one over-commits capacity. Due to all these issues, it is non-trivial to find realistic scenarios in which Quick-Start indeed outperforms end-to-end congestion control e.g. with CUBIC and IW 10...

More below...

> - The router tracks the unutilised link capacity at the current time, and uses
> this as some estimate of what may be available to new flows in the near
> future.

Even knowing the unutilized link capacity is a HUGE problem of its own. Even for Ethernet router ports, if there are VLAN-tagged ports, there is no such thing like a single unutilized link capacity... The Ethernet port is shared between different routing interfaces. And let's not look at wireless...

> - When accepted router updates it's count of the total unutilised capacity
> reducing by the accepted QS requests. (i.e., Subsequent QS requests in the
> same interval will currently have a smaller pool of unutilised capacity
> available).

Indeed, that is the overall principle, but within that concept there are a number of different designs possible.

> - After the end of the interval all "QS requests" expire and the router again
> makes a measure of the unutilised capacity at the current time.

That concept of an interval also has issues. In general, the mechanism will work if the interval is of the same order of magnitude like the RTT of most connections using Quick-Start. If the RTT distribution is very heterogeneous, it is non-trivial how to let QS requests expire. One might need to create per-QS request state in the router, which has huge scalability issues in the fast path. 

> None of this implies a "reservation" of link resource in my thinking. In QS, the
> whole link capacity is intended to be shared using congestion control, and if
> flows start to send more using normal CC within the interval where QS is
> operating, they would simply receive a larger share of the capacity. That is:
> QS request didn't "allocate" any link resource.

Indeed, but some potential algorithms in a QS router could come close to "reservation" schemes. And then one runs basically into the same issues like IntServ.

Michael

> > I think TCP QS router on the path also reserves bandwidth resource, if
> > it accepts a request.
> >
> > RFC4782 says:
> >>     http://www.ikr.uni-
> stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf
> >>     <http://www.ikr.uni-
> stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf>).
> >>

---

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Fri, 16 Mar 2018 00:34:26 +0000

> TCP quickstart really relates IMHO primarily to draft-han-6man-in-band-
> signaling-for-transport-qos, but not to draft-han-tsvwg-cc. The latter one is

This is wrong. Section 4.4 in RFC 4782 is very related to draft-han-tsvwg-cc. Of specific interest is e.g. the handling of ssthresh. RFC 4782 also considers packet headers.

> really meant to modify TCP assuming a known guaranteed CIR - whatever
> mechanism is used to provide that guarantee.
> 
> TCP quickstart is an interesting example for inband signaling, which is what
> Lin's in-band-signaling draft does too. The main difference is that our draft
> focusses on high-value traffic where per-flow state is feasible and beneficial,
> if not necessary. And TCP quickstart seems more targeted to ANY TCP flow.

No. Quick-Start only has benefits if it is enabled by the applications that can indeed leverage it, which is a subset of all TCP flows. If a host naively applies it to any TCP connection, there will be no benefit as most Quick-Start requests will be rejected. So the endpoint has a strong incentive to enable it only on these connections that actually leverage it. Actually doing this is a problem of its own. I have discussed this issue with AR/VR developers 10 years ago and it was non-trivial by then. It would be interesting to learn what has changed since then in the application developer community.

> Which raises a complete different scalability challenge to TCP quickstart.

Yep. Quick-Start can be implemented in a router without per-flow state and thus scales e.g. on network processors. 10 years ago we found that the key performance bottleneck in the fast path would be the state-synchronization between the different cores of a network processor. But as Quick-Start does not perform hard guarantees, this can be worked around.

I am not a hardware expert, any maybe state synchronization between many cores is cheaper these days. But I'd really like to understand how hard QoS guarantees for single TCP connections would be achieved e.g. on modern multi-core network processor (i.e., multiple TBit/s). 
 
> Would be interested in any more data points about the history of TCP
> quickstart, eg: where it was observed in the wild in deployments.

In my experiments 10 years ago, there was little performance benefit of Quick-Start as compared to IW10, which was published in RFC 6928. My experiments are published in http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf.

I strongly suggest to compare new congestion control schemes tot CUBIC+IW10 as baseline, and to show how much performance benefit one indeed gets, and at what risk and costs.

Michael

---

From: Lin Han <[email-masked]>
Date: Fri, 16 Mar 2018 00:59:47 +0000

Hi, Scarf

For draft-han-tsvwg-cc, we have some assumptions

1. User application setup TCP session with a given PIR/CIR,

2. Network devices on the path will satisfy such expectation. In details, the traffic with the rate below CIR is always guaranteed to pass, and above PIR will be dropped; If the rate is between PIR and CIR, the traffic may be competing with others to get the resource.

This draft try to propose what we should change for the current CC for above scenarios, the picture below may explain more clearly for what is the difference of new algorithm with reno:

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
Sent: Thursday, March 15, 2018 5:34 PM
To: Toerless Eckert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Thomas Nadeau <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; Katsushi Kobayashi <[email-masked]>; Yingzhen Qu <[email-masked]>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

> draft-han-6man-in-band- signaling-for-transport-qos, but not to

> draft-han-tsvwg-cc. The latter one is

> TCP quickstart is an interesting example for inband signaling, which

> is what Lin's in-band-signaling draft does too. The main difference is

> that our draft focusses on high-value traffic where per-flow state is

> feasible and beneficial, if not necessary. And TCP quickstart seems more targeted to ANY TCP flow.

> Which raises a complete different scalability challenge to TCP quickstart..

> This type of comparison discussion will go into draft updates on our side..

Michael

---

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Fri, 16 Mar 2018 01:07:06 +0000

As I wrote already, I believe there will be scenarios where the TCP Reno congestion control will complete data transfers significantly faster than what you propose, i.e., your proposal results in worse performance.

I suggest to draw a picture with PIR>>CIR and a larger RTT. Then you will see that there will be cases in which TCP Reno outperforms what you suggest. Which raises the question why an application should setup a TCP session that will be slower than the default.

Michael

From: Lin Han [mailto:[email-masked]]
Sent: Friday, March 16, 2018 2:00 AM
To: Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>; Toerless Eckert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Thomas Nadeau <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; Katsushi Kobayashi <[email-masked]>; Yingzhen Qu <[email-masked]>
Subject: RE: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Hi, Scarf

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
Sent: Thursday, March 15, 2018 5:34 PM
To: Toerless Eckert <[email-masked]<mailto:[email-masked]>>; Gorry Fairhurst <[email-masked]<mailto:[email-masked]>>
Cc: Thomas Nadeau <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Katsushi Kobayashi <[email-masked]<mailto:[email-masked]>>; Yingzhen Qu <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Michael

---

From: Lin Han <[email-masked]>
Date: Fri, 16 Mar 2018 02:01:31 +0000

Hi, Michael

As described in the original draft draft-han-6man-in-band-signaling-for-transport-qos, the new method is not designed for the normal application that current TCP works well. It does not try to replace or compete with TCP for most of current applications. It is designed for the scenario that the regular TCP cannot satisfy the requirement in terms of bandwidth or latency. I.e, ultra-high bandwidth or ultra-short latency.

For example, there is no networked AR/VR (with good quality) now since no network or transport technology can satisfy its unusual requirement (see detailed analysis in draft-han-iccrg-arvr-transport-problem-00). In order to provide such service, SP may need to dramatically increase network, link capacity and even topology. Currently, a user may be only able to get such service by a leased line, but it is too expensive for most of people.

When an application selects the PIR and CIR, it must know the consequence for the value. As you said, if PIR>>CIR, it means it's a very burst traffic, and using new method may not make sense since user should know for the rate above CIR, the behavior will be almost same as regular reno, and above PIR is worse than reno since it will be dropped.
How to smartly select PIR/CIR is another topic and irrelevant to the CC algorithm.

Regards

Lin

From: Scharf, Michael (Nokia - DE/Stuttgart) [mailto:[email-masked]]
Sent: Thursday, March 15, 2018 6:07 PM
To: Lin Han <[email-masked]>; Toerless Eckert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Thomas Nadeau <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; Katsushi Kobayashi <[email-masked]>; Yingzhen Qu <[email-masked]>
Subject: RE: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Michael

To: Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]<mailto:[email-masked]>>; Toerless Eckert <[email-masked]<mailto:[email-masked]>>; Gorry Fairhurst <[email-masked]<mailto:[email-masked]>>

Hi, Scarf

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
Sent: Thursday, March 15, 2018 5:34 PM
To: Toerless Eckert <[email-masked]<mailto:[email-masked]>>; Gorry Fairhurst <[email-masked]<mailto:[email-masked]>>
Cc: Thomas Nadeau <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Katsushi Kobayashi <[email-masked]<mailto:[email-masked]>>; Yingzhen Qu <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Michael

---

From: Tom Herbert <[email-masked]>
Date: Thu, 15 Mar 2018 22:50:46 -0700

On Thu, Mar 15, 2018 at 7:01 PM, Lin Han <[email-masked]> wrote:

> Hi, Michael
>
>
>
> As described in the original draft draft-han-6man-in-band-signaling-for-transport-qos,
> the new method is not designed for the normal application that current TCP
> works well.. It does not try to replace or compete with TCP for most of
> current applications. It is designed for the scenario that the regular TCP
> cannot satisfy the requirement in terms of bandwidth or latency. I.e,
> ultra-high bandwidth or ultra-short latency.
>
>
>
Hi Lin,

The draft does seem to be TCP centric. You might want to look at FAST
(draft-herbert-fast-00). It's similar in that it uses extension headers for
network signaling of things such as QoS, but is designed to be completely
transport protocol agnostic. In fact one of the goals of FAST is to not
have the network perform DPI to extract 5-tuples any more. With the use of
non-TCP protocols or increased use of encryption (tunnel mode IPsec) the
need or ability for the network to do DPI hopefully decreases (IMO a good
thing since DPI tends to be a source of protocol ossification). We still
want the same functionality though which is what FAST tries to provide.

Tom

> For example, there is no networked AR/VR (with good quality) now since no
> network or transport technology can satisfy its unusual requirement (see
> detailed analysis in draft-han-iccrg-arvr-transport-problem-00). In order
> to provide such service, SP may need to dramatically increase network, link
> capacity and even topology. Currently, a user may be only able to get such
> service by a leased line, but it is too expensive for most of people.
>
>
>
> When an application selects the PIR and CIR, it must know the consequence
> for the value. As you said, if PIR>>CIR, it means it’s a very burst
> traffic, and using new method may not make sense since user should know for
> the rate above CIR, the behavior will be almost same as regular reno, and
> above PIR is worse than reno since it will be dropped.
>
> *From:* Scharf, Michael (Nokia - DE/Stuttgart) [mailto:
> [email-masked]]
> *Sent:* Thursday, March 15, 2018 6:07 PM
> *To:* Lin Han <[email-masked]>; Toerless Eckert <[email-masked]>;
> Gorry Fairhurst <[email-masked]>
> *Cc:* Thomas Nadeau <[email-masked]>; [email-masked];
> [email-masked]; [email-masked]; Katsushi Kobayashi <[email-masked]>;
> Yingzhen Qu <[email-masked]>
> *Subject:* RE: [tsvwg] inband signaling (was: Re: Agenda requests for
> TSVWG@IETF101)
>
>
>
> As I wrote already, I believe there will be scenarios where the TCP Reno
> congestion control will complete data transfers significantly faster than
> what you propose, i.e., your proposal results in worse performance.
>
>
>
> I suggest to draw a picture with PIR>>CIR and a larger RTT. Then you will
> see that there will be cases in which TCP Reno outperforms what you
> suggest. Which raises the question why an application should setup a TCP
> session that will be slower than the default.
>
>
>
> Michael
>
>
>
>
>
>
>
> *From:* Lin Han [mailto:[email-masked] <[email-masked]>]
> *Sent:* Friday, March 16, 2018 2:00 AM
> *To:* Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>;
> Toerless Eckert <[email-masked]>; Gorry Fairhurst <[email-masked]>
> *Cc:* Thomas Nadeau <[email-masked]>; [email-masked];
> [email-masked]; [email-masked]; Katsushi Kobayashi <[email-masked]>;
> Yingzhen Qu <[email-masked]>
> *Subject:* RE: [tsvwg] inband signaling (was: Re: Agenda requests for
> TSVWG@IETF101)
>
>
>
> Hi, Scarf
>
>
>
> 2. Network devices on the path will satisfy such expectation. In details,
> the traffic with the rate below CIR is always guaranteed to pass, and above
> PIR will be dropped; If the rate is between PIR and CIR, the traffic may be
> competing with others to get the resource.
>
> This draft try to propose what we should change for the current CC for
> above scenarios, the picture below may explain more clearly for what is the
> difference of new algorithm with reno:
>
>
>
>
>
>
> From: tsvwg [mailto:[email-masked] <[email-masked]>] On
> Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
> To: Toerless Eckert <[email-masked]>; Gorry Fairhurst <[email-masked]
> >
> Cc: Thomas Nadeau <[email-masked]>; [email-masked]; [email-masked];
> [email-masked]; Katsushi Kobayashi <[email-masked]>; Yingzhen Qu <
> [email-masked]>
> This is wrong. Section 4.4 in RFC 4782 is very related to
> draft-han-tsvwg-cc. Of specific interest is e.g. the handling of ssthresh.
> RFC 4782 also considers packet headers.
>
>
>
> No. Quick-Start only has benefits if it is enabled by the applications
> that can indeed leverage it, which is a subset of all TCP flows. If a host
> naively applies it to any TCP connection, there will be no benefit as most
> Quick-Start requests will be rejected. So the endpoint has a strong
> incentive to enable it only on these connections that actually leverage it.
> Actually doing this is a problem of its own. I have discussed this issue
> with AR/VR developers 10 years ago and it was non-trivial by then. It would
> be interesting to learn what has changed since then in the application
> developer community.
>
>
>
> Yep. Quick-Start can be implemented in a router without per-flow state and
> thus scales e.g. on network processors. 10 years ago we found that the key
> performance bottleneck in the fast path would be the state-synchronization
> between the different cores of a network processor. But as Quick-Start does
> not perform hard guarantees, this can be worked around.
>
>
>
> I am not a hardware expert, any maybe state synchronization between many
> cores is cheaper these days. But I'd really like to understand how hard QoS
> guarantees for single TCP connections would be achieved e.g. on modern
> multi-core network processor (i.e., multiple TBit/s).
>
>
>
> In my experiments 10 years ago, there was little performance benefit of
> Quick-Start as compared to IW10, which was published in RFC 6928. My
> experiments are published in http://www.ikr.uni-stuttgart.
> de/Content/Publications/Archive/Sf_Diss_40112.pdf.
>
>
>
> I strongly suggest to compare new congestion control schemes tot
> CUBIC+IW10 as baseline, and to show how much performance benefit one indeed
> gets, and at what risk and costs.
>
>
>
> Michael
>
>
>
>
>

---

From: "Scharf, Michael (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Fri, 16 Mar 2018 08:24:44 +0000

Hi Lin,

Thanks a lot. I read from your explanation that a leased line would solve the performance problem. Leased lines are offered by service providers already. Have you compared the performance of your proposed TCP modification over an existing leased line service of a service provider? And have you compared the result with a modern TCP stack with CUBIC, IW10, and the like running over the same service, without having to guess a PIR/CIR? For which cases would your scheme provide a better service to the application compared to what TCP can do today?

I suggest to study real measurements to understand how much headroom for improvement indeed exists, as compared to what TCP congestion control already can do today without any new technology.

For instance, a single TCP connection scales up to multiple 10 Gbps, i.e., as far as I know it basically can reach the limit what a single computer can process with current processors. TCP works very well over short latencies since more than 20 years.

Thanks

Michael

From: tsvwg [mailto:[email-masked]] On Behalf Of Lin Han
Sent: Friday, March 16, 2018 3:02 AM
To: Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>; Toerless Eckert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Thomas Nadeau <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; Katsushi Kobayashi <[email-masked]>; Yingzhen Qu <[email-masked]>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Hi, Michael

As described in the original draft draft-han-6man-in-band-signaling-for-transport-qos, the new method is not designed for the normal application that current TCP works well.. It does not try to replace or compete with TCP for most of current applications. It is designed for the scenario that the regular TCP cannot satisfy the requirement in terms of bandwidth or latency. I.e, ultra-high bandwidth or ultra-short latency.

Regards

Lin

To: Lin Han <[email-masked]<mailto:[email-masked]>>; Toerless Eckert <[email-masked]<mailto:[email-masked]>>; Gorry Fairhurst <[email-masked]<mailto:[email-masked]>>

Michael

Hi, Scarf

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
Sent: Thursday, March 15, 2018 5:34 PM
To: Toerless Eckert <[email-masked]<mailto:[email-masked]>>; Gorry Fairhurst <[email-masked]<mailto:[email-masked]>>
Cc: Thomas Nadeau <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Katsushi Kobayashi <[email-masked]<mailto:[email-masked]>>; Yingzhen Qu <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Michael

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Mar 2018 17:44:50 +0100

Michael:

IMHO, the goal is to establish the model of mixed inelastic and elastic CIR..PIR
traffic elasticity as an option for TCP. The problem is to define it in such a way that
it is implementable, and as you say there there are now many different CC schemes
for TCP. We choose Reno as a well known reference in the hope that others could
then pick up and adopt to more modern, more complex CC algorithms, whereas
the newer ones have likely a lot more intricacies and fewer folks who would know how
to adopt the goals in details to those algorithms (i may be wrong on this one).

Maybe using a rather older refeence CC scheme like reno to define an easily
implementable version of the CIR..PIR goal is not the right approach,
but the description shuold be more abstract ? Or just use as Emmanual did
rely on TFRC as a reference ? Given how i think ( i may be wrong) that TFRC is
not really applied to TCP, i thought another approch would have to be used.

Advice very much welcome.

The principles intended to be established are not really rocket science:

 - Constrain yourself to a max of PIR
 - Operate under expectation to have CIR guaranteed
   (expectation to be provided from other components)
   - fast-start to CIR, reduce only to CIR on bad congestion feedback.
   - When operating at/below CIR:
     - distinguish congestion from random loss
       - repair random loss. Keep rate at CIR.
       - break or revert to CIR=0 on congestion loss
       - optional (shared queue DSCP/PHB option)
         defend CIR against congestion from up to a point where its
	 likely to be an CIR admission failure and not bullying contenders
 - Operate unchanged in all other cases (eg: between CIR..PIR).

Cheers
    Toerless

On Fri, Mar 16, 2018 at 08:24:44AM +0000, Scharf, Michael (Nokia - DE/Stuttgart) wrote:
> As described in the original draft draft-han-6man-in-band-signaling-for-transport-qos, the new method is not designed for the normal application that current TCP works well.. It does not try to replace or compete with TCP for most of current applications. It is designed for the scenario that the regular TCP cannot satisfy the requirement in terms of bandwidth or latency. I..e, ultra-high bandwidth or ultra-short latency.
> 
> To: Lin Han <[email-masked]<mailto:[email-masked]>>; Toerless Eckert <[email-masked]<mailto:[email-masked]>>; Gorry Fairhurst <[email-masked]..uk<mailto:[email-masked]>>
> > Which raises a complete different scalability challenge to TCP quickstart...
> 
> 
> 
> > This type of comparison discussion will go into draft updates on our side...
> 
> >
> 

-- 
---
[email-masked]

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 16 Mar 2018 17:54:30 +0000

I'm watching this thread, in all its variations across mailing lists, with
great interest, but not participating, except to say ...

> Michael:

deleted down to

> not really applied to TCP, i thought another approch would have to be used.

I wouldn't use TFRC as a reference unless you can find people who are
actively using it as if it was a good thing.

I was still on the IAB when RMCAT was chartered, but I was also on the IAB
program committee for https://www.iab.org/activities/workshops/cc-workshop/,
which was the immediate predecessor activity for RMCAT, and IIRC, we
couldn't find anyone who actually used TFRC for serious work. It was
apparently being used as a fig leaf because prior to TFRC we had no advice
for non-TCP rate control at all. So, at least people could say "do
something, and TFRC is something", in their drafts without being flushed
out of the IESG.

IMO, of course.

Spencer, speaking NOT as an AD, in this case ...

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Mar 2018 20:20:13 +0100

Thanks, Spencer!

Is there any newer formalistic reference for "TCP friendly CC" or "Internet fair CC"
that one could reference ? Or are we down to the battle ground of coming
up with a random CC algorithm and showing through testing that it will
be within +- 10x speed of some "internet CC mix - Reno, QBR? BBR, nada, ..." ?

If so, is there any defined "Internet CC mix" to do testing with ?

how else... ;-))

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Lin Han <[email-masked]>
Date: Sun, 18 Mar 2018 12:58:50 +0000

Hi, Tom

Sorry for the late reply, and thanks for your reference for your draft.

I read your draft. If I am correct, I believe we both share some basic idea that using data packet to signal either network device or peer host for some network service. The method described in draft-han-6man-in-band-signaling-for-transport-qos is an example for TCP service, and can also apply to UDP or others. Actually we are in the testing the same method to other application as well. I have also submitted a presentation for hotRFC for the further study of this topic in IETF.

Not sure if you attend IETF101. If possible, we can discuss more details.

Regards

Lin

From: Tom Herbert [mailto:[email-masked]]
Sent: Thursday, March 15, 2018 10:51 PM
To: Lin Han <[email-masked]>
Cc: Scharf, Michael (Nokia - DE/Stuttgart) <[email-masked]>; Toerless Eckert <[email-masked]>; Gorry Fairhurst <[email-masked]>; Thomas Nadeau <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; Katsushi Kobayashi <[email-masked]>; Yingzhen Qu <[email-masked]>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

On Thu, Mar 15, 2018 at 7:01 PM, Lin Han <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Michael

Hi Lin,

The draft does seem to be TCP centric. You might want to look at FAST (draft-herbert-fast-00). It's similar in that it uses extension headers for network signaling of things such as QoS, but is designed to be completely transport protocol agnostic. In fact one of the goals of FAST is to not have the network perform DPI to extract 5-tuples any more. With the use of non-TCP protocols or increased use of encryption (tunnel mode IPsec) the need or ability for the network to do DPI hopefully decreases (IMO a good thing since DPI tends to be a source of protocol ossification). We still want the same functionality though which is what FAST tries to provide.

Tom

When an application selects the PIR and CIR, it must know the consequence for the value. As you said, if PIR>>CIR, it means it’s a very burst traffic, and using new method may not make sense since user should know for the rate above CIR, the behavior will be almost same as regular reno, and above PIR is worse than reno since it will be dropped.
How to smartly select PIR/CIR is another topic and irrelevant to the CC algorithm.

Regards

Lin

From: Scharf, Michael (Nokia - DE/Stuttgart) [mailto:[email-masked]<mailto:[email-masked]>]

Michael

Hi, Scarf

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
Sent: Thursday, March 15, 2018 5:34 PM
To: Toerless Eckert <[email-masked]<mailto:[email-masked]>>; Gorry Fairhurst <[email-masked]<mailto:[email-masked]>>
Cc: Thomas Nadeau <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Katsushi Kobayashi <[email-masked]<mailto:[email-masked]>>; Yingzhen Qu <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Michael

---

From: Emmanuel Lochin <[email-masked]>
Date: Mon, 19 Mar 2018 10:20:24 +0100

Dear Lin,

Looking at your scheme below, I reckon the choice to limit the cwnd 
progression at PIR must be optional as this could be inadequate 
following the policy used. For instance : if you used a two-rate 
three-color marker to policy your traffic (basically a dual-rate dual 
token bucket algorithm to assess my traffic against two rate limits) : 
when conforms to the CIR, the TRTCM marks green, then between CIR and 
PIR yellow and above the PIR red : authorizing excess traffic if there 
is room in the network. So you should have the possibility to both 
optionally discard packets or mark it : and in the second case, you 
should not limit the cwnd, otherwise this CC won't take benefit of the 
full capacity.

EL

Le 16/03/2018 à 01:59, Lin Han a écrit :
> 2. Network devices on the path will satisfy such expectation. In 
> details, the traffic with the rate below CIR is always guaranteed to 
> pass, and above PIR will be dropped; If the rate is between PIR and 
> CIR, the traffic may be competing with others to get the resource.
>
> above scenarios, the picture below may explain more clearly for what 
> is the difference of new algorithm with reno:
>
> [email-masked]; [email-masked]; Katsushi Kobayashi 
> draft-han-tsvwg-cc. Of specific interest is e.g. the handling of 
> ssthresh. RFC 4782 also considers packet headers.
>
> that can indeed leverage it, which is a subset of all TCP flows. If a 
> host naively applies it to any TCP connection, there will be no 
> benefit as most Quick-Start requests will be rejected. So the endpoint 
> has a strong incentive to enable it only on these connections that 
> actually leverage it. Actually doing this is a problem of its own. I 
> have discussed this issue with AR/VR developers 10 years ago and it 
> was non-trivial by then. It would be interesting to learn what has 
> changed since then in the application developer community.
>
> Yep. Quick-Start can be implemented in a router without per-flow state 
> and thus scales e.g. on network processors. 10 years ago we found that 
> the key performance bottleneck in the fast path would be the 
> state-synchronization between the different cores of a network 
> processor. But as Quick-Start does not perform hard guarantees, this 
> can be worked around.
>
> I am not a hardware expert, any maybe state synchronization between 
> many cores is cheaper these days. But I'd really like to understand 
> how hard QoS guarantees for single TCP connections would be achieved 
> e.g. on modern multi-core network processor (i.e., multiple TBit/s).
>
> In my experiments 10 years ago, there was little performance benefit 
> of Quick-Start as compared to IW10, which was published in RFC 6928. 
> My experiments are published in 
> http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/Sf_Diss_40112.pdf.
>
> CUBIC+IW10 as baseline, and to show how much performance benefit one 
> indeed gets, and at what risk and costs.
>
> Michael
>

---

From: Lin Han <[email-masked]>
Date: Mon, 19 Mar 2018 11:03:06 +0000

Hi, Emmanuel

Thanks for your comments.

I guess everyone agrees the color of the packets for different range of rate of traffic : GREEN (rate<CIR); YELLOW (CIR<rate<PIR); and RED (rate>PIR).

Right now, our argument is the policy for traffic management at the hardware.
Our current TM policy is that

1.      Transmit GREEN, all time even when the egress buffer exceeds the pre-configured threshold

2.      Transmit YELLOW if the egress buffer is less than the threshold, discard YELLOW if buffer depth exceeds the threshold

3.      Discard the RED at the ingress, we drop it as early as possible to reduce the system burden in switch fabric.

Since CURRENTLY the RED packet will be dropped at the ingress, to prevent the re-transmit of RED packet and reduce the associated latency, we will do rate limit to the host to below the PIR in our CC algorithm. This can reduce the buffer bloat at egress, and reduce possible congestion and thus reduce all NEW TCP flow's worst case latency.

If we allow RED at the ingress, what should be behavior at egress, and how to differentiate it with YELLOW? Unless we introduce two thresholds, one for YELLOW and one for RED. This may have more complexity at hardware. In the implementation, buffer depth exceeding the threshold is also an indication of possible congestion, we have to report this as warning to host by OAM. BTW, in our implementation, all NEW TCP  share same egress buffer.

Literally PIR already means the maximum rate allowed, do we want to give more freedom (may still pass) to user? Actually, If user think the exceeding PIR cause drop may impact its throughput, he can easily increase PIR to get higher throughput.

I totally understand to allow RED will have higher link utilization for the flow, but the utilization is not the only target the design has to consider. Actually, we think the link utilization is the target of traditional TCP, the buffer bloat and latency should have higher priority in our design for NEW TCP.

I want to make it clear, our current CC algorithm is to adapt the current TM policy. We can easily change the TM design and CC algorithm. I want to hear what community thinks in this regard.

Regards

Lin

From: Emmanuel Lochin [mailto:[email-masked]]
Sent: Monday, March 19, 2018 2:20 AM
To: [email-masked]; Lin Han <[email-masked]>
Subject: Re: [tsvwg] inband signaling

Dear Lin,

Looking at your scheme below, I reckon the choice to limit the cwnd progression at PIR must be optional as this could be inadequate following the policy used. For instance : if you used a two-rate three-color marker to policy your traffic (basically a dual-rate dual token bucket algorithm to assess my traffic against two rate limits) : when conforms to the CIR, the TRTCM marks green, then between CIR and PIR yellow and above the PIR red : authorizing excess traffic if there is room in the network. So you should have the possibility to both optionally discard packets or mark it : and in the second case, you should not limit the cwnd, otherwise this CC won't take benefit of the full capacity.

EL

Hi, Scarf

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
Sent: Thursday, March 15, 2018 5:34 PM
To: Toerless Eckert <[email-masked]><mailto:[email-masked]>; Gorry Fairhurst <[email-masked]><mailto:[email-masked]>
Cc: Thomas Nadeau <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Katsushi Kobayashi <[email-masked]><mailto:[email-masked]>; Yingzhen Qu <[email-masked]><mailto:[email-masked]>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Michael

--

Emmanuel LOCHIN

Professeur ISAE

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 19 Mar 2018 12:19:46 +0100

Lin, Emmanuel:

This is all great deal, but shouldn't we try to up level this ?

In one instance we should just claim that the inband signaling
can easily support a variety of existing PHB/DSCP options we have
standardized. Such as the AF options with its resulting differences
in marking. It looks to me as if the discus you have is about the
AF model, but if i am wrong, maybe just circle back to which
existing marking model we're referring.

And of course, it would be worthwhile to document which parameters
of the inband signaling would be needed to support which currently
defined PHB/DSCP option.

If we are talking about some additional, new & better PHB/marking
strategy we see as useful, it would look more logical to me to
introduce describe this independent of an inband signaling option,
but of course, then also describe which parameters of the
inband signaling are required to support it.

[ Just wanting to avoid the impression as if the inband signaling
  is kinda trying to reinvent the IETF QoS universe, when in effekt
  its really about a more lightweight,faster,scalable and simplified
  alternative to RSVP signaling possible through the evolution
  of NPU technologies. ]

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Black, David" <[email-masked]>
Date: Mon, 19 Mar 2018 18:32:38 +0000

Inline ... with <WG chair hat OFF>

Thanks, --David

From: tsvwg [mailto:[email-masked]] On Behalf Of Lin Han
Sent: Monday, March 19, 2018 7:03 AM
To: Emmanuel Lochin <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] inband signaling

Hi, Emmanuel

Since CURRENTLY the RED packet will be dropped at the ingress, to prevent the re-transmit of RED packet and reduce the associated latency, we will do rate limit to the host to below the PIR in our CC algorithm. This can reduce the buffer bloat at egress, and reduce possible congestion and thus reduce all NEW TCP flow's worst case latency.
[David>] I hope "prevent the re-transmit" was a poor choice of words.  TCP will retransmit that packet (segment), hopefully at a point where it won't be colored RED.

If we allow RED at the ingress, what should be behavior at egress, and how to differentiate it with YELLOW? Unless we introduce two thresholds, one for YELLOW and one for RED. This may have more complexity at hardware. In the implementation, buffer depth exceeding the threshold is also an indication of possible congestion, we have to report this as warning to host by OAM. BTW, in our implementation, all NEW TCP  share same egress buffer.
[David>] This is about the number of levels of drop precedence - see RFC 2597 (Diffserv AF) for further discussion.  Whether to use multiple thresholds is an implementation design decision.  At the time that the Diffserv work (RFC 2597 and friends) was done, there was no consensus on the number of thresholds - e.g., whether to use 2 vs. 3 drop precedence levels in the network.  I would guess that this lack of consensus is still the current situation.

Regards

Lin

From: Emmanuel Lochin [mailto:[email-masked]]
Sent: Monday, March 19, 2018 2:20 AM
To: [email-masked]<mailto:[email-masked]>; Lin Han <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] inband signaling

Dear Lin,

EL

Hi, Scarf

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
Sent: Thursday, March 15, 2018 5:34 PM
To: Toerless Eckert <[email-masked]><mailto:[email-masked]>; Gorry Fairhurst <[email-masked]><mailto:[email-masked]>
Cc: Thomas Nadeau <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Katsushi Kobayashi <[email-masked]><mailto:[email-masked]>; Yingzhen Qu <[email-masked]><mailto:[email-masked]>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Michael

--

Emmanuel LOCHIN

Professeur ISAE

---

From: Emmanuel Lochin <[email-masked]>
Date: Tue, 20 Mar 2018 10:37:48 +0100

Fine for me and thanks for your answers, I just wanted to understand the 
rationale of this design choice. Well, putting aside**MaxBandwidthWND, I 
have another question with MinBandwidthWND.

Looking at the end of section 4.4 from your draft, ssthresh is always 
set to MinBandwidthWND (basically you set it to the CIR). However, 
depending on the values of CIR and PIR, after a congestion event, you 
might get a value of cwnd/2 higher than MinBandwidthWND. So I believe 
MinBandwidthWND should be equal to max(MinBandwidthWND, halving(cwnd)); 
to reach better TCP performance.

EL

---

From: Lin Han <[email-masked]>
Date: Tue, 20 Mar 2018 11:08:06 +0000

Hi, David

See the inline reply with [LH]

Thanks

Lin

From: Black, David [mailto:[email-masked]]
Sent: Monday, March 19, 2018 11:33 AM
To: Lin Han <[email-masked]>; Emmanuel Lochin <[email-masked]>; [email-masked]
Cc: Black, David <[email-masked]>
Subject: RE: [tsvwg] inband signaling

Thanks, --David

From: tsvwg [mailto:[email-masked]] On Behalf Of Lin Han
Sent: Monday, March 19, 2018 7:03 AM
To: Emmanuel Lochin <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [tsvwg] inband signaling

Hi, Emmanuel

[LH] The retransmit of the packet will be in GREEN since the rate will be reduced to CIR, but retransmission will have more delay.
I guess the current controversial point is should we let host rate stay below PIR?  I think this depends on how we define and implement the "guaranteed bandwidth" for a flow.

1.      If we only define CIR, then we can let the rate fly above CIR until the congestion, this is the same as current CC, no change is needed.

2.      If we define PIR and CIR, we only have argument for RED packet process, and I think this is related to how QoS enforcement is realized. If HW drop RED packet, I believe CC should not let rate exceeding PIR; If HW doesn't drop RED but still depends on the remained bandwidth or only drop after a threshold is exceeded, I don't see the necessary of setting PIR for user, since the behavior is same as the case that the rate is above CIR. And introduce another parameter does not change traffic management too much.

[LH] agreed

Regards

Lin

Dear Lin,

EL

Hi, Scarf

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Scharf, Michael (Nokia - DE/Stuttgart)
Sent: Thursday, March 15, 2018 5:34 PM
To: Toerless Eckert <[email-masked]><mailto:[email-masked]>; Gorry Fairhurst <[email-masked]><mailto:[email-masked]>
Cc: Thomas Nadeau <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Katsushi Kobayashi <[email-masked]><mailto:[email-masked]>; Yingzhen Qu <[email-masked]><mailto:[email-masked]>
Subject: Re: [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Michael

--

Emmanuel LOCHIN

Professeur ISAE

---

From: Lin Han <[email-masked]>
Date: Tue, 20 Mar 2018 11:11:57 +0000

Hi, Emmanuel

Yes, this behavior can be changed easily, we just reduce the rate to CIR for more conservation and simpler implementation.

Thanks

Lin

From: Emmanuel Lochin [mailto:[email-masked]]
Sent: Tuesday, March 20, 2018 2:38 AM
To: Toerless Eckert <[email-masked]>; Lin Han <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] inband signaling

Fine for me and thanks for your answers, I just wanted to understand the rationale of this design choice. Well, putting aside MaxBandwidthWND, I have another question with MinBandwidthWND.

Looking at the end of section 4.4 from your draft, ssthresh is always set to MinBandwidthWND (basically you set it to the CIR). However, depending on the values of CIR and PIR, after a congestion event, you might get a value of cwnd/2 higher than MinBandwidthWND. So I believe MinBandwidthWND should be equal to max(MinBandwidthWND, halving(cwnd)); to reach better TCP performance.

EL

Lin, Emmanuel:

defined PHB/DSCP option.

  of NPU technologies. ]

Cheers

    Toerless

Hi, Emmanuel

Regards

Lin

To: [email-masked]<mailto:[email-masked]>; Lin Han <[email-masked]><mailto:[email-masked]>

Dear Lin,

EL

Hi, Scarf

-----Original Message-----

To: Toerless Eckert <[email-masked]><mailto:[email-masked]><mailto:[email-masked]><mailto:[email-masked]>; Gorry Fairhurst <[email-masked]><mailto:[email-masked]><mailto:[email-masked]><mailto:[email-masked]>

Cc: Thomas Nadeau <[email-masked]><mailto:[email-masked]><mailto:[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]><mailto:[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]><mailto:[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]><mailto:[email-masked]><mailto:[email-masked]>; Katsushi Kobayashi <[email-masked]><mailto:[email-masked]><mailto:[email-masked]><mailto:[email-masked]>; Yingzhen Qu <[email-masked]><mailto:[email-masked]><mailto:[email-masked]><mailto:[email-masked]>

Michael

--

Emmanuel LOCHIN

Professeur ISAE

--

Emmanuel LOCHIN

Professeur ISAE