Subject: Re: [tsvwg] [tcpm] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 14 Mar 2018 07:55:59 +0000

Hi

This subject pops up every once in a while. 

I would like to add my perspective, given how I understand that 3GPP technology works. The 3GPP QoS framework can be used to offer for instance a Guaranteed bandwidth (GBR) to applications. The operation is that the "bearer" gets increased scheduling priority once the throughput is below the GBR value.
In simulations I have verified that this framework can deliver a GBR to TCP flows when the network load increases, at least as long it is possible to push back other bearers with a lower priority. So, seen from my experience so far I am not fully confident that additional inband signaling is needed. 

There can however be a benefit to e.g. determine an appropriate bitrate for a DASH session, depending on network load and/or coverage, but I would argue that out of band signaling is better in this case.

As regards to "worst case latency", I am not sure that I follow here, are we discussing TSN/Detnet technology here, general link failures, or is it about means to control the queue delay in the network ?. As to the latter, I believe that it is worthwhile to look more closely at L4S. 

Regards
Ingemar Johansson

> Message: 2
> Date: Tue, 13 Mar 2018 19:27:18 +0000
> From: Lin Han <[email-masked]>
> To: Toerless Eckert <[email-masked]>, Gorry Fairhurst
> 	<[email-masked]>
> Cc: Thomas Nadeau <[email-masked]>, "[email-masked]"
> 	<[email-masked]>,  "[email-masked]" <[email-masked]>, "Scharf,
> Michael
> 	(Nokia - DE/Stuttgart)" <[email-masked]>,
> 	"[email-masked]" <[email-masked]>, Katsushi Kobayashi
> 	<[email-masked]>, Yingzhen Qu <[email-masked]>
> Subject: Re: [tcpm] [tsvwg] inband signaling (was: Re: Agenda requests
> 	for TSVWG@IETF101)
> Message-ID:
> 	<1D30AF33624CDD4A99E8C395069A2A162CDB9BA3@sjceml521-
> mbs.china.huawei.com>
> 
> Content-Type: text/plain; charset="us-ascii"
> 
> Hi, Gorry/Toerless
> 
> Here is some explanation and clarification for two drafts (draft-han-tsvwg-cc
> and draft-han-6man-in-band-signaling-for-transport-qos)
> 
> 1. About draft-han-tsvwg-cc compared with other CC for bandwidth
> guaranteed network, such as gTFRC.
> draft-han-tsvwg-cc is a split draft from draft-han-6man-in-band-signaling-for-
> transport-qos . It proposes a new CC for a network if the TCP session's
> bandwidth is guaranteed. Here the guaranteed  bandwidth is per TCP session
> and described by PIR/CIR and given directly by TCP end user.
> 
> 2. About TCP-QS and draft-han-6man-in-band-signaling-for-transport-qos
> >From my understanding, TCP-QS and draft-han-6man-in-band-signaling-for-
> transport-qos do have similarity, both are kind of using in-band signaling for
> the resource reservation. But they still have big difference.
> 1). draft-han-6man-in-band-signaling-for-transport-qos is more general in-
> band signaling method, it not only applies to TCP, but also UDP or other
> upper layer of protocol, even the draft only exemplifies the method to TCP.
> 2). draft-han-6man-in-band-signaling-for-transport-qos provides reserved
> resource for bandwidth and latency for the whole life of TCP session. It gives
> more flavors and more granularity for the QoS (unlike TCP-QS only has 16
> values of starting bandwidth). Moreover, it can support the "worst case
> latency" at each hop; also, it can support the OAM to detect the congestion
> states, and support the "forwarding state" for host to be notified in real-time
> about if the resource reservation or QoS forwarding is interrupted by
> abnormal situation such as link/node failures.
> 
> Best regards
> 
> Lin
>

---

From: "Smith, Kevin, (R&D) Vodafone Group" <[email-masked]>
Date: Wed, 14 Mar 2018 09:31:15 +0000

HI Ingemar,

> The 3GPP QoS framework can be used to offer for instance a Guaranteed bandwidth (GBR) to applications. The operation is that the "bearer" gets increased scheduling priority once the throughput is below the GBR value.

That is technically correct: but a consideration to note here is that in LTE, operators typically operate only two bearers - one for IMS/VoLTE, which is GBR; and one 'best effort' (non-GBR) bearer for Internet traffic.  This is likely due to non-technical reasons.

Hence I also favour a bearer*-agnostic approach such as L4S, but appreciate that L4S will also have to operate within the constraints of the radio scheduler (which will still be 'best effort')

For 5G, certainly there can be more flexible and holistic scheduling approaches. For practical implementation though, all 'prioritisations' will need to meet regulatory directives on traffic management.

* i.e. GBR vs non-GBR bearer

All best,
Kevin

Vodafone Proprietary classified as C2 - Internal

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Ingemar Johansson S
Sent: 14 March 2018 07:56
To: [email-masked]; [email-masked]
Cc: Ingemar Johansson S <[email-masked]>; [email-masked]; [email-masked]; [email-masked]' <[email-masked]>
Subject: Re: [tsvwg] [tcpm] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Hi

Regards
Ingemar Johansson

> <[email-masked]>,  "[email-masked]" <[email-masked]>, "Scharf, Michael
> Here is some explanation and clarification for two drafts
> (draft-han-tsvwg-cc and
> draft-han-6man-in-band-signaling-for-transport-qos)
>
> transport-qos . It proposes a new CC for a network if the TCP
> session's bandwidth is guaranteed. Here the guaranteed  bandwidth is
> per TCP session and described by PIR/CIR and given directly by TCP end user.
>
> transport-qos do have similarity, both are kind of using in-band
> signaling for the resource reservation. But they still have big difference.
> 1). draft-han-6man-in-band-signaling-for-transport-qos is more general
> in- band signaling method, it not only applies to TCP, but also UDP or
> other upper layer of protocol, even the draft only exemplifies the method to TCP.
> 2). draft-han-6man-in-band-signaling-for-transport-qos provides
> reserved resource for bandwidth and latency for the whole life of TCP
> session. It gives more flavors and more granularity for the QoS
> (unlike TCP-QS only has 16 values of starting bandwidth). Moreover, it
> can support the "worst case latency" at each hop; also, it can support
> the OAM to detect the congestion states, and support the "forwarding
> state" for host to be notified in real-time about if the resource
> reservation or QoS forwarding is interrupted by abnormal situation such as link/node failures.
>
> Best regards
>
> Lin
>

---

From: Lin Han <[email-masked]>
Date: Wed, 14 Mar 2018 21:21:49 +0000

Hi, Ingemar

Tanks for the comments, See inline answer

Best Regards

Lin

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Wednesday, March 14, 2018 12:56 AM
To: Lin Han <[email-masked]>; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]' <[email-masked]>; Ingemar Johansson S <[email-masked]>
Subject: Re: [tcpm] [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Hi

This subject pops up every once in a while. 
[LH] You are right -:) But this time, we bring up this issue for the future applications like AR/VR, not for applications that current TCP works well. See the detailed analysis in draft-han-iccrg-arvr-transport-problem-00. We believe for the applications, such as AR/VR/TSN, that are much sensitive to congestion or delay, current technologies are hard to satisfy the requirement. 

I would like to add my perspective, given how I understand that 3GPP technology works. The 3GPP QoS framework can be used to offer for instance a Guaranteed bandwidth (GBR) to applications. The operation is that the "bearer" gets increased scheduling priority once the throughput is below the GBR value.
In simulations I have verified that this framework can deliver a GBR to TCP flows when the network load increases, at least as long it is possible to push back other bearers with a lower priority. So, seen from my experience so far I am not fully confident that additional inband signaling is needed. 
[LH] Does it work well when the GW or eNodeB are congested? The purpose for in-band signaling is to guarantee network resource for a critical application no matter what happens. The congestion is the major problem for such requirement by current technology, because all flows can only share the bandwidth during congestion.

There can however be a benefit to e.g. determine an appropriate bitrate for a DASH session, depending on network load and/or coverage, but I would argue that out of band signaling is better in this case.
[LH] One benefit of in-band signaling is it can provide the dynamic resource (like Q depth) to the source. By using this signal, we can distinguish congestion packet loss and random physical loss. It can increase the good-put for application as we described in the new draft.

As regards to "worst case latency", I am not sure that I follow here, are we discussing TSN/Detnet technology here, general link failures, or is it about means to control the queue delay in the network ?. As to the latter, I believe that it is worthwhile to look more closely at L4S. 
[LH] Yes, we want to control the delay for specified flow by change the queue/scheduler parameters. We believe only through such very fine control, we can achieve the expected service from application. We are in talk with detnet to see if they want an alternative solution other than PWE. L4S still only provides a statistical bounded latency when all hosts are upgraded to support L4S. For our solution, we don't need to change hosts if they don't want to use new TCP.

Regards
Ingemar Johansson

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 15 Mar 2018 08:23:35 +0000

Hi

Please see inline, marked [IJ]
/Ingemar

> From: Lin Han [mailto:[email-masked]]
> Sent: den 14 mars 2018 22:22
> To: Ingemar Johansson S <[email-masked]>;
> [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]' <[email-masked]>
> Subject: RE: [tcpm] [tsvwg] inband signaling (was: Re: Agenda requests for
> TSVWG@IETF101)
> 
> Hi, Ingemar
> 
> Cc: [email-masked]; [email-masked]; [email-masked]' <[email-masked]>; Ingemar
> Johansson S <[email-masked]>
> [LH] You are right -:) But this time, we bring up this issue for the future
> applications like AR/VR, not for applications that current TCP works well. See
> the detailed analysis in draft-han-iccrg-arvr-transport-problem-00. We
> believe for the applications, such as AR/VR/TSN, that are much sensitive to
> congestion or delay, current technologies are hard to satisfy the
> requirement.
[IJ] Please have a look at the RITE project experiments https://riteproject.eu/, these show that you can achieve very short latency with L4S technology, L4S can in many ways coexist with diffserv and this draft outlines how the two can work together https://tools.ietf.org/pdf/draft-briscoe-tsvwg-l4s-diffserv-00.pdf . Given this I am not convinced why an inband QoS solution is needed for AR/VR.

> 
> I would like to add my perspective, given how I understand that 3GPP
> technology works. The 3GPP QoS framework can be used to offer for
> instance a Guaranteed bandwidth (GBR) to applications. The operation is that
> the "bearer" gets increased scheduling priority once the throughput is below
> the GBR value.
> In simulations I have verified that this framework can deliver a GBR to TCP
> flows when the network load increases, at least as long it is possible to push
> back other bearers with a lower priority. So, seen from my experience so far I
> am not fully confident that additional inband signaling is needed.
> [LH] Does it work well when the GW or eNodeB are congested? The purpose
> for in-band signaling is to guarantee network resource for a critical application
> no matter what happens. The congestion is the major problem for such
> requirement by current technology, because all flows can only share the
> bandwidth during congestion.
[IJ] As for eNB, the allocation of GBR/MBR comes with admission control. However because of physical limitations in terms of limited power coverage and/or radio interference, no in-band or out of band QoS signaling in the world can guarantee a given throughput in all conditions. It is all matter of prioritization of the available resource blocks, where each resources worth in terms of bits is dictated by the laws of Shannon, somewhat on steroids with the use of adaptive antenna beamforming. 

> 
> There can however be a benefit to e.g. determine an appropriate bitrate for
> a DASH session, depending on network load and/or coverage, but I would
> argue that out of band signaling is better in this case.
> [LH] One benefit of in-band signaling is it can provide the dynamic resource
> (like Q depth) to the source. By using this signal, we can distinguish
> congestion packet loss and random physical loss. It can increase the good-put
> for application as we described in the new draft.
[IJ] But isn't it better to use good AQMs instead as the goal should always be to avoid large standing queues ?. 

> 
> As regards to "worst case latency", I am not sure that I follow here, are we
> discussing TSN/Detnet technology here, general link failures, or is it about
> means to control the queue delay in the network ?. As to the latter, I believe
> that it is worthwhile to look more closely at L4S.
> [LH] Yes, we want to control the delay for specified flow by change the
> queue/scheduler parameters. We believe only through such very fine
> control, we can achieve the expected service from application. We are in talk
> with detnet to see if they want an alternative solution other than PWE. L4S
> still only provides a statistical bounded latency when all hosts are upgraded
> to support L4S. For our solution, we don't need to change hosts if they don't
> want to use new TCP.
[IJ] The L4S issue is that of deployment but you will face similar deployment issues with inband QoS signaling, probably worse. L4S is basically very simple technology whereas inband QoS implies more packet inspection and processing. 
And I believe that TSN/Detnet is a different story as it tries to achieve extremely tight scheduling, so far in confined domains.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 16 Mar 2018 08:56:20 +1300

On 15/03/2018 21:23, Ingemar Johansson S wrote:

...
> [IJ] As for eNB, the allocation of GBR/MBR comes with admission control. However because of physical limitations in terms of limited power coverage and/or radio interference, no in-band or out of b…

I just wanted to applaud that. In a wireless network where random interference is always possible, physics tells us that you can never ignore Shannon's theorem, and you can never ignore queueing theory as a result.

    Brian

---

From: Lin Han <[email-masked]>
Date: Fri, 16 Mar 2018 00:31:46 +0000

Hi,

See inline with [LH]

Thanks

Lin

-----Original Message-----
From: Ingemar Johansson S [mailto:[email-masked]] 
Sent: Thursday, March 15, 2018 1:24 AM
To: Lin Han <[email-masked]>; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]' <[email-masked]>; Ingemar Johansson S <[email-masked]>
Subject: RE: [tcpm] [tsvwg] inband signaling (was: Re: Agenda requests for TSVWG@IETF101)

Hi

> Cc: [email-masked]; [email-masked]; [email-masked]' <[email-masked]>; 
> Ingemar Johansson S <[email-masked]>
> for
> [LH] You are right -:) But this time, we bring up this issue for the 
> future applications like AR/VR, not for applications that current TCP 
> works well. See the detailed analysis in 
> draft-han-iccrg-arvr-transport-problem-00. We believe for the 
> applications, such as AR/VR/TSN, that are much sensitive to congestion 
> or delay, current technologies are hard to satisfy the requirement.

[LH] Thanks for the reference, L4S can achieve the small latency for a closed network like DC where all hosts must behave same according to the requirement to adjust the rate when the queue is exceeding the threshold. If some host does not follow, i.e, ignore the ECN bit, I don't believe the latency can be that good because all L4S flows share the same queue.
Another aspect is that even using ECN, we still cannot provide an expected and differentiated bandwidth to different user. DiffServ can only provide differentiated service by PHB, but without the quantity control in terms of bandwidth or latency.

> instance a Guaranteed bandwidth (GBR) to applications. The operation 
> is that the "bearer" gets increased scheduling priority once the 
> throughput is below the GBR value.
> In simulations I have verified that this framework can deliver a GBR 
> to TCP flows when the network load increases, at least as long it is 
> possible to push back other bearers with a lower priority. So, seen 
> from my experience so far I am not fully confident that additional inband signaling is needed.
> [LH] Does it work well when the GW or eNodeB are congested? The 
> purpose for in-band signaling is to guarantee network resource for a 
> critical application no matter what happens. The congestion is the 
> major problem for such requirement by current technology, because all 
> flows can only share the bandwidth during congestion.

[LH] Yes, if the air interface has limit, no matter what signaling we use will not help. In-band signaling just provide a way to control the network device on the path to give a session expected resource. If the expected resource cannot be given, the in-band signaling will notify the source host about the failure. The user can either reduce the resource demanded or use the traditional TCP. If there is traffic engineering tool involved, the user may use different path that can satisfy the user's expectation for the TCP session setup, this is not conflicting with the basic frame work of in-band signaling. We can still use it with segment routing for such non-shortest path scenario.

> 
> There can however be a benefit to e.g. determine an appropriate 
> bitrate for a DASH session, depending on network load and/or coverage, 
> but I would argue that out of band signaling is better in this case.
> [LH] One benefit of in-band signaling is it can provide the dynamic 
> resource (like Q depth) to the source. By using this signal, we can 
> distinguish congestion packet loss and random physical loss. It can 
> increase the good-put for application as we described in the new draft.

[LH] Same as using L4S, AQM still try to provide the faired resource allocation to all TCP flows by the rule of max-min fairness. This works well if the link capacity is big enough and all TCP flows will get what its application want. But when the link is congested, we cannot give some critical application the expected bandwidth, each flow will get the equal instantaneous bandwidth determined the total number of flows. To overcome this difficulty caused by TCP fairness, we have to reserve the resource for some application and exclude them from the resource sharing with others if the resource is not enough (treat those flow differently as others, i.e, different queuing/scheduler, etc).

> 
> As regards to "worst case latency", I am not sure that I follow here, 
> are we discussing TSN/Detnet technology here, general link failures, 
> or is it about means to control the queue delay in the network ?. As 
> to the latter, I believe that it is worthwhile to look more closely at L4S.
> [LH] Yes, we want to control the delay for specified flow by change 
> the queue/scheduler parameters. We believe only through such very fine 
> control, we can achieve the expected service from application. We are 
> in talk with detnet to see if they want an alternative solution other 
> than PWE. L4S still only provides a statistical bounded latency when 
> all hosts are upgraded to support L4S. For our solution, we don't need 
> to change hosts if they don't want to use new TCP.

[LH] Yes, in-band signaling is more complicated technology. We are aware of this, and only report it as an informational or experiment draft.
In-band signaling does not need to program all network device. In real deployment, only those aggregation device (may experience congestion) such as GW, Edge router are required to be involved. Normally core routers link utilization is less than 60% or something like that.

> > per TCP session and described by PIR/CIR and given directly by TCP 
> > end
> user.
> >
> > 2. About TCP-QS and 
> > draft-han-6man-in-band-signaling-for-transport-qos
> > in- band signaling method, it not only applies to TCP, but also UDP 
> > or other upper layer of protocol, even the draft only exemplifies 
> > the method
> to TCP.
> > reserved resource for bandwidth and latency for the whole life of 
> > TCP session. It gives more flavors and more granularity for the QoS 
> > (unlike TCP-QS only has 16 values of starting bandwidth). Moreover, 
> > it can support the "worst case latency" at each hop; also, it can 
> > support the OAM to detect the congestion states, and support the 
> > "forwarding state" for host to be notified in real-time about if the 
> > resource reservation or QoS forwarding is interrupted by abnormal 
> > situation such as
> link/node failures.
> >
> > Best regards
> >
> > Lin
> >

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 16 Mar 2018 15:58:46 +1300

Lin,

On 16/03/2018 13:31, Lin Han wrote:
...
> [LH] Same as using L4S, AQM still try to provide the faired resource allocation to all TCP flows by the rule of max-min fairness. This works well if the link capacity is big enough and all TCP flow…

That's pretty much exactly the philosophy behind diffserv, with "flow"
defined as "all packets marked with the same diffserv code point".

The problem is scaling, and always has been. Diffserv sets a hard
scaling limit at 64 classes. With modern technology, what value between
64 and infinity is a reasonable limit? (The discussion on scalability
in the draft is rather brief.)

Regards
    Brian

---

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 17 Mar 2018 10:29:39 +0000

Hi

I have not been able to follow this discussion lately, I have however seen that Michael Scharf wrote up a long list of questions that should be addressed.
https://mailarchive.ietf.org/arch/msg/tcpm/_2lh_jLTuUbYNLX8lNzGE-DyBZw
I would suggest that these are addressed thoroughly.  

One final remark on your comment
> [LH] Thanks for the reference, L4S can achieve the small latency for a closed
> network like DC where all hosts must behave same according to the
> requirement to adjust the rate when the queue is exceeding the threshold.
> If some host does not follow, i.e, ignore the ECN bit, I don't believe the
> latency can be that good because all L4S flows share the same queue.
> Another aspect is that even using ECN, we still cannot provide an expected
> and differentiated bandwidth to different user. DiffServ can only provide
> differentiated service by PHB, but without the quantity control in terms of
> bandwidth or latency.
This argument "if ECN is ignored" has been raised many times. ConEx was proposed as a remedy to issues where endpoints do not comply to the rules that are set up, but it has not gained attraction so far. The counter argument should be, what is really gained with congestion controls that misbehave in one way or the other. In most of the work that I am active in you mainly spoil your own performance with excessive end to end delay or packet loss if the congestion control is badly configured.

BR
/Ingemar

> Sent: den 16 mars 2018 01:32
> [IJ] Please have a look at the RITE project experiments
> https://riteproject.eu/, these show that you can achieve very short latency
> with L4S technology, L4S can in many ways coexist with diffserv and this draft
> outlines how the two can work together https://tools.ietf.org/pdf/draft-
> briscoe-tsvwg-l4s-diffserv-00.pdf . Given this I am not convinced why an
> inband QoS solution is needed for AR/VR.
> 
> [LH] Thanks for the reference, L4S can achieve the small latency for a closed
> network like DC where all hosts must behave same according to the
> requirement to adjust the rate when the queue is exceeding the threshold.
> If some host does not follow, i.e, ignore the ECN bit, I don't believe the
> latency can be that good because all L4S flows share the same queue.
> Another aspect is that even using ECN, we still cannot provide an expected
> and differentiated bandwidth to different user. DiffServ can only provide
> differentiated service by PHB, but without the quantity control in terms of
> bandwidth or latency.
> 
> >
> [IJ] As for eNB, the allocation of GBR/MBR comes with admission control.
> However because of physical limitations in terms of limited power coverage
> and/or radio interference, no in-band or out of band QoS signaling in the
> world can guarantee a given throughput in all conditions. It is all matter of
> prioritization of the available resource blocks, where each resources worth in
> terms of bits is dictated by the laws of Shannon, somewhat on steroids with
> the use of adaptive antenna beamforming.
> 
> [LH] Yes, if the air interface has limit, no matter what signaling we use will not
> help. In-band signaling just provide a way to control the network device on
> the path to give a session expected resource. If the expected resource
> cannot be given, the in-band signaling will notify the source host about the
> failure. The user can either reduce the resource demanded or use the
> traditional TCP. If there is traffic engineering tool involved, the user may use
> different path that can satisfy the user's expectation for the TCP session
> setup, this is not conflicting with the basic frame work of in-band signaling.
> We can still use it with segment routing for such non-shortest path scenario.
> 
> >
> [LH] Same as using L4S, AQM still try to provide the faired resource allocation
> to all TCP flows by the rule of max-min fairness. This works well if the link
> capacity is big enough and all TCP flows will get what its application want. But
> when the link is congested, we cannot give some critical application the
> expected bandwidth, each flow will get the equal instantaneous bandwidth
> determined the total number of flows. To overcome this difficulty caused by
> TCP fairness, we have to reserve the resource for some application and
> exclude them from the resource sharing with others if the resource is not
> enough (treat those flow differently as others, i.e, different
> queuing/scheduler, etc).
> 
> >
> [IJ] The L4S issue is that of deployment but you will face similar deployment
> issues with inband QoS signaling, probably worse. L4S is basically very simple
> technology whereas inband QoS implies more packet inspection and
> processing.
> And I believe that TSN/Detnet is a different story as it tries to achieve
> extremely tight scheduling, so far in confined domains.
> 
> [LH] Yes, in-band signaling is more complicated technology. We are aware of
> this, and only report it as an informational or experiment draft.
> In-band signaling does not need to program all network device. In real
> deployment, only those aggregation device (may experience congestion)
> such as GW, Edge router are required to be involved. Normally core routers
> link utilization is less than 60% or something like that.
> 
> >
> > Regards
> > Ingemar Johansson
> >
> >
> > > Message: 2