Subject: Re: [tsvwg] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 4 Nov 2015 13:27:51 +0000

Hi

And thanks for the review, comments/answers/questions inline below

/Ingemar

From: Bob Briscoe [mailto:[email-masked]]
Sent: den 29 oktober 2015 18:54
To: Ingemar Johansson S; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

Ingemar,

As promised, here's my comments. They are a mix of suggested improvements to the draft, suggested improvements to 5G, or requests for clarification. I also agree with all Kevin Smith's comments, so I didn't repeat any.
[Sorry for delay - I had to find time to re-type after losing my review during a power outage]

Context questions.
Q1. Is this exercise treating the 5G specs solely in read-only mode? Or, if some useful design suggestions arise, is there any chance that the 3GPP might take them on board, given 5G is not fully baked yet?
[IJ] Guess it depends, need to say that I probably the small cog in the wheel. I would say that a lot of the standards work in IETF has been picked up by 3GPP and, I cannot say what will happen in the 5G context.

It's great that you have taken this initiative. Does the 3GPP ever formally ask the IETF for comments on its ideas for each new generation of radio? Given the layers above can be thought of as the 'customers' of the 3GPP, and these 'customer' layers are pretty much all maintained by the IETF, it would seem essential (and courteous) for the 3GPP to ask for customer comment, not least to co-ordinate with whatever future plans the IETF might have for the higher layers.
[IJ] What you say makes much sense. I guess there is some natural leakage between 3GPP and IETF as many of the IETFers are also active in 3GPP standardization. I don’t have full insight in the formal relation between 3GPP and IETF in these areas, not sure if that need to be better, but if that is the case, then it is probably a question for the area directors. Need to say that even though the formal channels are scarce it does probably not mean that 3GPP ignores IETF or what happens above IP in general. For instance the recent work around TCP RACK raises the question if it is necessary that LTE default radio bearers ensure in order delivery.

Q2. What are your plans for this draft? Are you asking for adoption? And if so, in which WG or RG? ICCRG looks the most appropriate, given the charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are scoped on just a subset of your doc.
[IJ] First objective is to get an increased interest in congestion control algorithm work in the IETF. Traditionally this kind of work seem to be more in the terms of conference papers in e.g. Sigcomm, the benefit with having such work in IETF is that input from people with a wider expertise area (including 3GPP experitise) can give algorithm proposals that work well for a wider set of access types. The work around Cubic is a first good step as is the work around DCTCP. What I wonder is if it feasible to take this a bit further ? I would too believe that the current doc fits better in ICCRG.  The algorithm examples are just.. examples. More fruitful work is probably to document different access types and from there derive a set of best current practices for congestion control algorithms

Section by Section Review

All sections

The document generally talks about the LTE protocol stack as if data is travelling down it. Are you implying that most sources of buffering delay are at the sending end, and the feedback end is pretty unencumbered by delays? I think it would be useful to explicitly say something about this (whether true or not), to explain why only one direction of travel is discussed in the doc.
[IJ] The feedback path is encumbered by delays, it is perhaps not that clear but section 2.3.2 implies that delay (variation) in uplink will affect the ACKs for downlink data traffic. That said, the difference in DL/UL traffic is today something like 10:1 and that is probably one reason why one can get the feeling that the draft is slanted towards downlink data traffic, this may however change with time as more machine type traffic may enter the system.

2.1 PDCP layer

    PDCP also ensures that all
   packets are delivered in order up to higher layers.
I think this means "in the order sent into the link (PDCP) layer". This ought to be clarified, otherwise, when writing in the context of a layer 4 audience, it might be assumed this means in the order sent by the original transport layer source.
[IJ] Yes, you are right , reordering can still occur e.g. in the wireless backhaul

   keep the amount of data in flight as small as possible, without sacrificing throughput.
Is this just a rather convoluted way of saying "keep the RTT as low as possible"? (which in turn implies keeping queuing delay as low as possible assuming you cannot influence the physical path.) Or were you trying to make a more subtle point? - if so, it was lost on me.
[IJ] No, it is exactly as you describe below

   Reliable delivery at handover may however be turned off or is simply not implemented,
It would be useful if you could give a feel for roughly how often (in %) these cases hold.
[IJ] I don’t believe that it is possible to give a figure as this is vendor specific.

2.2 RLC layer

   role ... is to ensure that packets are delivered in order up to the higher layers
Eh? The PDCP section just said it did ordering. And the rest of the RLC section talks exclusively about buffering delays necessary while filling transport blocks. If ordering is the role of both layers, surely one will have nothing to do once the other has got everything in order?
[IJ] RLC handles ordering due to various MAC layer failures. PDCP handles ordering  when handover occurs

   varying size of the available transport
Nit: Given 'transport' means e2e for the audience of this draft, pls use bearer or somehow disambiguate this word.
[IJ] OK

Gap#1: In the draft, it seems like transport blocks arrive by magic, to be filled by data buffered in the RLC layer. To understand the delay compromises here, I think the draft needs to explain what determines how much transport block capacity each flow is given, and how they can influence it. In 3G, I believe that the transport layer could ask the radio resource controller to change radio resource allocation. Is this the same in 4G & 5G? Does the backlog of the buffer into the RLC layer have any automatic influence over allocation of available resources? I assume traffic class can also be used to influence allocation.
[IJ] Will updated with more detail

Gap#2: In a 3G stack, the RLC layer multiplexes all the active application streams into MAC layer transport block by dicing up buffered packets and packing the pieces into each transport block. Then, at the other end of the link, each piece has to be held until the rest of the packet arrives. This sacrifices delay for bandwidth efficiency. Is this still done in 4G and 5G? If so, it would be good to write about mux delay in the draft too.
[IJ] OK, will add more info

2.3 MAC layer

   the maximum number of retransmissions is configurable

Wouldn't it be better to set a limit on the time spent retransmitting, so it would automatically give up with less retransmissions for longer transmission distances?
[IJ] Possible, 4G specifies a limit to the number of retransmissions, in practice it can be translated to time as each retransmission is 8ms later.

s/only checks for the presence on download data only at regular intervals/
 /only checks for the presence of download data at regular intervals/
[OK]

2.3.2 Uplink scheduling

   The scheduling request does not indicate how many bytes there are in the uplink queue.
Seems like a bad omission. Is there a reason? If there are less bytes queued than the base station's grant, surely it would be useful for the base station to know, so that it can grant more to something else while the previous short transmission is completing.
[IJ] It is a design choice. A buffer status report is more bulky, a scheduling request is just one bit.

You say that when HARQ breaks up a sequence of ACKs, it can also trigger "coalescing issues". I can understand that it could cause ACK compression, but are you implying something coalesces the ACKs or the subsequent data? What does the coalescing? Similarly, in the last para of 2.3.2 you say "Reduced ACKs can unfortunately cause coalescing." and again I'm not sure what is coalescing what here.
[IJ] What I can see in simulations is that the ACK compression leads to coalescing which further can increase jitter.

There could be a positive side to the interaction between ACK clocking and link aggregation, at least for long-running flows. This has been observed in 802.11n WiFi where the buffering and aggregation of multiple ACKs into one transmission grant tend to lead the sender to bunch multiple data segments so that they all nicely fill one transmission grant in the other direction [Showail14]. As long as the transmission grant does not wait for more data (which unfortunately I don't think is the case in 3G/4G/5G), this could be good for bandwidth utilisation of elephants and latency of mice, including when they are mixed.

[Showail14] Showail, A., Jamshaid, K. & Shihada, B., "WQM: An Aggregation-aware Queue Management Scheme for IEEE 802.11N Based Networks," In: Proceedings of the 2014 ACM SIGCOMM Workshop on Capacity Sharing Workshop CSWS '14 pp.15-20 ACM (2014)
<http://dl.acm.org/citation.cfm?id=2630097>
[IJ] Interesting ,will look at it, not sure though that it will work that well with 4G (cant say anything about 5G), in 4G the transmission grants can vary quite a lot over short time spans.

3.  4G and 5G evolution

s/a explicit/an explicit/

4.  Requirements for improved performance

s/bufferbloat/buffer/

5.  Congestion control examples

Regarding Hybrid Cubic using OWD measurements, I have seen a paper (under submission) about getting delay-based congestion controls (e.g. LEDBAT or CAIA Delay Gradient) to interwork with AQMs with various low delay targets. It's not in the context of cellular networks, but I'll try to remember to point it out once it gets published.
[IJ] Sounds interesting , looking forward to see the paper.

5.1 HyStartRestart

s/algorithm used/algorithm is used/

In the new code, it would be useful to explain why the second test before doubling ssthresh, ie, the test for how long it has been since the last hyrestart. Also, in the header definitions, if you intend there to be constraints on the relation between N_RTT_HYRESTART and N_RTT_LOW (e.g.  N_RTT_HYRESTART > N_RTT_LOW) you ought to say, because people will try other values.

Rather than the rather arbitrary wait before an arbitrary doubling of ssthresh, wouldn't it be better to continually increase ssthresh (not necessarily linearly) the longer the RTT has remained only just higher than the min RTT?
[IJ] Yes, could be an option

(BTW, surely the main problem with this modification to HyStartRestart is that is requires a number of round trips to reduce delay - a sort-of oxymoron.)
[IJ] Yes, have tried some more experiments with this algorithm. And I am getting more hesitant. The problem is that when I add more noise (modeling of small objects traffic) into the system simulation, then I also get more delay jitter and the delay detection algorithm in HyStart is notoriously sensitive to delay jitter.

5.2.  Hybrid Cubic

   Furthermore it is assumed than the timestamp clock frequency in

   sender and receiver are identical or that the sender can infer the

   timestamp clock frequency of the receiver and recompute timestamp

   values based on this information.
Recently on tcpm I sent you a pointer to the chirping paper Mirja & I did, which also made this assumption. Partly prompted by that, Richard Scheffenegger tried to get people interested in standardising use of the timestamp option on a SYN to negotiate stuff like timer resolution. I don't think it went anywhere. Do you think we ought to revitalise that? You mention inference - have you tried that - are there cases where it doesn't work?
[IJ] Yes, actually wondered what happened to Richards draft. I only briefly tried out the inference algorithm with a TCP LEDBAT implementation (https://github.com/silviov/TCP-LEDBAT/blob/master/src/tcp_ledbat.c). I recall that it took a while to converge. Also I am not sure here. Is Hz in e.g a linux stack fixed. ?

That's it. Thanks again for the draft. V useful.

Bob

On 15/10/15 06:08, Ingemar Johansson S wrote:

Hi

A new IETF draft is submitted in response to general discussion that for instance TCP Cubic is not an appropriate congestion control algorithm for LTE.

The draft briefly outlines typical 4G access behavior on the PDCP, RLC and MAC  layers that can have an impact on transport protocol performance. The draft also outlines two relatively simple modifications to the Cubic congestion control.

/Ingemar

-----Original Message-----

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]

Sent: den 14 oktober 2015 20:17

To: Ingemar Johansson S

Subject: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

A new version of I-D, draft-johansson-cc-for-4g-5g-00.txt

has been successfully submitted by Ingemar Johansson and posted to the IETF repository.

Name:           draft-johansson-cc-for-4g-5g

Revision:       00

Title:          Congestion control for 4G and 5G access

Document date:  2015-10-14

Group:          Individual Submission

Pages:          14

URL:            https://www.ietf.org/internet-drafts/draft-johansson-cc-for-4g-5g-00.txt

Status:         https://datatracker.ietf.org/doc/draft-johansson-cc-for-4g-5g/

Htmlized:       https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-00

Abstract:

   This memo outlines the challenge that 4G and 5G access brings for

   transport protocol congestion control and also outlines a few simple

   examples that can improve transport protocol congestion control

   performance in 4G and 5G access.

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

--

________________________________________________________________

Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 4 Nov 2015 22:27:01 +0000

Ingemar,

On 04/11/15 13:27, Ingemar Johansson S wrote:
> *From:*Bob Briscoe [mailto:[email-masked]]
> *Sent:* den 29 oktober 2015 18:54
> *To:* Ingemar Johansson S; [email-masked]; [email-masked]; [email-masked]
> *Subject:* Re: [tsvwg] FW: New Version Notification for 
> draft-johansson-cc-for-4g-5g-00.txt
>
> Ingemar,
>
> As promised, here's my comments. They are a mix of suggested 
> improvements to the draft, suggested improvements to 5G, or requests 
> for clarification. I also agree with all Kevin Smith's comments, so I 
> didn't repeat any.
> *_Context questions._
> *Q1. Is this exercise treating the 5G specs solely in read-only mode? 
> Or, if some useful design suggestions arise, is there any chance that 
> the 3GPP might take them on board, given 5G is not fully baked yet?
>
> [IJ] Guess it depends, need to say that I probably the small cog in 
> the wheel. I would say that a lot of the standards work in IETF has 
> been picked up by 3GPP and, I cannot say what will happen in the 5G 
> context.
>
> It's great that you have taken this initiative. Does the 3GPP ever 
> formally ask the IETF for comments on its ideas for each new 
> generation of radio? Given the layers above can be thought of as the 
> 'customers' of the 3GPP, and these 'customer' layers are pretty much 
> all maintained by the IETF, it would seem essential (and courteous) 
> for the 3GPP to ask for customer comment, not least to co-ordinate 
> with whatever future plans the IETF might have for the higher layers.
>
> [IJ] What you say makes much sense. I guess there is some natural 
> leakage between 3GPP and IETF as many of the IETFers are also active 
> in 3GPP standardization. I don’t have full insight in the formal 
> relation between 3GPP and IETF in these areas, not sure if that need 
> to be better, but if that is the case, then it is probably a question 
> for the area directors. Need to say that even though the formal 
> channels are scarce it does probably not mean that 3GPP ignores IETF 
> or what happens above IP in general. For instance the recent work 
> around TCP RACK raises the question if it is necessary that LTE 
> default radio bearers ensure in order delivery.
>
I suspect that 3GPP radio resource control people do not cross-fertilize 
much with the IETF.
So maybe it is as much a question of how to co-ordinate between these 
radio people and L4, which I would imagine is as much a problem within 
3GPP as between 3GPP and IETF.
>
>
>
> Q2. What are your plans for this draft? Are you asking for adoption? 
> And if so, in which WG or RG? ICCRG looks the most appropriate, given 
> the charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are 
> scoped on just a subset of your doc.
>
> [IJ] First objective is to get an increased interest in congestion 
> control algorithm work in the IETF. Traditionally this kind of work 
> seem to be more in the terms of conference papers in e.g. Sigcomm, the 
> benefit with having such work in IETF is that input from people with a 
> wider expertise area (including 3GPP experitise) can give algorithm 
> proposals that work well for a wider set of access types. The work 
> around Cubic is a first good step as is the work around DCTCP. What I 
> wonder is if it feasible to take this a bit further ? I would too 
> believe that the current doc fits better in ICCRG.  The algorithm 
> examples are just.. examples. More fruitful work is probably to 
> document different access types and from there derive a set of best 
> current practices for congestion control algorithms
>
I'm sure the iccrg chairs would like this to happen. And initiatives 
like yours to come with proposals to propose specific code to take 
account of L2 I think generate the right level of dialogue.
>
>
>
> *_Section by Section Review_
> *
> *All sections
> *
> The document generally talks about the LTE protocol stack as if data 
> is travelling down it. Are you implying that most sources of buffering 
> delay are at the sending end, and the feedback end is pretty 
> unencumbered by delays? I think it would be useful to explicitly say 
> something about this (whether true or not), to explain why only one 
> direction of travel is discussed in the doc.
>
> [IJ] The feedback path is encumbered by delays, it is perhaps not that 
> clear but section 2.3.2 implies that delay (variation) in uplink will 
> affect the ACKs for downlink data traffic. That said, the difference 
> in DL/UL traffic is today something like 10:1 and that is probably one 
> reason why one can get the feeling that the draft is slanted towards 
> downlink data traffic, this may however change with time as more 
> machine type traffic may enter the system.
>
I was thinking more in terms of up the stack vs. down the stack, not 
uplink/downlink. I.e. for both cases:
* downlink direction: are there any buffering concerns coming up the 
stack at the UE, or on the feedback back-channel from the UE?
* uplink direction: are there any buffering concerns coming up the stack 
at the eNodeB, or on the feedback back-channel from the eNodeB?

For instance, in the UE, perhaps one app blocking the read of another 
due to the way {3-5}G multiplexes different streams into the same PDCP 
frames?
> *2.1 PDCP layer
> *
> I think this means "in the order sent into the link (PDCP) layer". 
> This ought to be clarified, otherwise, when writing in the context of 
> a layer 4 audience, it might be assumed this means in the order sent 
> by the original transport layer source.
>
> Is this just a rather convoluted way of saying "keep the RTT as low as 
> possible"? (which in turn implies keeping queuing delay as low as 
> possible assuming you cannot influence the physical path.) Or were you 
> trying to make a more subtle point? - if so, it was lost on me.
>
> *2.2 RLC layer*
> Eh? The PDCP section just said it did ordering. And the rest of the 
> RLC section talks exclusively about buffering delays necessary while 
> filling transport blocks. If ordering is the role of both layers, 
> surely one will have nothing to do once the other has got everything 
> in order?
>
> Nit: Given 'transport' means e2e for the audience of this draft, pls 
> use bearer or somehow disambiguate this word.
>
> [IJ] OK
>
>
>
>
> Gap#1: In the draft, it seems like transport blocks arrive by magic, 
> to be filled by data buffered in the RLC layer. To understand the 
> delay compromises here, I think the draft needs to explain what 
> determines how much transport block capacity each flow is given, and 
> how they can influence it. In 3G, I believe that the transport layer 
> could ask the radio resource controller to change radio resource 
> allocation. Is this the same in 4G & 5G? Does the backlog of the 
> buffer into the RLC layer have any automatic influence over allocation 
> of available resources? I assume traffic class can also be used to 
> influence allocation.
>
> Gap#2: In a 3G stack, the RLC layer multiplexes all the active 
> application streams into MAC layer transport block by dicing up 
> buffered packets and packing the pieces into each transport block. 
> Then, at the other end of the link, each piece has to be held until 
> the rest of the packet arrives. This sacrifices delay for bandwidth 
> efficiency. Is this still done in 4G and 5G? If so, it would be good 
> to write about mux delay in the draft too.
>
> *2.3 MAC layer
> *
> Wouldn't it be better to set a limit on the /time/ spent 
> retransmitting, so it would automatically give up with less 
> retransmissions for longer transmission distances?
> [IJ] Possible, 4G specifies a limit to the number of retransmissions, 
> in practice it can be translated to time as each retransmission is 8ms 
> later.
>
I didn't realise it was independent of link-RTT. Then the more basic 
question is why is it independent of link-RTT? Surely on a short link it 
can assume that it needs to retransmit more quickly than on a long link?

You can see that my question was assuming that retransmission happened 
more often on a short link, then I was saying it should still be allowed 
the same amount of time to keep retrying (so on a short link it would be 
allowed to retransmit more times before giving up, but the max delay 
would be the same).

8ms! I was assuming it would be rather much less than that. Surely a 
link-layer ACK could never take that long if the frame was going to get 
through.
> *2.3.2 Uplink scheduling
> *
> Seems like a bad omission. Is there a reason? If there are less bytes 
> queued than the base station's grant, surely it would be useful for 
> the base station to know, so that it can grant more to something else 
> while the previous short transmission is completing.
> You say that when HARQ breaks up a sequence of ACKs, it can also 
> trigger "coalescing issues". I can understand that it could cause ACK 
> compression, but are you implying something coalesces the ACKs or the 
> subsequent data? What does the coalescing? Similarly, in the last para 
> of 2.3.2 you say "Reduced ACKs can unfortunately cause coalescing." 
> and again I'm not sure what is coalescing what here.
>
> [IJ] What I can see in simulations is that the ACK compression leads 
> to coalescing which further can increase jitter.
>
I still don't understand. The question was "What is coalescing what?"
>
>
> There could be a positive side to the interaction between ACK clocking 
> and link aggregation, at least for long-running flows. This has been 
> observed in 802.11n WiFi where the buffering and aggregation of 
> multiple ACKs into one transmission grant tend to lead the sender to 
> bunch multiple data segments so that they all nicely fill one 
> transmission grant in the other direction [Showail14]. As long as the 
> transmission grant does not wait for more data (which unfortunately I 
> don't think is the case in 3G/4G/5G), this could be good for bandwidth 
> utilisation of elephants and latency of mice, including when they are 
> mixed.
>
> [Showail14] Showail, A., Jamshaid, K. & Shihada, B., "WQM: An 
> Aggregation-aware Queue Management Scheme for IEEE 802.11N Based 
> Networks," In: Proceedings of the 2014 ACM SIGCOMM Workshop on 
> Capacity Sharing Workshop CSWS '14 pp.15-20 ACM (2014)
> [IJ] Interesting ,will look at it, not sure though that it will work 
> that well with 4G (cant say anything about 5G), in 4G the transmission 
> grants can vary quite a lot over short time spans.
>
>
> *3.  4G and 5G evolution
> *
> *4.  Requirements for improved performance
> *
> s/bufferbloat/buffer/
>
> *5.  Congestion control examples
> *
> Regarding Hybrid Cubic using OWD measurements, I have seen a paper 
> (under submission) about getting delay-based congestion controls (e.g. 
> LEDBAT or CAIA Delay Gradient) to interwork with AQMs with various low 
> delay targets. It's not in the context of cellular networks, but I'll 
> try to remember to point it out once it gets published.
>
> *5.1 HyStartRestart
> *
> In the new code, it would be useful to explain why the second test 
> before doubling ssthresh, ie, the test for how long it has been since 
> the last hyrestart. Also, in the header definitions, if you intend 
> there to be constraints on the relation between N_RTT_HYRESTART and 
> N_RTT_LOW (e.g.  N_RTT_HYRESTART > N_RTT_LOW) you ought to say, 
> because people will try other values.
>
> Rather than the rather arbitrary wait before an arbitrary doubling of 
> ssthresh, wouldn't it be better to continually increase ssthresh (not 
> necessarily linearly) the longer the RTT has remained only just higher 
> than the min RTT?
>
> (BTW, surely the main problem with this modification to HyStartRestart 
> is that is requires a number of round trips to reduce delay - a 
> sort-of oxymoron.)
>
> [IJ] Yes, have tried some more experiments with this algorithm. And I 
> am getting more hesitant. The problem is that when I add more noise 
> (modeling of small objects traffic) into the system simulation, then I 
> also get more delay jitter and the delay detection algorithm in 
> HyStart is notoriously sensitive to delay jitter.
>
>
>
> *5.2.  Hybrid Cubic*
>
> Recently on tcpm I sent you a pointer to the chirping paper Mirja & I 
> did, which also made this assumption. Partly prompted by that, Richard 
> Scheffenegger tried to get people interested in standardising use of 
> the timestamp option on a SYN to negotiate stuff like timer 
> resolution. I don't think it went anywhere. Do you think we ought to 
> revitalise that? You mention inference - have you tried that - are 
> there cases where it doesn't work?
>
> [IJ] Yes, actually wondered what happened to Richards draft. I only 
> briefly tried out the inference algorithm with a TCP LEDBAT 
> implementation 
> (https://github.com/silviov/TCP-LEDBAT/blob/master/src/tcp_ledbat.c). 
> I recall that it took a while to converge. Also I am not sure here. Is 
> Hz in e.g a linux stack fixed. ?
>
I believe so. But I don't know about embedded systems etc.

Cheers

Bob
>     From:[email-masked] <mailto:[email-masked]>  [mailto:[email-masked]]
>
>     URL:https://www.ietf.org/internet-drafts/draft-johansson-cc-for-4g-5g-00.txt
>
>     Status:https://datatracker.ietf.org/doc/draft-johansson-cc-for-4g-5g/
>
>     Htmlized:https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-00
>
>     Abstract:
>
> Bob Briscoehttp://bobbriscoe.net/

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 5 Nov 2015 14:12:37 +0900

I have a question re: how much buffer or backlog is needed to prevent the
schedular from reducing the rate or even increase the rate. I think this
key why cubic or bufferbloat cc wins bc other queue controlling cc would
lose out.

For example if the scheduler needs 1mb backlog to maintain at 40mbps. A
dctcp or vegas keeping the queue well below will get say 5mbps.

The draft mentions about this but the advise is a bit vague for cc designer.
On Nov 5, 2015 7:27 AM, "Bob Briscoe" <[email-masked]> wrote:

> *From:* Bob Briscoe [mailto:[email-masked] <[email-masked]>]
> As promised, here's my comments. They are a mix of suggested improvements
> to the draft, suggested improvements to 5G, or requests for clarification.
> I also agree with all Kevin Smith's comments, so I didn't repeat any.
> *Context questions. *Q1. Is this exercise treating the 5G specs solely in
> read-only mode? Or, if some useful design suggestions arise, is there any
> chance that the 3GPP might take them on board, given 5G is not fully baked
> yet?
>
> [IJ] Guess it depends, need to say that I probably the small cog in the
> wheel. I would say that a lot of the standards work in IETF has been picked
> up by 3GPP and, I cannot say what will happen in the 5G context.
>
> formally ask the IETF for comments on its ideas for each new generation of
> radio? Given the layers above can be thought of as the 'customers' of the
> 3GPP, and these 'customer' layers are pretty much all maintained by the
> IETF, it would seem essential (and courteous) for the 3GPP to ask for
> customer comment, not least to co-ordinate with whatever future plans the
> IETF might have for the higher layers.
>
> [IJ] What you say makes much sense. I guess there is some natural leakage
> between 3GPP and IETF as many of the IETFers are also active in 3GPP
> standardization. I don’t have full insight in the formal relation between
> 3GPP and IETF in these areas, not sure if that need to be better, but if
> that is the case, then it is probably a question for the area directors.
> Need to say that even though the formal channels are scarce it does
> probably not mean that 3GPP ignores IETF or what happens above IP in
> general. For instance the recent work around TCP RACK raises the question
> if it is necessary that LTE default radio bearers ensure in order delivery.
>
> radio people and L4, which I would imagine is as much a problem within 3GPP
> as between 3GPP and IETF.
>
>
>
> Q2. What are your plans for this draft? Are you asking for adoption? And
> if so, in which WG or RG? ICCRG looks the most appropriate, given the
> charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are scoped on
> just a subset of your doc.
>
> [IJ] First objective is to get an increased interest in congestion control
> algorithm work in the IETF. Traditionally this kind of work seem to be more
> in the terms of conference papers in e.g. Sigcomm, the benefit with having
> such work in IETF is that input from people with a wider expertise area
> (including 3GPP experitise) can give algorithm proposals that work well for
> a wider set of access types. The work around Cubic is a first good step as
> is the work around DCTCP. What I wonder is if it feasible to take this a
> bit further ? I would too believe that the current doc fits better in
> ICCRG.  The algorithm examples are just.. examples. More fruitful work is
> probably to document different access types and from there derive a set of
> best current practices for congestion control algorithms
>
> I'm sure the iccrg chairs would like this to happen. And initiatives like
> yours to come with proposals to propose specific code to take account of L2
> I think generate the right level of dialogue.
>
>
>
>
> *Section by Section Review *
>
> *All sections *
> The document generally talks about the LTE protocol stack as if data is
> travelling down it. Are you implying that most sources of buffering delay
> are at the sending end, and the feedback end is pretty unencumbered by
> delays? I think it would be useful to explicitly say something about this
> (whether true or not), to explain why only one direction of travel is
> discussed in the doc.
>
> affect the ACKs for downlink data traffic. That said, the difference in
> DL/UL traffic is today something like 10:1 and that is probably one reason
> why one can get the feeling that the draft is slanted towards downlink data
> traffic, this may however change with time as more machine type traffic may
> enter the system.
>
> * downlink direction: are there any buffering concerns coming up the stack
> at the UE, or on the feedback back-channel from the UE?
> For instance, in the UE, perhaps one app blocking the read of another due
> to the way {3-5}G multiplexes different streams into the same PDCP frames?
>
>
>
>
> *2.1 PDCP layer *
> I think this means "in the order sent into the link (PDCP) layer". This
> ought to be clarified, otherwise, when writing in the context of a layer 4
> audience, it might be assumed this means in the order sent by the original
> transport layer source.
>
> possible"? (which in turn implies keeping queuing delay as low as possible
> assuming you cannot influence the physical path.) Or were you trying to
> make a more subtle point? - if so, it was lost on me.
>
> Eh? The PDCP section just said it did ordering. And the rest of the RLC
> section talks exclusively about buffering delays necessary while filling
> transport blocks. If ordering is the role of both layers, surely one will
> have nothing to do once the other has got everything in order?
>
> Nit: Given 'transport' means e2e for the audience of this draft, pls use
> bearer or somehow disambiguate this word.
>
> [IJ] OK
>
>
>
>
> Gap#1: In the draft, it seems like transport blocks arrive by magic, to be
> filled by data buffered in the RLC layer. To understand the delay
> compromises here, I think the draft needs to explain what determines how
> much transport block capacity each flow is given, and how they can
> influence it. In 3G, I believe that the transport layer could ask the radio
> resource controller to change radio resource allocation. Is this the same
> in 4G & 5G? Does the backlog of the buffer into the RLC layer have any
> automatic influence over allocation of available resources? I assume
> traffic class can also be used to influence allocation.
>
> Gap#2: In a 3G stack, the RLC layer multiplexes all the active application
> streams into MAC layer transport block by dicing up buffered packets and
> packing the pieces into each transport block. Then, at the other end of
> the link, each piece has to be held until the rest of the packet arrives.
> This sacrifices delay for bandwidth efficiency. Is this still done in 4G
> and 5G? If so, it would be good to write about mux delay in the draft too.
>
> *2.3 MAC layer *
> Wouldn't it be better to set a limit on the *time* spent retransmitting,
> so it would automatically give up with less retransmissions for longer
> transmission distances?
> [IJ] Possible, 4G specifies a limit to the number of retransmissions, in
> practice it can be translated to time as each retransmission is 8ms later.
>
> allowed to retransmit more times before giving up, but the max delay would
> be the same).
>
> *2.3.2 Uplink scheduling *
> queued than the base station's grant, surely it would be useful for the
> base station to know, so that it can grant more to something else while the
> previous short transmission is completing.
> You say that when HARQ breaks up a sequence of ACKs, it can also trigger
> "coalescing issues". I can understand that it could cause ACK
> subsequent data? What does the coalescing? Similarly, in the last para of
> 2.3.2 you say "Reduced ACKs can unfortunately cause coalescing." and again
> I'm not sure what is coalescing what here.
>
> [IJ] What I can see in simulations is that the ACK compression leads to
> coalescing which further can increase jitter.
>
> There could be a positive side to the interaction between ACK clocking and
> link aggregation, at least for long-running flows. This has been observed
> in 802.11n WiFi where the buffering and aggregation of multiple ACKs into
> one transmission grant tend to lead the sender to bunch multiple data
> segments so that they all nicely fill one transmission grant in the other
> direction [Showail14]. As long as the transmission grant does not wait
> for more data (which unfortunately I don't think is the case in 3G/4G/5G),
> this could be good for bandwidth utilisation of elephants and latency of
> mice, including when they are mixed.
>
> Aggregation-aware Queue Management Scheme for IEEE 802.11N Based Networks,"
> In: Proceedings of the 2014 ACM SIGCOMM Workshop on Capacity Sharing
> Workshop CSWS '14 pp.15-20 ACM (2014)
> [IJ] Interesting ,will look at it, not sure though that it will work that
> well with 4G (cant say anything about 5G), in 4G the transmission grants
> can vary quite a lot over short time spans.
>
>
>
>
>
> *3.  4G and 5G evolution *
> *4.  Requirements for improved performance *
> s/bufferbloat/buffer/
>
>
> *5.  Congestion control examples *
> Regarding Hybrid Cubic using OWD measurements, I have seen a paper (under
> submission) about getting delay-based congestion controls (e.g. LEDBAT or
> CAIA Delay Gradient) to interwork with AQMs with various low delay targets.
> It's not in the context of cellular networks, but I'll try to remember to
> point it out once it gets published.
>
> *5.1 HyStartRestart *
> In the new code, it would be useful to explain why the second test before
> doubling ssthresh, ie, the test for how long it has been since the last
> hyrestart. Also, in the header definitions, if you intend there to be
> constraints on the relation between N_RTT_HYRESTART and N_RTT_LOW (e.g.
> N_RTT_HYRESTART > N_RTT_LOW) you ought to say, because people will try
> other values.
>
> necessarily linearly) the longer the RTT has remained only just higher than
> the min RTT?
>
> (BTW, surely the main problem with this modification to HyStartRestart is
> that is requires a number of round trips to reduce delay - a sort-of
> oxymoron.)
>
> [IJ] Yes, have tried some more experiments with this algorithm. And I am
> getting more hesitant. The problem is that when I add more noise (modeling
> of small objects traffic) into the system simulation, then I also get more
> delay jitter and the delay detection algorithm in HyStart is notoriously
> sensitive to delay jitter.
>
>
>
> *5.2.  Hybrid Cubic*
>
> Recently on tcpm I sent you a pointer to the chirping paper Mirja & I did,
> which also made this assumption. Partly prompted by that, Richard
> Scheffenegger tried to get people interested in standardising use of the
> timestamp option on a SYN to negotiate stuff like timer resolution. I don't
> think it went anywhere. Do you think we ought to revitalise that? You
> mention inference - have you tried that - are there cases where it doesn't
> work?
>
> briefly tried out the inference algorithm with a TCP LEDBAT implementation (
> https://github.com/silviov/TCP-LEDBAT/blob/master/src/tcp_ledbat.c). I
> recall that it took a while to converge. Also I am not sure here. Is Hz in
> e.g a linux stack fixed. ?
>
> From: [email-masked] [mailto:[email-masked] <[email-masked]>]
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>
>

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 5 Nov 2015 14:26:40 +0900

I also support this doc to be adopted by iccrg or tsvwg

---

From: David Ros <[email-masked]>
Date: Thu, 5 Nov 2015 19:13:08 +0900

Ingemar,

We believe this is a useful document, agree that it's a good match for ICCRG, and would be happy with ICCRG being the home of this draft.

Thanks,

Michael & David (as ICCRG chairs)

On 04 Nov 2015, at 22:27, Ingemar Johansson S <[email-masked]> wrote:

> From: [email-masked] [mailto:[email-masked]]

---

From: "Black, David" <[email-masked]>
Date: Thu, 5 Nov 2015 22:11:57 +0000

As a tsvwg chair, I concur that ICCRG would be a good initial home for this draft.

Thanks,
--David

From: tsvwg [mailto:[email-masked]] On Behalf Of David Ros
Sent: Thursday, November 05, 2015 5:13 AM
To: Ingemar Johansson S
Cc: [email-masked]; [email-masked]; [email-masked]; Michael Welzl
Subject: Re: [tsvwg] [tcpm] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

Ingemar,

Thanks,

On 04 Nov 2015, at 22:27, Ingemar Johansson S <[email-masked]<mailto:[email-masked]>> wrote:

Hi

/Ingemar

From: Bob Briscoe [mailto:[email-masked]]
Sent: den 29 oktober 2015 18:54
To: Ingemar Johansson S; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [tsvwg] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

Ingemar,

Context questions.

It's great that you have taken this initiative. Does the 3GPP ever formally ask the IETF for comments on its ideas for each new generation of radio? Given the layers above can be thought of as the 'customers' of the 3GPP, and these 'customer' layers are pretty much all maintained by the IETF, it would seem essential (and courteous) for the 3GPP to ask for customer comment, not least to co-ordinate with whatever future plans the IETF might have for the higher layers.
[IJ] What you say makes much sense. I guess there is some natural leakage between 3GPP and IETF as many of the IETFers are also active in 3GPP standardization. I don't have full insight in the formal relation between 3GPP and IETF in these areas, not sure if that need to be better, but if that is the case, then it is probably a question for the area directors. Need to say that even though the formal channels are scarce it does probably not mean that 3GPP ignores IETF or what happens above IP in general. For instance the recent work around TCP RACK raises the question if it is necessary that LTE default radio bearers ensure in order delivery.

All sections

2.1 PDCP layer

   Reliable delivery at handover may however be turned off or is simply not implemented,
It would be useful if you could give a feel for roughly how often (in %) these cases hold.
[IJ] I don't believe that it is possible to give a figure as this is vendor specific.

2.2 RLC layer

2.3 MAC layer

2.3.2 Uplink scheduling

3.  4G and 5G evolution

s/bufferbloat/buffer/

5.1 HyStartRestart

5.2.  Hybrid Cubic

Bob

Hi

/Ingemar

-----Original Message-----

To: Ingemar Johansson S

Revision:       00

Pages:          14

Abstract:

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org/>.

The IETF Secretariat

--

Bob Briscoe                               http://bobbriscoe.net/
_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Nov 2015 20:29:01 +0000

Hi

You are right that the message is quite vague, the reason is that at least I don’t have an exact answer to that question.
An attempt at an exact rule is that there should be exactly as much data available as can be scheduled in the next scheduling event.
The problem is that it is not obvious when that scheduling event occurs (depends on scheduling algorithm) or how much data can be scheduled.
So, if one look at peak throughput alone it is true that it pays off to use CC algorithms that generate larger standing queues (Cubic is better than LEDBAT or Vegas). The price paid for this is of course that latency sensitive services can suffer when large standing queues are allowed.
In short, one need to compromise, if one take the example that WebRTC media traverses the same access, then it is likelky better to accept that the file transfer takes a while longer.

/Ingemar

From: Yuchung Cheng [mailto:[email-masked]]
Sent: den 5 november 2015 06:13
To: Bob Briscoe
Cc: [email-masked] Extensions; Ingemar Johansson S; [email-masked]; [email-masked]
Subject: Re: [tcpm] [tsvwg] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

I have a question re: how much buffer or backlog is needed to prevent the schedular from reducing the rate or even increase the rate. I think this key why cubic or bufferbloat cc wins bc other queue controlling cc would lose out.

For example if the scheduler needs 1mb backlog to maintain at 40mbps. A dctcp or vegas keeping the queue well below will get say 5mbps.

The draft mentions about this but the advise is a bit vague for cc designer.
On Nov 5, 2015 7:27 AM, "Bob Briscoe" <[email-masked]<mailto:[email-masked]>> wrote:
Ingemar,
On 04/11/15 13:27, Ingemar Johansson S wrote:
Hi

/Ingemar

Ingemar,

Context questions.

It's great that you have taken this initiative. Does the 3GPP ever formally ask the IETF for comments on its ideas for each new generation of radio? Given the layers above can be thought of as the 'customers' of the 3GPP, and these 'customer' layers are pretty much all maintained by the IETF, it would seem essential (and courteous) for the 3GPP to ask for customer comment, not least to co-ordinate with whatever future plans the IETF might have for the higher layers.
[IJ] What you say makes much sense. I guess there is some natural leakage between 3GPP and IETF as many of the IETFers are also active in 3GPP standardization. I don’t have full insight in the formal relation between 3GPP and IETF in these areas, not sure if that need to be better, but if that is the case, then it is probably a question for the area directors. Need to say that even though the formal channels are scarce it does probably not mean that 3GPP ignores IETF or what happens above IP in general. For instance the recent work around TCP RACK raises the question if it is necessary that LTE default radio bearers ensure in order delivery.
I suspect that 3GPP radio resource control people do not cross-fertilize much with the IETF.
So maybe it is as much a question of how to co-ordinate between these radio people and L4, which I would imagine is as much a problem within 3GPP as between 3GPP and IETF.

Q2. What are your plans for this draft? Are you asking for adoption? And if so, in which WG or RG? ICCRG looks the most appropriate, given the charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are scoped on just a subset of your doc.
[IJ] First objective is to get an increased interest in congestion control algorithm work in the IETF. Traditionally this kind of work seem to be more in the terms of conference papers in e.g. Sigcomm, the benefit with having such work in IETF is that input from people with a wider expertise area (including 3GPP experitise) can give algorithm proposals that work well for a wider set of access types. The work around Cubic is a first good step as is the work around DCTCP. What I wonder is if it feasible to take this a bit further ? I would too believe that the current doc fits better in ICCRG.  The algorithm examples are just.. examples. More fruitful work is probably to document different access types and from there derive a set of best current practices for congestion control algorithms
I'm sure the iccrg chairs would like this to happen. And initiatives like yours to come with proposals to propose specific code to take account of L2 I think generate the right level of dialogue.

All sections

The document generally talks about the LTE protocol stack as if data is travelling down it. Are you implying that most sources of buffering delay are at the sending end, and the feedback end is pretty unencumbered by delays? I think it would be useful to explicitly say something about this (whether true or not), to explain why only one direction of travel is discussed in the doc.
[IJ] The feedback path is encumbered by delays, it is perhaps not that clear but section 2.3.2 implies that delay (variation) in uplink will affect the ACKs for downlink data traffic. That said, the difference in DL/UL traffic is today something like 10:1 and that is probably one reason why one can get the feeling that the draft is slanted towards downlink data traffic, this may however change with time as more machine type traffic may enter the system.
I was thinking more in terms of up the stack vs. down the stack, not uplink/downlink. I.e. for both cases:
* downlink direction: are there any buffering concerns coming up the stack at the UE, or on the feedback back-channel from the UE?
* uplink direction: are there any buffering concerns coming up the stack at the eNodeB, or on the feedback back-channel from the eNodeB?

For instance, in the UE, perhaps one app blocking the read of another due to the way {3-5}G multiplexes different streams into the same PDCP frames?

2.1 PDCP layer

2.2 RLC layer

2.3 MAC layer

Wouldn't it be better to set a limit on the time spent retransmitting, so it would automatically give up with less retransmissions for longer transmission distances?
[IJ] Possible, 4G specifies a limit to the number of retransmissions, in practice it can be translated to time as each retransmission is 8ms later.
I didn't realise it was independent of link-RTT. Then the more basic question is why is it independent of link-RTT? Surely on a short link it can assume that it needs to retransmit more quickly than on a long link?

You can see that my question was assuming that retransmission happened more often on a short link, then I was saying it should still be allowed the same amount of time to keep retrying (so on a short link it would be allowed to retransmit more times before giving up, but the max delay would be the same).

8ms! I was assuming it would be rather much less than that. Surely a link-layer ACK could never take that long if the frame was going to get through.

2.3.2 Uplink scheduling

3.  4G and 5G evolution

s/bufferbloat/buffer/

5.1 HyStartRestart

5.2.  Hybrid Cubic

Cheers

Bob

Bob

Hi

/Ingemar

-----Original Message-----

To: Ingemar Johansson S

Revision:       00

Pages:          14

Abstract:

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

The IETF Secretariat

--

--