Subject: Re: [iccrg] [tsvwg] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 4 Nov 2015 13:27:51 +0000

Hi

And thanks for the review, comments/answers/questions inline below

/Ingemar

From: Bob Briscoe [mailto:[email-masked]]
Sent: den 29 oktober 2015 18:54
To: Ingemar Johansson S; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

Ingemar,

As promised, here's my comments. They are a mix of suggested improvements to the draft, suggested improvements to 5G, or requests for clarification. I also agree with all Kevin Smith's comments, so I didn't repeat any.
[Sorry for delay - I had to find time to re-type after losing my review during a power outage]

Context questions.
Q1. Is this exercise treating the 5G specs solely in read-only mode? Or, if some useful design suggestions arise, is there any chance that the 3GPP might take them on board, given 5G is not fully baked yet?
[IJ] Guess it depends, need to say that I probably the small cog in the wheel. I would say that a lot of the standards work in IETF has been picked up by 3GPP and, I cannot say what will happen in the 5G context.

It's great that you have taken this initiative. Does the 3GPP ever formally ask the IETF for comments on its ideas for each new generation of radio? Given the layers above can be thought of as the 'customers' of the 3GPP, and these 'customer' layers are pretty much all maintained by the IETF, it would seem essential (and courteous) for the 3GPP to ask for customer comment, not least to co-ordinate with whatever future plans the IETF might have for the higher layers.
[IJ] What you say makes much sense. I guess there is some natural leakage between 3GPP and IETF as many of the IETFers are also active in 3GPP standardization. I don’t have full insight in the formal relation between 3GPP and IETF in these areas, not sure if that need to be better, but if that is the case, then it is probably a question for the area directors. Need to say that even though the formal channels are scarce it does probably not mean that 3GPP ignores IETF or what happens above IP in general. For instance the recent work around TCP RACK raises the question if it is necessary that LTE default radio bearers ensure in order delivery.

Q2. What are your plans for this draft? Are you asking for adoption? And if so, in which WG or RG? ICCRG looks the most appropriate, given the charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are scoped on just a subset of your doc.
[IJ] First objective is to get an increased interest in congestion control algorithm work in the IETF. Traditionally this kind of work seem to be more in the terms of conference papers in e.g. Sigcomm, the benefit with having such work in IETF is that input from people with a wider expertise area (including 3GPP experitise) can give algorithm proposals that work well for a wider set of access types. The work around Cubic is a first good step as is the work around DCTCP. What I wonder is if it feasible to take this a bit further ? I would too believe that the current doc fits better in ICCRG.  The algorithm examples are just.. examples. More fruitful work is probably to document different access types and from there derive a set of best current practices for congestion control algorithms

Section by Section Review

All sections

The document generally talks about the LTE protocol stack as if data is travelling down it. Are you implying that most sources of buffering delay are at the sending end, and the feedback end is pretty unencumbered by delays? I think it would be useful to explicitly say something about this (whether true or not), to explain why only one direction of travel is discussed in the doc.
[IJ] The feedback path is encumbered by delays, it is perhaps not that clear but section 2.3.2 implies that delay (variation) in uplink will affect the ACKs for downlink data traffic. That said, the difference in DL/UL traffic is today something like 10:1 and that is probably one reason why one can get the feeling that the draft is slanted towards downlink data traffic, this may however change with time as more machine type traffic may enter the system.

2.1 PDCP layer

    PDCP also ensures that all
   packets are delivered in order up to higher layers.
I think this means "in the order sent into the link (PDCP) layer". This ought to be clarified, otherwise, when writing in the context of a layer 4 audience, it might be assumed this means in the order sent by the original transport layer source.
[IJ] Yes, you are right , reordering can still occur e.g. in the wireless backhaul

   keep the amount of data in flight as small as possible, without sacrificing throughput.
Is this just a rather convoluted way of saying "keep the RTT as low as possible"? (which in turn implies keeping queuing delay as low as possible assuming you cannot influence the physical path.) Or were you trying to make a more subtle point? - if so, it was lost on me.
[IJ] No, it is exactly as you describe below

   Reliable delivery at handover may however be turned off or is simply not implemented,
It would be useful if you could give a feel for roughly how often (in %) these cases hold.
[IJ] I don’t believe that it is possible to give a figure as this is vendor specific.

2.2 RLC layer

   role ... is to ensure that packets are delivered in order up to the higher layers
Eh? The PDCP section just said it did ordering. And the rest of the RLC section talks exclusively about buffering delays necessary while filling transport blocks. If ordering is the role of both layers, surely one will have nothing to do once the other has got everything in order?
[IJ] RLC handles ordering due to various MAC layer failures. PDCP handles ordering  when handover occurs

   varying size of the available transport
Nit: Given 'transport' means e2e for the audience of this draft, pls use bearer or somehow disambiguate this word.
[IJ] OK

Gap#1: In the draft, it seems like transport blocks arrive by magic, to be filled by data buffered in the RLC layer. To understand the delay compromises here, I think the draft needs to explain what determines how much transport block capacity each flow is given, and how they can influence it. In 3G, I believe that the transport layer could ask the radio resource controller to change radio resource allocation. Is this the same in 4G & 5G? Does the backlog of the buffer into the RLC layer have any automatic influence over allocation of available resources? I assume traffic class can also be used to influence allocation.
[IJ] Will updated with more detail

Gap#2: In a 3G stack, the RLC layer multiplexes all the active application streams into MAC layer transport block by dicing up buffered packets and packing the pieces into each transport block. Then, at the other end of the link, each piece has to be held until the rest of the packet arrives. This sacrifices delay for bandwidth efficiency. Is this still done in 4G and 5G? If so, it would be good to write about mux delay in the draft too.
[IJ] OK, will add more info

2.3 MAC layer

   the maximum number of retransmissions is configurable

Wouldn't it be better to set a limit on the time spent retransmitting, so it would automatically give up with less retransmissions for longer transmission distances?
[IJ] Possible, 4G specifies a limit to the number of retransmissions, in practice it can be translated to time as each retransmission is 8ms later.

s/only checks for the presence on download data only at regular intervals/
 /only checks for the presence of download data at regular intervals/
[OK]

2.3.2 Uplink scheduling

   The scheduling request does not indicate how many bytes there are in the uplink queue.
Seems like a bad omission. Is there a reason? If there are less bytes queued than the base station's grant, surely it would be useful for the base station to know, so that it can grant more to something else while the previous short transmission is completing.
[IJ] It is a design choice. A buffer status report is more bulky, a scheduling request is just one bit.

You say that when HARQ breaks up a sequence of ACKs, it can also trigger "coalescing issues". I can understand that it could cause ACK compression, but are you implying something coalesces the ACKs or the subsequent data? What does the coalescing? Similarly, in the last para of 2.3.2 you say "Reduced ACKs can unfortunately cause coalescing." and again I'm not sure what is coalescing what here.
[IJ] What I can see in simulations is that the ACK compression leads to coalescing which further can increase jitter.

There could be a positive side to the interaction between ACK clocking and link aggregation, at least for long-running flows. This has been observed in 802.11n WiFi where the buffering and aggregation of multiple ACKs into one transmission grant tend to lead the sender to bunch multiple data segments so that they all nicely fill one transmission grant in the other direction [Showail14]. As long as the transmission grant does not wait for more data (which unfortunately I don't think is the case in 3G/4G/5G), this could be good for bandwidth utilisation of elephants and latency of mice, including when they are mixed.

[Showail14] Showail, A., Jamshaid, K. & Shihada, B., "WQM: An Aggregation-aware Queue Management Scheme for IEEE 802.11N Based Networks," In: Proceedings of the 2014 ACM SIGCOMM Workshop on Capacity Sharing Workshop CSWS '14 pp.15-20 ACM (2014)
<http://dl.acm.org/citation.cfm?id=2630097>
[IJ] Interesting ,will look at it, not sure though that it will work that well with 4G (cant say anything about 5G), in 4G the transmission grants can vary quite a lot over short time spans.

3.  4G and 5G evolution

s/a explicit/an explicit/

4.  Requirements for improved performance

s/bufferbloat/buffer/

5.  Congestion control examples

Regarding Hybrid Cubic using OWD measurements, I have seen a paper (under submission) about getting delay-based congestion controls (e.g. LEDBAT or CAIA Delay Gradient) to interwork with AQMs with various low delay targets. It's not in the context of cellular networks, but I'll try to remember to point it out once it gets published.
[IJ] Sounds interesting , looking forward to see the paper.

5.1 HyStartRestart

s/algorithm used/algorithm is used/

In the new code, it would be useful to explain why the second test before doubling ssthresh, ie, the test for how long it has been since the last hyrestart. Also, in the header definitions, if you intend there to be constraints on the relation between N_RTT_HYRESTART and N_RTT_LOW (e.g.  N_RTT_HYRESTART > N_RTT_LOW) you ought to say, because people will try other values.

Rather than the rather arbitrary wait before an arbitrary doubling of ssthresh, wouldn't it be better to continually increase ssthresh (not necessarily linearly) the longer the RTT has remained only just higher than the min RTT?
[IJ] Yes, could be an option

(BTW, surely the main problem with this modification to HyStartRestart is that is requires a number of round trips to reduce delay - a sort-of oxymoron.)
[IJ] Yes, have tried some more experiments with this algorithm. And I am getting more hesitant. The problem is that when I add more noise (modeling of small objects traffic) into the system simulation, then I also get more delay jitter and the delay detection algorithm in HyStart is notoriously sensitive to delay jitter.

5.2.  Hybrid Cubic

   Furthermore it is assumed than the timestamp clock frequency in

   sender and receiver are identical or that the sender can infer the

   timestamp clock frequency of the receiver and recompute timestamp

   values based on this information.
Recently on tcpm I sent you a pointer to the chirping paper Mirja & I did, which also made this assumption. Partly prompted by that, Richard Scheffenegger tried to get people interested in standardising use of the timestamp option on a SYN to negotiate stuff like timer resolution. I don't think it went anywhere. Do you think we ought to revitalise that? You mention inference - have you tried that - are there cases where it doesn't work?
[IJ] Yes, actually wondered what happened to Richards draft. I only briefly tried out the inference algorithm with a TCP LEDBAT implementation (https://github.com/silviov/TCP-LEDBAT/blob/master/src/tcp_ledbat.c). I recall that it took a while to converge. Also I am not sure here. Is Hz in e.g a linux stack fixed. ?

That's it. Thanks again for the draft. V useful.

Bob

On 15/10/15 06:08, Ingemar Johansson S wrote:

Hi

A new IETF draft is submitted in response to general discussion that for instance TCP Cubic is not an appropriate congestion control algorithm for LTE.

The draft briefly outlines typical 4G access behavior on the PDCP, RLC and MAC  layers that can have an impact on transport protocol performance. The draft also outlines two relatively simple modifications to the Cubic congestion control.

/Ingemar

-----Original Message-----

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]

Sent: den 14 oktober 2015 20:17

To: Ingemar Johansson S

Subject: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

A new version of I-D, draft-johansson-cc-for-4g-5g-00.txt

has been successfully submitted by Ingemar Johansson and posted to the IETF repository.

Name:           draft-johansson-cc-for-4g-5g

Revision:       00

Title:          Congestion control for 4G and 5G access

Document date:  2015-10-14

Group:          Individual Submission

Pages:          14

URL:            https://www.ietf.org/internet-drafts/draft-johansson-cc-for-4g-5g-00.txt

Status:         https://datatracker.ietf.org/doc/draft-johansson-cc-for-4g-5g/

Htmlized:       https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-00

Abstract:

   This memo outlines the challenge that 4G and 5G access brings for

   transport protocol congestion control and also outlines a few simple

   examples that can improve transport protocol congestion control

   performance in 4G and 5G access.

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

--

________________________________________________________________

Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 4 Nov 2015 22:27:01 +0000

Ingemar,

On 04/11/15 13:27, Ingemar Johansson S wrote:
> *From:*Bob Briscoe [mailto:[email-masked]]
> *Sent:* den 29 oktober 2015 18:54
> *To:* Ingemar Johansson S; [email-masked]; [email-masked]; [email-masked]
> *Subject:* Re: [tsvwg] FW: New Version Notification for 
> draft-johansson-cc-for-4g-5g-00.txt
>
> Ingemar,
>
> As promised, here's my comments. They are a mix of suggested 
> improvements to the draft, suggested improvements to 5G, or requests 
> for clarification. I also agree with all Kevin Smith's comments, so I 
> didn't repeat any.
> *_Context questions._
> *Q1. Is this exercise treating the 5G specs solely in read-only mode? 
> Or, if some useful design suggestions arise, is there any chance that 
> the 3GPP might take them on board, given 5G is not fully baked yet?
>
> [IJ] Guess it depends, need to say that I probably the small cog in 
> the wheel. I would say that a lot of the standards work in IETF has 
> been picked up by 3GPP and, I cannot say what will happen in the 5G 
> context.
>
> It's great that you have taken this initiative. Does the 3GPP ever 
> formally ask the IETF for comments on its ideas for each new 
> generation of radio? Given the layers above can be thought of as the 
> 'customers' of the 3GPP, and these 'customer' layers are pretty much 
> all maintained by the IETF, it would seem essential (and courteous) 
> for the 3GPP to ask for customer comment, not least to co-ordinate 
> with whatever future plans the IETF might have for the higher layers.
>
> [IJ] What you say makes much sense. I guess there is some natural 
> leakage between 3GPP and IETF as many of the IETFers are also active 
> in 3GPP standardization. I don’t have full insight in the formal 
> relation between 3GPP and IETF in these areas, not sure if that need 
> to be better, but if that is the case, then it is probably a question 
> for the area directors. Need to say that even though the formal 
> channels are scarce it does probably not mean that 3GPP ignores IETF 
> or what happens above IP in general. For instance the recent work 
> around TCP RACK raises the question if it is necessary that LTE 
> default radio bearers ensure in order delivery.
>
I suspect that 3GPP radio resource control people do not cross-fertilize 
much with the IETF.
So maybe it is as much a question of how to co-ordinate between these 
radio people and L4, which I would imagine is as much a problem within 
3GPP as between 3GPP and IETF.
>
>
>
> Q2. What are your plans for this draft? Are you asking for adoption? 
> And if so, in which WG or RG? ICCRG looks the most appropriate, given 
> the charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are 
> scoped on just a subset of your doc.
>
> [IJ] First objective is to get an increased interest in congestion 
> control algorithm work in the IETF. Traditionally this kind of work 
> seem to be more in the terms of conference papers in e.g. Sigcomm, the 
> benefit with having such work in IETF is that input from people with a 
> wider expertise area (including 3GPP experitise) can give algorithm 
> proposals that work well for a wider set of access types. The work 
> around Cubic is a first good step as is the work around DCTCP. What I 
> wonder is if it feasible to take this a bit further ? I would too 
> believe that the current doc fits better in ICCRG.  The algorithm 
> examples are just.. examples. More fruitful work is probably to 
> document different access types and from there derive a set of best 
> current practices for congestion control algorithms
>
I'm sure the iccrg chairs would like this to happen. And initiatives 
like yours to come with proposals to propose specific code to take 
account of L2 I think generate the right level of dialogue.
>
>
>
> *_Section by Section Review_
> *
> *All sections
> *
> The document generally talks about the LTE protocol stack as if data 
> is travelling down it. Are you implying that most sources of buffering 
> delay are at the sending end, and the feedback end is pretty 
> unencumbered by delays? I think it would be useful to explicitly say 
> something about this (whether true or not), to explain why only one 
> direction of travel is discussed in the doc.
>
> [IJ] The feedback path is encumbered by delays, it is perhaps not that 
> clear but section 2.3.2 implies that delay (variation) in uplink will 
> affect the ACKs for downlink data traffic. That said, the difference 
> in DL/UL traffic is today something like 10:1 and that is probably one 
> reason why one can get the feeling that the draft is slanted towards 
> downlink data traffic, this may however change with time as more 
> machine type traffic may enter the system.
>
I was thinking more in terms of up the stack vs. down the stack, not 
uplink/downlink. I.e. for both cases:
* downlink direction: are there any buffering concerns coming up the 
stack at the UE, or on the feedback back-channel from the UE?
* uplink direction: are there any buffering concerns coming up the stack 
at the eNodeB, or on the feedback back-channel from the eNodeB?

For instance, in the UE, perhaps one app blocking the read of another 
due to the way {3-5}G multiplexes different streams into the same PDCP 
frames?
> *2.1 PDCP layer
> *
> I think this means "in the order sent into the link (PDCP) layer". 
> This ought to be clarified, otherwise, when writing in the context of 
> a layer 4 audience, it might be assumed this means in the order sent 
> by the original transport layer source.
>
> Is this just a rather convoluted way of saying "keep the RTT as low as 
> possible"? (which in turn implies keeping queuing delay as low as 
> possible assuming you cannot influence the physical path.) Or were you 
> trying to make a more subtle point? - if so, it was lost on me.
>
> *2.2 RLC layer*
> Eh? The PDCP section just said it did ordering. And the rest of the 
> RLC section talks exclusively about buffering delays necessary while 
> filling transport blocks. If ordering is the role of both layers, 
> surely one will have nothing to do once the other has got everything 
> in order?
>
> Nit: Given 'transport' means e2e for the audience of this draft, pls 
> use bearer or somehow disambiguate this word.
>
> [IJ] OK
>
>
>
>
> Gap#1: In the draft, it seems like transport blocks arrive by magic, 
> to be filled by data buffered in the RLC layer. To understand the 
> delay compromises here, I think the draft needs to explain what 
> determines how much transport block capacity each flow is given, and 
> how they can influence it. In 3G, I believe that the transport layer 
> could ask the radio resource controller to change radio resource 
> allocation. Is this the same in 4G & 5G? Does the backlog of the 
> buffer into the RLC layer have any automatic influence over allocation 
> of available resources? I assume traffic class can also be used to 
> influence allocation.
>
> Gap#2: In a 3G stack, the RLC layer multiplexes all the active 
> application streams into MAC layer transport block by dicing up 
> buffered packets and packing the pieces into each transport block. 
> Then, at the other end of the link, each piece has to be held until 
> the rest of the packet arrives. This sacrifices delay for bandwidth 
> efficiency. Is this still done in 4G and 5G? If so, it would be good 
> to write about mux delay in the draft too.
>
> *2.3 MAC layer
> *
> Wouldn't it be better to set a limit on the /time/ spent 
> retransmitting, so it would automatically give up with less 
> retransmissions for longer transmission distances?
> [IJ] Possible, 4G specifies a limit to the number of retransmissions, 
> in practice it can be translated to time as each retransmission is 8ms 
> later.
>
I didn't realise it was independent of link-RTT. Then the more basic 
question is why is it independent of link-RTT? Surely on a short link it 
can assume that it needs to retransmit more quickly than on a long link?

You can see that my question was assuming that retransmission happened 
more often on a short link, then I was saying it should still be allowed 
the same amount of time to keep retrying (so on a short link it would be 
allowed to retransmit more times before giving up, but the max delay 
would be the same).

8ms! I was assuming it would be rather much less than that. Surely a 
link-layer ACK could never take that long if the frame was going to get 
through.
> *2.3.2 Uplink scheduling
> *
> Seems like a bad omission. Is there a reason? If there are less bytes 
> queued than the base station's grant, surely it would be useful for 
> the base station to know, so that it can grant more to something else 
> while the previous short transmission is completing.
> You say that when HARQ breaks up a sequence of ACKs, it can also 
> trigger "coalescing issues". I can understand that it could cause ACK 
> compression, but are you implying something coalesces the ACKs or the 
> subsequent data? What does the coalescing? Similarly, in the last para 
> of 2.3.2 you say "Reduced ACKs can unfortunately cause coalescing." 
> and again I'm not sure what is coalescing what here.
>
> [IJ] What I can see in simulations is that the ACK compression leads 
> to coalescing which further can increase jitter.
>
I still don't understand. The question was "What is coalescing what?"
>
>
> There could be a positive side to the interaction between ACK clocking 
> and link aggregation, at least for long-running flows. This has been 
> observed in 802.11n WiFi where the buffering and aggregation of 
> multiple ACKs into one transmission grant tend to lead the sender to 
> bunch multiple data segments so that they all nicely fill one 
> transmission grant in the other direction [Showail14]. As long as the 
> transmission grant does not wait for more data (which unfortunately I 
> don't think is the case in 3G/4G/5G), this could be good for bandwidth 
> utilisation of elephants and latency of mice, including when they are 
> mixed.
>
> [Showail14] Showail, A., Jamshaid, K. & Shihada, B., "WQM: An 
> Aggregation-aware Queue Management Scheme for IEEE 802.11N Based 
> Networks," In: Proceedings of the 2014 ACM SIGCOMM Workshop on 
> Capacity Sharing Workshop CSWS '14 pp.15-20 ACM (2014)
> [IJ] Interesting ,will look at it, not sure though that it will work 
> that well with 4G (cant say anything about 5G), in 4G the transmission 
> grants can vary quite a lot over short time spans.
>
>
> *3.  4G and 5G evolution
> *
> *4.  Requirements for improved performance
> *
> s/bufferbloat/buffer/
>
> *5.  Congestion control examples
> *
> Regarding Hybrid Cubic using OWD measurements, I have seen a paper 
> (under submission) about getting delay-based congestion controls (e.g. 
> LEDBAT or CAIA Delay Gradient) to interwork with AQMs with various low 
> delay targets. It's not in the context of cellular networks, but I'll 
> try to remember to point it out once it gets published.
>
> *5.1 HyStartRestart
> *
> In the new code, it would be useful to explain why the second test 
> before doubling ssthresh, ie, the test for how long it has been since 
> the last hyrestart. Also, in the header definitions, if you intend 
> there to be constraints on the relation between N_RTT_HYRESTART and 
> N_RTT_LOW (e.g.  N_RTT_HYRESTART > N_RTT_LOW) you ought to say, 
> because people will try other values.
>
> Rather than the rather arbitrary wait before an arbitrary doubling of 
> ssthresh, wouldn't it be better to continually increase ssthresh (not 
> necessarily linearly) the longer the RTT has remained only just higher 
> than the min RTT?
>
> (BTW, surely the main problem with this modification to HyStartRestart 
> is that is requires a number of round trips to reduce delay - a 
> sort-of oxymoron.)
>
> [IJ] Yes, have tried some more experiments with this algorithm. And I 
> am getting more hesitant. The problem is that when I add more noise 
> (modeling of small objects traffic) into the system simulation, then I 
> also get more delay jitter and the delay detection algorithm in 
> HyStart is notoriously sensitive to delay jitter.
>
>
>
> *5.2.  Hybrid Cubic*
>
> Recently on tcpm I sent you a pointer to the chirping paper Mirja & I 
> did, which also made this assumption. Partly prompted by that, Richard 
> Scheffenegger tried to get people interested in standardising use of 
> the timestamp option on a SYN to negotiate stuff like timer 
> resolution. I don't think it went anywhere. Do you think we ought to 
> revitalise that? You mention inference - have you tried that - are 
> there cases where it doesn't work?
>
> [IJ] Yes, actually wondered what happened to Richards draft. I only 
> briefly tried out the inference algorithm with a TCP LEDBAT 
> implementation 
> (https://github.com/silviov/TCP-LEDBAT/blob/master/src/tcp_ledbat.c). 
> I recall that it took a while to converge. Also I am not sure here. Is 
> Hz in e.g a linux stack fixed. ?
>
I believe so. But I don't know about embedded systems etc.

Cheers

Bob
>     From:[email-masked] <mailto:[email-masked]>  [mailto:[email-masked]]
>
>     URL:https://www.ietf.org/internet-drafts/draft-johansson-cc-for-4g-5g-00.txt
>
>     Status:https://datatracker.ietf.org/doc/draft-johansson-cc-for-4g-5g/
>
>     Htmlized:https://tools.ietf.org/html/draft-johansson-cc-for-4g-5g-00
>
>     Abstract:
>
> Bob Briscoehttp://bobbriscoe.net/

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 5 Nov 2015 14:12:37 +0900

I have a question re: how much buffer or backlog is needed to prevent the
schedular from reducing the rate or even increase the rate. I think this
key why cubic or bufferbloat cc wins bc other queue controlling cc would
lose out.

For example if the scheduler needs 1mb backlog to maintain at 40mbps. A
dctcp or vegas keeping the queue well below will get say 5mbps.

The draft mentions about this but the advise is a bit vague for cc designer.
On Nov 5, 2015 7:27 AM, "Bob Briscoe" <[email-masked]> wrote:

> *From:* Bob Briscoe [mailto:[email-masked] <[email-masked]>]
> As promised, here's my comments. They are a mix of suggested improvements
> to the draft, suggested improvements to 5G, or requests for clarification.
> I also agree with all Kevin Smith's comments, so I didn't repeat any.
> *Context questions. *Q1. Is this exercise treating the 5G specs solely in
> read-only mode? Or, if some useful design suggestions arise, is there any
> chance that the 3GPP might take them on board, given 5G is not fully baked
> yet?
>
> [IJ] Guess it depends, need to say that I probably the small cog in the
> wheel. I would say that a lot of the standards work in IETF has been picked
> up by 3GPP and, I cannot say what will happen in the 5G context.
>
> formally ask the IETF for comments on its ideas for each new generation of
> radio? Given the layers above can be thought of as the 'customers' of the
> 3GPP, and these 'customer' layers are pretty much all maintained by the
> IETF, it would seem essential (and courteous) for the 3GPP to ask for
> customer comment, not least to co-ordinate with whatever future plans the
> IETF might have for the higher layers.
>
> [IJ] What you say makes much sense. I guess there is some natural leakage
> between 3GPP and IETF as many of the IETFers are also active in 3GPP
> standardization. I don’t have full insight in the formal relation between
> 3GPP and IETF in these areas, not sure if that need to be better, but if
> that is the case, then it is probably a question for the area directors.
> Need to say that even though the formal channels are scarce it does
> probably not mean that 3GPP ignores IETF or what happens above IP in
> general. For instance the recent work around TCP RACK raises the question
> if it is necessary that LTE default radio bearers ensure in order delivery.
>
> radio people and L4, which I would imagine is as much a problem within 3GPP
> as between 3GPP and IETF.
>
>
>
> Q2. What are your plans for this draft? Are you asking for adoption? And
> if so, in which WG or RG? ICCRG looks the most appropriate, given the
> charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are scoped on
> just a subset of your doc.
>
> [IJ] First objective is to get an increased interest in congestion control
> algorithm work in the IETF. Traditionally this kind of work seem to be more
> in the terms of conference papers in e.g. Sigcomm, the benefit with having
> such work in IETF is that input from people with a wider expertise area
> (including 3GPP experitise) can give algorithm proposals that work well for
> a wider set of access types. The work around Cubic is a first good step as
> is the work around DCTCP. What I wonder is if it feasible to take this a
> bit further ? I would too believe that the current doc fits better in
> ICCRG.  The algorithm examples are just.. examples. More fruitful work is
> probably to document different access types and from there derive a set of
> best current practices for congestion control algorithms
>
> I'm sure the iccrg chairs would like this to happen. And initiatives like
> yours to come with proposals to propose specific code to take account of L2
> I think generate the right level of dialogue.
>
>
>
>
> *Section by Section Review *
>
> *All sections *
> The document generally talks about the LTE protocol stack as if data is
> travelling down it. Are you implying that most sources of buffering delay
> are at the sending end, and the feedback end is pretty unencumbered by
> delays? I think it would be useful to explicitly say something about this
> (whether true or not), to explain why only one direction of travel is
> discussed in the doc.
>
> affect the ACKs for downlink data traffic. That said, the difference in
> DL/UL traffic is today something like 10:1 and that is probably one reason
> why one can get the feeling that the draft is slanted towards downlink data
> traffic, this may however change with time as more machine type traffic may
> enter the system.
>
> * downlink direction: are there any buffering concerns coming up the stack
> at the UE, or on the feedback back-channel from the UE?
> For instance, in the UE, perhaps one app blocking the read of another due
> to the way {3-5}G multiplexes different streams into the same PDCP frames?
>
>
>
>
> *2.1 PDCP layer *
> I think this means "in the order sent into the link (PDCP) layer". This
> ought to be clarified, otherwise, when writing in the context of a layer 4
> audience, it might be assumed this means in the order sent by the original
> transport layer source.
>
> possible"? (which in turn implies keeping queuing delay as low as possible
> assuming you cannot influence the physical path.) Or were you trying to
> make a more subtle point? - if so, it was lost on me.
>
> Eh? The PDCP section just said it did ordering. And the rest of the RLC
> section talks exclusively about buffering delays necessary while filling
> transport blocks. If ordering is the role of both layers, surely one will
> have nothing to do once the other has got everything in order?
>
> Nit: Given 'transport' means e2e for the audience of this draft, pls use
> bearer or somehow disambiguate this word.
>
> [IJ] OK
>
>
>
>
> Gap#1: In the draft, it seems like transport blocks arrive by magic, to be
> filled by data buffered in the RLC layer. To understand the delay
> compromises here, I think the draft needs to explain what determines how
> much transport block capacity each flow is given, and how they can
> influence it. In 3G, I believe that the transport layer could ask the radio
> resource controller to change radio resource allocation. Is this the same
> in 4G & 5G? Does the backlog of the buffer into the RLC layer have any
> automatic influence over allocation of available resources? I assume
> traffic class can also be used to influence allocation.
>
> Gap#2: In a 3G stack, the RLC layer multiplexes all the active application
> streams into MAC layer transport block by dicing up buffered packets and
> packing the pieces into each transport block. Then, at the other end of
> the link, each piece has to be held until the rest of the packet arrives.
> This sacrifices delay for bandwidth efficiency. Is this still done in 4G
> and 5G? If so, it would be good to write about mux delay in the draft too.
>
> *2.3 MAC layer *
> Wouldn't it be better to set a limit on the *time* spent retransmitting,
> so it would automatically give up with less retransmissions for longer
> transmission distances?
> [IJ] Possible, 4G specifies a limit to the number of retransmissions, in
> practice it can be translated to time as each retransmission is 8ms later.
>
> allowed to retransmit more times before giving up, but the max delay would
> be the same).
>
> *2.3.2 Uplink scheduling *
> queued than the base station's grant, surely it would be useful for the
> base station to know, so that it can grant more to something else while the
> previous short transmission is completing.
> You say that when HARQ breaks up a sequence of ACKs, it can also trigger
> "coalescing issues". I can understand that it could cause ACK
> subsequent data? What does the coalescing? Similarly, in the last para of
> 2.3.2 you say "Reduced ACKs can unfortunately cause coalescing." and again
> I'm not sure what is coalescing what here.
>
> [IJ] What I can see in simulations is that the ACK compression leads to
> coalescing which further can increase jitter.
>
> There could be a positive side to the interaction between ACK clocking and
> link aggregation, at least for long-running flows. This has been observed
> in 802.11n WiFi where the buffering and aggregation of multiple ACKs into
> one transmission grant tend to lead the sender to bunch multiple data
> segments so that they all nicely fill one transmission grant in the other
> direction [Showail14]. As long as the transmission grant does not wait
> for more data (which unfortunately I don't think is the case in 3G/4G/5G),
> this could be good for bandwidth utilisation of elephants and latency of
> mice, including when they are mixed.
>
> Aggregation-aware Queue Management Scheme for IEEE 802.11N Based Networks,"
> In: Proceedings of the 2014 ACM SIGCOMM Workshop on Capacity Sharing
> Workshop CSWS '14 pp.15-20 ACM (2014)
> [IJ] Interesting ,will look at it, not sure though that it will work that
> well with 4G (cant say anything about 5G), in 4G the transmission grants
> can vary quite a lot over short time spans.
>
>
>
>
>
> *3.  4G and 5G evolution *
> *4.  Requirements for improved performance *
> s/bufferbloat/buffer/
>
>
> *5.  Congestion control examples *
> Regarding Hybrid Cubic using OWD measurements, I have seen a paper (under
> submission) about getting delay-based congestion controls (e.g. LEDBAT or
> CAIA Delay Gradient) to interwork with AQMs with various low delay targets.
> It's not in the context of cellular networks, but I'll try to remember to
> point it out once it gets published.
>
> *5.1 HyStartRestart *
> In the new code, it would be useful to explain why the second test before
> doubling ssthresh, ie, the test for how long it has been since the last
> hyrestart. Also, in the header definitions, if you intend there to be
> constraints on the relation between N_RTT_HYRESTART and N_RTT_LOW (e.g.
> N_RTT_HYRESTART > N_RTT_LOW) you ought to say, because people will try
> other values.
>
> necessarily linearly) the longer the RTT has remained only just higher than
> the min RTT?
>
> (BTW, surely the main problem with this modification to HyStartRestart is
> that is requires a number of round trips to reduce delay - a sort-of
> oxymoron.)
>
> [IJ] Yes, have tried some more experiments with this algorithm. And I am
> getting more hesitant. The problem is that when I add more noise (modeling
> of small objects traffic) into the system simulation, then I also get more
> delay jitter and the delay detection algorithm in HyStart is notoriously
> sensitive to delay jitter.
>
>
>
> *5.2.  Hybrid Cubic*
>
> Recently on tcpm I sent you a pointer to the chirping paper Mirja & I did,
> which also made this assumption. Partly prompted by that, Richard
> Scheffenegger tried to get people interested in standardising use of the
> timestamp option on a SYN to negotiate stuff like timer resolution. I don't
> think it went anywhere. Do you think we ought to revitalise that? You
> mention inference - have you tried that - are there cases where it doesn't
> work?
>
> briefly tried out the inference algorithm with a TCP LEDBAT implementation (
> https://github.com/silviov/TCP-LEDBAT/blob/master/src/tcp_ledbat.c). I
> recall that it took a while to converge. Also I am not sure here. Is Hz in
> e.g a linux stack fixed. ?
>
> From: [email-masked] [mailto:[email-masked] <[email-masked]>]
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>
>

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 5 Nov 2015 14:26:40 +0900

I also support this doc to be adopted by iccrg or tsvwg

---

From: David Ros <[email-masked]>
Date: Thu, 5 Nov 2015 19:13:08 +0900

Ingemar,

We believe this is a useful document, agree that it's a good match for ICCRG, and would be happy with ICCRG being the home of this draft.

Thanks,

Michael & David (as ICCRG chairs)

On 04 Nov 2015, at 22:27, Ingemar Johansson S <[email-masked]> wrote:

> From: [email-masked] [mailto:[email-masked]]

---

From: "Black, David" <[email-masked]>
Date: Thu, 5 Nov 2015 22:11:57 +0000

As a tsvwg chair, I concur that ICCRG would be a good initial home for this draft.

Thanks,
--David

From: tsvwg [mailto:[email-masked]] On Behalf Of David Ros
Sent: Thursday, November 05, 2015 5:13 AM
To: Ingemar Johansson S
Cc: [email-masked]; [email-masked]; [email-masked]; Michael Welzl
Subject: Re: [tsvwg] [tcpm] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

Ingemar,

Thanks,

On 04 Nov 2015, at 22:27, Ingemar Johansson S <[email-masked]<mailto:[email-masked]>> wrote:

Hi

/Ingemar

From: Bob Briscoe [mailto:[email-masked]]
Sent: den 29 oktober 2015 18:54
To: Ingemar Johansson S; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [tsvwg] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

Ingemar,

Context questions.

It's great that you have taken this initiative. Does the 3GPP ever formally ask the IETF for comments on its ideas for each new generation of radio? Given the layers above can be thought of as the 'customers' of the 3GPP, and these 'customer' layers are pretty much all maintained by the IETF, it would seem essential (and courteous) for the 3GPP to ask for customer comment, not least to co-ordinate with whatever future plans the IETF might have for the higher layers.
[IJ] What you say makes much sense. I guess there is some natural leakage between 3GPP and IETF as many of the IETFers are also active in 3GPP standardization. I don't have full insight in the formal relation between 3GPP and IETF in these areas, not sure if that need to be better, but if that is the case, then it is probably a question for the area directors. Need to say that even though the formal channels are scarce it does probably not mean that 3GPP ignores IETF or what happens above IP in general. For instance the recent work around TCP RACK raises the question if it is necessary that LTE default radio bearers ensure in order delivery.

All sections

2.1 PDCP layer

   Reliable delivery at handover may however be turned off or is simply not implemented,
It would be useful if you could give a feel for roughly how often (in %) these cases hold.
[IJ] I don't believe that it is possible to give a figure as this is vendor specific.

2.2 RLC layer

2.3 MAC layer

2.3.2 Uplink scheduling

3.  4G and 5G evolution

s/bufferbloat/buffer/

5.1 HyStartRestart

5.2.  Hybrid Cubic

Bob

Hi

/Ingemar

-----Original Message-----

To: Ingemar Johansson S

Revision:       00

Pages:          14

Abstract:

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org/>.

The IETF Secretariat

--

Bob Briscoe                               http://bobbriscoe.net/
_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Michael Welzl <[email-masked]>
Date: Fri, 6 Nov 2015 08:54:25 +0100

[I'm dropping the TCPM and TSVWG lists now]

.... it doesn't need to be initial only; we can also publish RFCs, as Informational or Experimental, carrying an IRTF boilerplate.

Cheers,
Michael

> On 05 Nov 2015, at 23:11, Black, David <[email-masked]> wrote:
>

---

From: "Black, David" <[email-masked]>
Date: Sat, 7 Nov 2015 08:00:03 +0000

Hi Michael,

No problem here - I'm just noticing that actually putting any congestion
control algorithm changes into IETF protocols usually involves a draft that
is at least jointly handled with the WG(s) responsible for the protocol(s),
e.g., TCPM, TSVWG.

ICCRG publication of this draft as a statement of principles and/or direction
is fine, but specific transport congestion control changes (even Experimental,
IMHO) SHOULD be worked out with the involvement of the responsible IETF WG,
(e.g., for SCTP, that would be TSVWG).  That said, I believe that ICCRG will
be the most productive discussion forum for this draft, and hence I strongly
support ICCRG taking it up.

Thanks,
--David

> From: Michael Welzl [mailto:[email-masked]]
> Sent: Friday, November 06, 2015 2:54 AM
> To: Black, David
> Cc: David Ros; Ingemar Johansson S; [email-masked]
> .... it doesn't need to be initial only; we can also publish RFCs, as
> Informational or Experimental, carrying an IRTF boilerplate.
> 
> Cheers,
> Michael
> 
> 
> > We believe this is a useful document, agree that it's a good match for
> ICCRG, and would be happy with ICCRG being the home of this draft.
> >
> > Thanks,
> >
> > On 04 Nov 2015, at 22:27, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> >
> > Hi
> >
> > As promised, here's my comments. They are a mix of suggested improvements to
> the draft, suggested improvements to 5G, or requests for clarification. I also
> agree with all Kevin Smith's comments, so I didn't repeat any.
> > Q1. Is this exercise treating the 5G specs solely in read-only mode? Or, if
> some useful design suggestions arise, is there any chance that the 3GPP might
> take them on board, given 5G is not fully baked yet?
> >
> wheel. I would say that a lot of the standards work in IETF has been picked up
> by 3GPP and, I cannot say what will happen in the 5G context.
> >
> > It's great that you have taken this initiative. Does the 3GPP ever formally
> ask the IETF for comments on its ideas for each new generation of radio? Given
> the layers above can be thought of as the 'customers' of the 3GPP, and these
> 'customer' layers are pretty much all maintained by the IETF, it would seem
> essential (and courteous) for the 3GPP to ask for customer comment, not least
> to co-ordinate with whatever future plans the IETF might have for the higher
> layers.
> >
> standardization. I don’t have full insight in the formal relation between 3GPP
> and IETF in these areas, not sure if that need to be better, but if that is
> the case, then it is probably a question for the area directors. Need to say
> that even though the formal channels are scarce it does probably not mean that
> 3GPP ignores IETF or what happens above IP in general. For instance the recent
> work around TCP RACK raises the question if it is necessary that LTE default
> radio bearers ensure in order delivery.
> >
> >
> >
> > Q2. What are your plans for this draft? Are you asking for adoption? And if
> so, in which WG or RG? ICCRG looks the most appropriate, given the charters of
> all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are scoped on just a subset of
> your doc.
> >
> algorithm work in the IETF. Traditionally this kind of work seem to be more in
> the terms of conference papers in e.g. Sigcomm, the benefit with having such
> work in IETF is that input from people with a wider expertise area (including
> 3GPP experitise) can give algorithm proposals that work well for a wider set
> of access types. The work around Cubic is a first good step as is the work
> around DCTCP. What I wonder is if it feasible to take this a bit further ? I
> would too believe that the current doc fits better in ICCRG.  The algorithm
> examples are just.. examples. More fruitful work is probably to document
> different access types and from there derive a set of best current practices
> for congestion control algorithms
> >
> >
> >
> travelling down it. Are you implying that most sources of buffering delay are
> at the sending end, and the feedback end is pretty unencumbered by delays? I
> think it would be useful to explicitly say something about this (whether true
> or not), to explain why only one direction of travel is discussed in the doc.
> >
> > [IJ] The feedback path is encumbered by delays, it is perhaps not that clear
> but section 2.3.2 implies that delay (variation) in uplink will affect the
> ACKs for downlink data traffic. That said, the difference in DL/UL traffic is
> today something like 10:1 and that is probably one reason why one can get the
> feeling that the draft is slanted towards downlink data traffic, this may
> however change with time as more machine type traffic may enter the system.
> >
> >
> >
> > 2.1 PDCP layer
> >
> assuming you cannot influence the physical path.) Or were you trying to make a
> more subtle point? - if so, it was lost on me.
> >
> transport blocks. If ordering is the role of both layers, surely one will have
> nothing to do once the other has got everything in order?
> >
> filled by data buffered in the RLC layer. To understand the delay compromises
> here, I think the draft needs to explain what determines how much transport
> block capacity each flow is given, and how they can influence it. In 3G, I
> believe that the transport layer could ask the radio resource controller to
> change radio resource allocation. Is this the same in 4G & 5G? Does the
> backlog of the buffer into the RLC layer have any automatic influence over
> allocation of available resources? I assume traffic class can also be used to
> influence allocation.
> >
> packing the pieces into each transport block. Then, at the other end of the
> link, each piece has to be held until the rest of the packet arrives. This
> sacrifices delay for bandwidth efficiency. Is this still done in 4G and 5G? If
> so, it would be good to write about mux delay in the draft too.
> >
> > Wouldn't it be better to set a limit on the time spent retransmitting, so it
> would automatically give up with less retransmissions for longer transmission
> distances?
> > Seems like a bad omission. Is there a reason? If there are less bytes queued
> than the base station's grant, surely it would be useful for the base station
> to know, so that it can grant more to something else while the previous short
> transmission is completing.
> "coalescing issues". I can understand that it could cause ACK compression, but
> are you implying something coalesces the ACKs or the subsequent data? What
> does the coalescing? Similarly, in the last para of 2.3.2 you say "Reduced
> ACKs can unfortunately cause coalescing." and again I'm not sure what is
> coalescing what here.
> >
> link aggregation, at least for long-running flows. This has been observed in
> 802.11n WiFi where the buffering and aggregation of multiple ACKs into one
> transmission grant tend to lead the sender to bunch multiple data segments so
> that they all nicely fill one transmission grant in the other direction
> [Showail14]. As long as the transmission grant does not wait for more data
> (which unfortunately I don't think is the case in 3G/4G/5G), this could be
> good for bandwidth utilisation of elephants and latency of mice, including
> when they are mixed.
> >
> > [Showail14] Showail, A., Jamshaid, K. & Shihada, B., "WQM: An Aggregation-
> aware Queue Management Scheme for IEEE 802.11N Based Networks," In:
> Proceedings of the 2014 ACM SIGCOMM Workshop on Capacity Sharing Workshop CSWS
> '14 pp.15-20 ACM (2014)
> well with 4G (cant say anything about 5G), in 4G the transmission grants can
> vary quite a lot over short time spans.
> >
> >
> >
> >
> > 3.  4G and 5G evolution
> >
> submission) about getting delay-based congestion controls (e.g. LEDBAT or CAIA
> Delay Gradient) to interwork with AQMs with various low delay targets. It's
> not in the context of cellular networks, but I'll try to remember to point it
> out once it gets published.
> >
> N_RTT_HYRESTART > N_RTT_LOW) you ought to say, because people will try other
> values.
> >
> getting more hesitant. The problem is that when I add more noise (modeling of
> small objects traffic) into the system simulation, then I also get more delay
> jitter and the delay detection algorithm in HyStart is notoriously sensitive
> to delay jitter.
> >
> >
> >
> > 5.2.  Hybrid Cubic
> >
> think it went anywhere. Do you think we ought to revitalise that? You mention
> inference - have you tried that - are there cases where it doesn't work?
> >
> > [IJ] Yes, actually wondered what happened to Richards draft. I only briefly
> tried out the inference algorithm with a TCP LEDBAT implementation
> (https://github.com/silviov/TCP-LEDBAT/blob/master/src/tcp_ledbat.c). I recall
> that it took a while to converge. Also I am not sure here. Is Hz in e.g a
> linux stack fixed. ?
> >
> >
> >
> >
> > A new IETF draft is submitted in response to general discussion that for
> instance TCP Cubic is not an appropriate congestion control algorithm for LTE.
> > The draft briefly outlines typical 4G access behavior on the PDCP, RLC and
> MAC  layers that can have an impact on transport protocol performance. The
> draft also outlines two relatively simple modifications to the Cubic
> congestion control.
> >
> > /Ingemar
> >
> > Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> >
> > The IETF Secretariat
> >
> >
> >
> >
> >
> > --

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Nov 2015 18:47:10 +0000

Hi

Sorry for the delay. It is OK with me to camp this work in ICCRG, the example algorithms was put in the draft more as just examples but I can accept that these are removed/put somewhere else to make the draft more clean and targeted more towards a principles/guidance draft.

One potential issue with the actual congestion control algorithms is that these are in general quite generic and are in most cases implementable for TCP, SCTP or QUIC.  Possibly with some exceptions due to protocol constraints. 
How is does general work on congestion control algorithms fit in IETF or IRTF, does it fit at all ? Is TSVWG a good place for general congestion control work ?

/Ingemar 

> From: Black, David [mailto:[email-masked]]
> Sent: den 7 november 2015 09:00
> To: Michael Welzl
> Cc: David Ros; Ingemar Johansson S; [email-masked]; Black, David;
> [email-masked]
> Subject: RE: [tsvwg] [tcpm] FW: New Version Notification for draft-
> johansson-cc-for-4g-5g-00.txt
> 
> Hi Michael,
> 
> control algorithm changes into IETF protocols usually involves a draft that is at
> least jointly handled with the WG(s) responsible for the protocol(s), e.g.,
> TCPM, TSVWG.
> 
> ICCRG publication of this draft as a statement of principles and/or direction is
> fine, but specific transport congestion control changes (even Experimental,
> IMHO) SHOULD be worked out with the involvement of the responsible IETF
> WG, (e.g., for SCTP, that would be TSVWG).  That said, I believe that ICCRG
> will be the most productive discussion forum for this draft, and hence I
> strongly support ICCRG taking it up.
> 
> Thanks,
> --David
> 
> > Subject: Re: [tsvwg] [tcpm] FW: New Version Notification for
> > draft-johansson- cc-for-4g-5g-00.txt
> >
> > > As a tsvwg chair, I concur that ICCRG would be a good initial home
> > > for this
> > draft.
> > >
> > > Thanks,
> > > --David
> > >
> > > Subject: Re: [tsvwg] FW: New Version Notification for
> > > draft-johansson-cc-
> > for-4g-5g-00.txt
> > >
> > > Ingemar,
> > >
> > > improvements to
> > the draft, suggested improvements to 5G, or requests for
> > clarification. I also agree with all Kevin Smith's comments, so I didn't repeat
> any.
> > > [Sorry for delay - I had to find time to re-type after losing my
> > > review
> > during a power outage]
> > >
> > > Context questions.
> > some useful design suggestions arise, is there any chance that the
> > 3GPP might take them on board, given 5G is not fully baked yet?
> > >
> > > the
> > wheel. I would say that a lot of the standards work in IETF has been
> > picked up by 3GPP and, I cannot say what will happen in the 5G context.
> > >
> > > formally
> > ask the IETF for comments on its ideas for each new generation of
> > radio? Given the layers above can be thought of as the 'customers' of
> > the 3GPP, and these 'customer' layers are pretty much all maintained
> > by the IETF, it would seem essential (and courteous) for the 3GPP to
> > ask for customer comment, not least to co-ordinate with whatever
> > future plans the IETF might have for the higher layers.
> > >
> > > leakage
> > standardization. I don’t have full insight in the formal relation
> > between 3GPP and IETF in these areas, not sure if that need to be
> > better, but if that is the case, then it is probably a question for
> > the area directors. Need to say that even though the formal channels
> > are scarce it does probably not mean that 3GPP ignores IETF or what
> > happens above IP in general. For instance the recent work around TCP
> > RACK raises the question if it is necessary that LTE default radio bearers
> ensure in order delivery.
> > >
> > >
> > >
> > > And if
> > so, in which WG or RG? ICCRG looks the most appropriate, given the
> > charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are scoped
> > on just a subset of your doc.
> > >
> > > control
> > algorithm work in the IETF. Traditionally this kind of work seem to be
> > more in the terms of conference papers in e.g. Sigcomm, the benefit
> > with having such work in IETF is that input from people with a wider
> > expertise area (including 3GPP experitise) can give algorithm
> > > is
> > travelling down it. Are you implying that most sources of buffering
> > but section 2.3.2 implies that delay (variation) in uplink will affect
> > the ACKs for downlink data traffic. That said, the difference in DL/UL
> > traffic is today something like 10:1 and that is probably one reason
> > why one can get the feeling that the draft is slanted towards downlink
> > data traffic, this may however change with time as more machine type
> traffic may enter the system.
> > >
> > >
> > >
> > > 2.1 PDCP layer
> > >
> > > This
> > ought to be clarified, otherwise, when writing in the context of a
> > layer 4 audience, it might be assumed this means in the order sent by
> > the original transport layer source.
> > >
> > > [IJ] Yes, you are right , reordering can still occur e.g. in the
> > > wireless
> > backhaul
> > >
> > >
> > >
> > >    Reliable delivery at handover may however be turned off or is
> > > simply not
> > implemented,
> > > It would be useful if you could give a feel for roughly how often
> > > (in %)
> > these cases hold.
> > >
> > > [IJ] I don’t believe that it is possible to give a figure as this is
> > > vendor
> > specific.
> > >
> > >
> > >
> > > 2.2 RLC layer
> > >
> > >    role ... is to ensure that packets are delivered in order up to
> > > the
> > higher layers
> > > RLC
> > section talks exclusively about buffering delays necessary while
> > surely one will have nothing to do once the other has got everything in
> order?
> > >
> > > [IJ] RLC handles ordering due to various MAC layer failures. PDCP
> > > handles
> > ordering  when handover occurs
> > >
> > >
> > >
> > > use
> > > to be
> > compromises here, I think the draft needs to explain what determines
> > how much transport block capacity each flow is given, and how they can
> > influence it. In 3G, I believe that the transport layer could ask the
> > radio resource controller to change radio resource allocation. Is this
> > the same in 4G & 5G? Does the backlog of the buffer into the RLC layer
> > have any automatic influence over allocation of available resources? I
> > assume traffic class can also be used to influence allocation.
> > >
> > > application
> > streams into MAC layer transport block by dicing up buffered packets
> > and packing the pieces into each transport block. Then, at the other
> > end of the link, each piece has to be held until the rest of the
> > packet arrives. This sacrifices delay for bandwidth efficiency. Is
> > this still done in 4G and 5G? If so, it would be good to write about mux
> delay in the draft too.
> > >
> > would automatically give up with less retransmissions for longer
> > transmission distances?
> > > s/only checks for the presence on download data only at regular
> > > intervals/  /only checks for the presence of download data at
> > > regular intervals/ [OK]
> > >
> > >
> > >
> > >
> > > 2.3.2 Uplink scheduling
> > >
> > >    The scheduling request does not indicate how many bytes there are
> > > in the
> > uplink queue.
> > than the base station's grant, surely it would be useful for the base
> > station to know, so that it can grant more to something else while the
> > > trigger
> > > to
> > link aggregation, at least for long-running flows. This has been
> > utilisation of elephants and latency of mice, including when they are mixed.
> > >
> > > Aggregation-
> > > that
> > well with 4G (cant say anything about 5G), in 4G the transmission
> > > (under
> > submission) about getting delay-based congestion controls (e.g. LEDBAT
> > or CAIA Delay Gradient) to interwork with AQMs with various low delay
> > targets. It's not in the context of cellular networks, but I'll try to
> > remember to point it out once it gets published.
> > >
> > > before
> > doubling ssthresh, ie, the test for how long it has been since the
> > last hyrestart. Also, in the header definitions, if you intend there
> > to be constraints on the relation between N_RTT_HYRESTART and
> N_RTT_LOW (e.g.
> > getting more hesitant. The problem is that when I add more noise
> > revitalise that? You mention inference - have you tried that - are there
> cases where it doesn't work?
> > >
> > > briefly
> > MAC  layers that can have an impact on transport protocol performance.
> > The draft also outlines two relatively simple modifications to the
> > Cubic congestion control.
> > >
> > > /Ingemar
> > >
> > > has been successfully submitted by Ingemar Johansson and posted to
> > > the IETF
> > repository.
> > >

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 10 Nov 2015 19:32:28 +0000

If a congestion control algorithm is used for TCP, the following sentence in the TCPM charter applies:

"In addition, TCPM may document alternative TCP congestion control algorithms that are known to be widely deployed, and that are considered safe for large-scale deployment in the Internet."

The emphasis here is *widely deployed*. Other discussion of congestion control IMHO perfectly fits into ICCRG.

Michael

> From: iccrg [mailto:[email-masked]] On Behalf Of Ingemar
> Johansson S
> Sent: Tuesday, November 10, 2015 7:47 PM
> To: Black, David; Michael Welzl
> Cc: David Ros; [email-masked]; Karen E. E. Nielsen;
> [email-masked]; Ingemar Johansson S
> Subject: Re: [iccrg] [tsvwg] [tcpm] FW: New Version Notification for
> Sorry for the delay. It is OK with me to camp this work in ICCRG, the
> example algorithms was put in the draft more as just examples but I can
> accept that these are removed/put somewhere else to make the draft more
> clean and targeted more towards a principles/guidance draft.
> 
> One potential issue with the actual congestion control algorithms is
> that these are in general quite generic and are in most cases
> implementable for TCP, SCTP or QUIC.  Possibly with some exceptions due
> to protocol constraints.
> How is does general work on congestion control algorithms fit in IETF
> or IRTF, does it fit at all ? Is TSVWG a good place for general
> congestion control work ?
> 
> /Ingemar
> 
> > > > We believe this is a useful document, agree that it's a good
> match
> > > > for
> > > > Q1. Is this exercise treating the 5G specs solely in read-only
> mode?
> > > > Or, if
> > > >    keep the amount of data in flight as small as possible,
> without
> > > sacrificing throughput.
> > > > Is this just a rather convoluted way of saying "keep the RTT as
> low
> > > > as
> > > > [IJ] It is a design choice. A buffer status report is more bulky,
> a
> > > scheduling request is just one bit.
> > > >
> > > >
> > > >
> > > >
> > > > Rather than the rather arbitrary wait before an arbitrary
> doubling
> > > > of
> > > > [IJ] Yes, have tried some more experiments with this algorithm.
> And
> > > > I am
> > > > Recently on tcpm I sent you a pointer to the chirping paper Mirja
> &
> > > > I did,
> > > > A new IETF draft is submitted in response to general discussion
> that
> > > > for
> > > instance TCP Cubic is not an appropriate congestion control
> algorithm for
> > LTE.
> > > > The draft briefly outlines typical 4G access behavior on the
> PDCP,
> > > > RLC and
> > > > URL:            https://www.ietf.org/internet-drafts/draft-
> johansson-cc-for-
> > > 4g-5g-00.txt
> > > > Status:         https://datatracker.ietf.org/doc/draft-johansson-
> cc-for-4g-
> > > 5g/
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: "Black, David" <[email-masked]>
Date: Tue, 10 Nov 2015 20:02:08 +0000

> Sorry for the delay. It is OK with me to camp this work in ICCRG, the example
> algorithms was put in the draft more as just examples but I can accept that
> these are removed/put somewhere else to make the draft more clean and targeted
> more towards a principles/guidance draft.

The general algorithms would be fine in an ICCRG draft, IMHO.

> One potential issue with the actual congestion control algorithms is that
> these are in general quite generic and are in most cases implementable for
> TCP, SCTP or QUIC.  Possibly with some exceptions due to protocol constraints.

I think that's actually a good split (generic vs. protocol-specific).

> How is does general work on congestion control algorithms fit in IETF or IRTF,
> does it fit at all ? Is TSVWG a good place for general congestion control work
> ?

It's a mix.  ICCRG is a good place for general congestion control work on
algorithms, etc., with protocol specifics handled by the IETF WG for each
protocol.  There are analogies elsewhere - e.g., CFRG 

TCPM handles TCP changes, and TSVWG handles SCTP changes; QUIC is not (currently)
an IETF protocol.  We rely on common participation of key people to keep these
consistent, and having a reference-able protocol-independent ICCRG output on how
the algorithm works would almost certainly help.

Thanks,
--David

> From: Ingemar Johansson S [mailto:[email-masked]]
> Sent: Tuesday, November 10, 2015 1:47 PM
> Cc: David Ros; [email-masked]; [email-masked]; Karen E. E. Nielsen;
> Ingemar Johansson S
> Subject: RE: [tsvwg] [tcpm] FW: New Version Notification for draft-johansson-
> cc-for-4g-5g-00.txt
> 
> Hi
> 
> Sorry for the delay. It is OK with me to camp this work in ICCRG, the example
> algorithms was put in the draft more as just examples but I can accept that
> these are removed/put somewhere else to make the draft more clean and targeted
> more towards a principles/guidance draft.
> 
> One potential issue with the actual congestion control algorithms is that
> these are in general quite generic and are in most cases implementable for
> TCP, SCTP or QUIC.  Possibly with some exceptions due to protocol constraints.
> How is does general work on congestion control algorithms fit in IETF or IRTF,
> does it fit at all ? Is TSVWG a good place for general congestion control work
> ?
> 
> /Ingemar
> 
> is at
> > > > URL:            https://www.ietf.org/internet-drafts/draft-johansson-cc-
> for-
> > > 4g-5g-00.txt
> > > > Status:         https://datatracker.ietf.org/doc/draft-johansson-cc-for-
> 4g-
> > > 5g/

---

From: Matt Mathis <[email-masked]>
Date: Tue, 10 Nov 2015 12:27:58 -0800

I want to remind people of a huge iceberg ahead: both the ATM industry and
the satellite industry came to the IETF requesting link technology specific
changes to TCP CC.  Both industries failed to get their wishes, which might
have had some impact on the life cycle of the respective technologies.

DCTCP was also rebuffed for it's narrow mission (e.g. Datacenters are not
the IETF's problem).  However, approximately the same problem/solution
rescoped to the entire Internet is getting traction as TCP Prague.

If you propose to try to put link technology specific hacks^H^H^H^H
features in TCP, you are very likely to hear objections from unexpected
places.

That said the Internet needs a globally acceptable answer to at least one
important question raised here:  For delay sensing congestion control, what
is the optimal target standing queue, and why?   My sense is that the
solution should be specified as a time (or perhaps bytes) as a function of
both data rate and intent (scavenger service?).   This question needs to be
answered in a uniform, transport protocol independent way (and eventually
in all end systems), where it will be applied to *all* link types.

This is not a trivial question: I can think of some cases where optimal for
LTE may be significantly suboptimal in other settings.  It may also be the
case that an IETF solution to this question will cause some of the LTE
algorithms and parameters to be tweaked.   This is sort of expected because
the current LTE controllers were designed in response to empirical
 measurements of the existing ad hoc behavior.  Some co-engineering might
improve both LTE and transport protocols.

This question needs to be explored in ICCRG, and if it converges the work
should migrate to TCPM.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy and
security as matters of life and death, because for some users, they are.

On Tue, Nov 10, 2015 at 11:32 AM, Scharf, Michael (Michael) <
[email-masked]> wrote:

> "In addition, TCPM may document alternative TCP congestion control
> algorithms that are known to be widely deployed, and that are considered
> safe for large-scale deployment in the Internet."
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 10 Nov 2015 20:29:01 +0000

Hi

You are right that the message is quite vague, the reason is that at least I don’t have an exact answer to that question.
An attempt at an exact rule is that there should be exactly as much data available as can be scheduled in the next scheduling event.
The problem is that it is not obvious when that scheduling event occurs (depends on scheduling algorithm) or how much data can be scheduled.
So, if one look at peak throughput alone it is true that it pays off to use CC algorithms that generate larger standing queues (Cubic is better than LEDBAT or Vegas). The price paid for this is of course that latency sensitive services can suffer when large standing queues are allowed.
In short, one need to compromise, if one take the example that WebRTC media traverses the same access, then it is likelky better to accept that the file transfer takes a while longer.

/Ingemar

From: Yuchung Cheng [mailto:[email-masked]]
Sent: den 5 november 2015 06:13
To: Bob Briscoe
Cc: [email-masked] Extensions; Ingemar Johansson S; [email-masked]; [email-masked]
Subject: Re: [tcpm] [tsvwg] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

I have a question re: how much buffer or backlog is needed to prevent the schedular from reducing the rate or even increase the rate. I think this key why cubic or bufferbloat cc wins bc other queue controlling cc would lose out.

For example if the scheduler needs 1mb backlog to maintain at 40mbps. A dctcp or vegas keeping the queue well below will get say 5mbps.

The draft mentions about this but the advise is a bit vague for cc designer.
On Nov 5, 2015 7:27 AM, "Bob Briscoe" <[email-masked]<mailto:[email-masked]>> wrote:
Ingemar,
On 04/11/15 13:27, Ingemar Johansson S wrote:
Hi

/Ingemar

Ingemar,

Context questions.

It's great that you have taken this initiative. Does the 3GPP ever formally ask the IETF for comments on its ideas for each new generation of radio? Given the layers above can be thought of as the 'customers' of the 3GPP, and these 'customer' layers are pretty much all maintained by the IETF, it would seem essential (and courteous) for the 3GPP to ask for customer comment, not least to co-ordinate with whatever future plans the IETF might have for the higher layers.
[IJ] What you say makes much sense. I guess there is some natural leakage between 3GPP and IETF as many of the IETFers are also active in 3GPP standardization. I don’t have full insight in the formal relation between 3GPP and IETF in these areas, not sure if that need to be better, but if that is the case, then it is probably a question for the area directors. Need to say that even though the formal channels are scarce it does probably not mean that 3GPP ignores IETF or what happens above IP in general. For instance the recent work around TCP RACK raises the question if it is necessary that LTE default radio bearers ensure in order delivery.
I suspect that 3GPP radio resource control people do not cross-fertilize much with the IETF.
So maybe it is as much a question of how to co-ordinate between these radio people and L4, which I would imagine is as much a problem within 3GPP as between 3GPP and IETF.

Q2. What are your plans for this draft? Are you asking for adoption? And if so, in which WG or RG? ICCRG looks the most appropriate, given the charters of all the IETF cc WGs (TSVWG, TCPM, RMCAT, etc) are scoped on just a subset of your doc.
[IJ] First objective is to get an increased interest in congestion control algorithm work in the IETF. Traditionally this kind of work seem to be more in the terms of conference papers in e.g. Sigcomm, the benefit with having such work in IETF is that input from people with a wider expertise area (including 3GPP experitise) can give algorithm proposals that work well for a wider set of access types. The work around Cubic is a first good step as is the work around DCTCP. What I wonder is if it feasible to take this a bit further ? I would too believe that the current doc fits better in ICCRG.  The algorithm examples are just.. examples. More fruitful work is probably to document different access types and from there derive a set of best current practices for congestion control algorithms
I'm sure the iccrg chairs would like this to happen. And initiatives like yours to come with proposals to propose specific code to take account of L2 I think generate the right level of dialogue.

All sections

The document generally talks about the LTE protocol stack as if data is travelling down it. Are you implying that most sources of buffering delay are at the sending end, and the feedback end is pretty unencumbered by delays? I think it would be useful to explicitly say something about this (whether true or not), to explain why only one direction of travel is discussed in the doc.
[IJ] The feedback path is encumbered by delays, it is perhaps not that clear but section 2.3.2 implies that delay (variation) in uplink will affect the ACKs for downlink data traffic. That said, the difference in DL/UL traffic is today something like 10:1 and that is probably one reason why one can get the feeling that the draft is slanted towards downlink data traffic, this may however change with time as more machine type traffic may enter the system.
I was thinking more in terms of up the stack vs. down the stack, not uplink/downlink. I.e. for both cases:
* downlink direction: are there any buffering concerns coming up the stack at the UE, or on the feedback back-channel from the UE?
* uplink direction: are there any buffering concerns coming up the stack at the eNodeB, or on the feedback back-channel from the eNodeB?

For instance, in the UE, perhaps one app blocking the read of another due to the way {3-5}G multiplexes different streams into the same PDCP frames?

2.1 PDCP layer

2.2 RLC layer

2.3 MAC layer

Wouldn't it be better to set a limit on the time spent retransmitting, so it would automatically give up with less retransmissions for longer transmission distances?
[IJ] Possible, 4G specifies a limit to the number of retransmissions, in practice it can be translated to time as each retransmission is 8ms later.
I didn't realise it was independent of link-RTT. Then the more basic question is why is it independent of link-RTT? Surely on a short link it can assume that it needs to retransmit more quickly than on a long link?

You can see that my question was assuming that retransmission happened more often on a short link, then I was saying it should still be allowed the same amount of time to keep retrying (so on a short link it would be allowed to retransmit more times before giving up, but the max delay would be the same).

8ms! I was assuming it would be rather much less than that. Surely a link-layer ACK could never take that long if the frame was going to get through.

2.3.2 Uplink scheduling

3.  4G and 5G evolution

s/bufferbloat/buffer/

5.1 HyStartRestart

5.2.  Hybrid Cubic

Cheers

Bob

Bob

Hi

/Ingemar

-----Original Message-----

To: Ingemar Johansson S

Revision:       00

Pages:          14

Abstract:

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

The IETF Secretariat

--

--

---

From: Michael Welzl <[email-masked]>
Date: Wed, 11 Nov 2015 10:55:19 +0100

> On 10 Nov 2015, at 21:27, Matt Mathis <[email-masked]> wrote:
> 
> I want to remind people of a huge iceberg ahead: both the ATM industry and the satellite industry came to the IETF requesting link technology specific changes to TCP CC.  Both industries failed to get their wishes, which might have had some impact on the life cycle of the respective technologies.
> 
> DCTCP was also rebuffed for it's narrow mission (e.g. Datacenters are not the IETF's problem).  However, approximately the same problem/solution rescoped to the entire Internet is getting traction as TCP Prague.
> 
> If you propose to try to put link technology specific hacks^H^H^H^H features in TCP, you are very likely to hear objections from unexpected places.

Like, from me!   :-)    More seriously, below ...

> That said the Internet needs a globally acceptable answer to at least one important question raised here:  For delay sensing congestion control, what is the optimal target standing queue, and why?   My sense is that the solution should be specified as a time (or perhaps bytes) as a function of both data rate and intent (scavenger service?).   This question needs to be answered in a uniform, transport protocol independent way (and eventually in all end systems), where it will be applied to *all* link types.
> 
> This is not a trivial question: I can think of some cases where optimal for LTE may be significantly suboptimal in other settings.  It may also be the case that an IETF solution to this question will cause some of the LTE algorithms and parameters to be tweaked.   This is sort of expected because the current LTE controllers were designed in response to empirical  measurements of the existing ad hoc behavior.  Some co-engineering might improve both LTE and transport protocols.
> 
> This question needs to be explored in ICCRG, and if it converges the work should migrate to TCPM.

So I think, as a first step, the next version of draft-johansson-cc-for-4g-5g should have a longer section on "Requirements for improved performance" where it is clarified which recommendations are only meant for transport connections that are known to *only* traverse the particular link layer being discussed, and which recommendations could be applied in a more general way (and what that would mean for the transport connection).

For example, "ACK reduction" is not without problems - see the long recent debate in the list of the AQM WG. However, "ECN support" is a more general recommendation that may not be less problematic for transport connections traversing multiple link layers. Yes, I know, ECN also has issues, but then there are already recommendations to cite....  recommending the use of ECN is perhaps less problematic than generally recommending to reduce the number of ACKs or to increase cwnd faster. Then again, cautioning against relying on precise RTT estimates is a much more general recommendation because it applies to any connection across a path that does include a 4G/5G link. Thus, categorizing these recommendations would really help IMO.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Wed, 11 Nov 2015 10:58:14 +0100

With ICCRG chair hat on,

> On 10 Nov 2015, at 21:02, Black, David <[email-masked]> wrote:
> 

I agree,

> I think that's actually a good split (generic vs. protocol-specific).

I agree,

> protocol.  There are analogies elsewhere - e.g., CFRG 

I agree,

> the algorithm works would almost certainly help.

I agree.  (well, that's not for me to agree on, so let's say: I can imagine that this is the case)

Cheers,
Michael

---

From: John Leslie <[email-masked]>
Date: Wed, 11 Nov 2015 12:02:16 -0500

Matt Mathis <[email-masked]> wrote:
>... 
> 
> That said the Internet needs a globally acceptable answer to at least
> one important question raised here:  For delay sensing congestion control,
> what is the optimal target standing queue, and why?

   The question, as phrased, has an almost trivial answer: zero is the
optimal standing queue size. Kathy Nichols and Van Jacobsen have defined
good queues and bad queues; and a standing queue is bad. It serves no-one
well except someone intent upon denial-of-service.

   Perhaps we should work on a better phrasing for this question?

> My sense is that the solution should be specified as a time (or perhaps
> bytes) as a function of both data rate and intent (scavenger service?).

   Indeed, for the question to be useful, it should be stated per-flow.
A pure "scavenger" service should aim for a slightly-negative standing
queue. Real-time services should aim for a zero standing queue. Web-
browsing will benefit from a positive standing queue (which nonetheless
can trend towards zero once the actual RTT is known). Et cetera.

> This question needs to be answered in a uniform, transport protocol
> independent way (and eventually in all end systems), where it will be
> applied to *all* link types.

   I don't believe transport-independent or link-independent are goals
we can achieve.

>... (mentions LTE)

   Again, I don't believe link-independence is a proper goal. We _might_
find that certain LTE features are _sometimes_ better disabled...

> This question needs to be explored in ICCRG, and if it converges the work

   With that, I agree!

--
John Leslie <[email-masked]>

---

From: Michael Welzl <[email-masked]>
Date: Thu, 12 Nov 2015 13:42:33 +0100

> On 11 Nov 2015, at 18:02, John Leslie <[email-masked]> wrote:
> 
> optimal standing queue size.

+1, and congestion control mechanisms can work without necessarily having to maintain a standing queue, e.g. by using delay gradients.
See CDG, for example: http://caia.swin.edu.au/cv/dahayes/content/networking2011-cdg-preprint.pdf

Cheers,
Michael

---

From: Mark Allman <[email-masked]>
Date: Thu, 12 Nov 2015 08:04:30 -0500

> optimal standing queue size.

Well, I would have given a different glib answer if I were going to
do such.  And, I guess I am because I think John is wrong ...

If there is nothing in the queue, you are not going to fully utilize
a link (because you'll have to rely on a very precise arrival
pattern that will never happen in practice).

So, my glib answer is "1".  If there is always one thing in the
queue then we always have something to transmit at the instant a
link has availability.

I think it is an important distinction and we should in fact not
shoot for "zero" when there is enough traffic in play to fully
utilize a link.

But, I took Matt's question---which I like---in a less pedantic
way.  I.e., where should we be aiming in terms of reasonable
standing queues?  Is it 1 packet?  Or, 10?  Or, 25msec?  Or, ...
That question I am not sure I have a good answer for.  And, if I do
then the followon "why?" gets me!

allman

--
http://www.icir.org/mallman/

---

From: John Leslie <[email-masked]>
Date: Thu, 12 Nov 2015 08:14:32 -0500

> See CDG, for example: http://caia.swin.edu.au/cv/dahayes/content/networking2011-cdg-preprint.pdf

   Ah, "delay gradient"... I shall read it carefully!

---

From: Dave Taht <[email-masked]>
Date: Thu, 12 Nov 2015 08:15:07 -0500

> So, my glib answer is "1".  If there is always one thing in the

My answer is 1, also. As is kathy/van's.

> then the followon "why?" gets me!

The numbers I consistently see e2e (e.g. tcp) folk dancing about is
100-150ms. I have seen CDG get to about 30ms, which was very
encouraging - except that it fails to compete with loss-based at any
amount of delay.

Given that we consistently get to less than 30ms with modern AQMs
(codel, pie) on the routers with near 0 if you add in fq (fq_codel,
cake) for sparse flows, and that using these aqm and fair queuing
systems affect both loss based and delay based ccs equally (and add in
the ability to use ecn marking), I have generally focused on fixing
the routers.

Perpetually citing this:

http://perso.telecom-paristech.fr/~drossi/paper/rossi14comnet-b.pdf

and hoping to add to this body of lit in the coming year.

I am glad to see the lte folk entering the fray here at ietf, and do
keep hoping we see some results from the various talks on aqm/fq and
data given to lte makers like alcatel/lucent and NEC in their next
generation of products.

---

From: Michael Welzl <[email-masked]>
Date: Thu, 12 Nov 2015 14:18:04 +0100

> On 12 Nov 2015, at 14:04, Mark Allman <[email-masked]> wrote:
> 
> 

+1 to that too and sorry for my imprecision in my own answer!   (indeed delay gradients need SOME signal, so you need SOMEthing, ideally "1")

> then the followon "why?" gets me!

So I think that you want some signal but react immediately as you get one. If your mechanism measures delay, we're talking in ms rather than packets anyway - and whether this gives you (on average) a standing queue of 10ms or 25ms or whatever will depend on mechanism dynamics (HOW it reacts to a signal of even only slight delay growth)... it's going to be a matter of trying to keep this delay as low as possible while ensuring that there's still enough of a signal to be able to react to something, and evaluating this broadly enough to see that it works reasonably well.

My point is: it seems to me to be a mechanism-dependent thing with no general mechanism-independent answer. I guess the only general answer is "as close to 0 as possible while still working well".

Cheers,
Michael

---

From: [email-masked]
Date: Thu, 12 Nov 2015 14:48:20 +0100

Hi,

Maybe 1 packet is the optimal queue size because it is value that can 
provide full bandwidth utilization while penalizing at minimum sensitive 
delay applications ?

Yours,

Dino Lopez

---

From: John Leslie <[email-masked]>
Date: Thu, 12 Nov 2015 09:07:37 -0500

Mark Allman <[email-masked]> wrote:
> [John Leslie wrote:]
>> {Matt Mathis <[email-masked]> wrote:]
> 

   (Please note: I suggested this question deserves to be better phrased.)

> do such.  And, I guess I am because I think John is wrong ...

   I've been wrong before...

> pattern that will never happen in practice).

   This presumes that "fully utilize" is a reasonable goal. I dissent.

> queue then we always have something to transmit at the instant a

   (I take it to mean you want "1 packet".)

   We have here "a failure t'communicate". :^(

   The ISPs of this world want to build least-cost networks. Thus many
of them think "fully utilize" to be a reasonable goal. IMHO, they are
wrong.

   There is always a lead-time on increasing link capacity. Thus, if
you aim for full utilization, you're dangerously close to management
saying, "Don't order an increase until you can show me full utilization
for N hours per day; and you end up with months of packet-drop before
the situation can be corrected.

   (Of course, we've designed the Internet to "work" in the presence
of substantial packet-drop: this means that folks can continue to
browse the web long after voice-over-IP has become objectionable.)

   Folks _can_, and do, seek alternatives once they experience "full"
utilization slowing them down. Alas, in the US we have regulators
limiting the alternatives. But that is not true everywhere!

   Folks who are dissatisfied tie up your support lines: leading to
them becoming _seriously_ dissatisfied when they get no useful support.

   But enough of the ISP view...

   On this list, we're supposed to be talking about research to make
the Internet better -- not how to make the ISP business more profitable.

   And, in fact, the difference between a target of one-packet standing
queue and zero-packet is probably not worth much argument.

> shoot for "zero" when there is enough traffic in play to fully

   That is a better statement of the issue... but it still falls short.
If your traffic is "greedy TCP", you _do_ want to fully utilize a link,
and the standing-queue needs to get fairly large before it actually
penalizes you. ("Fairly-large" here probably means 20% of RTT.)

   But if your traffic is VoIP, ten milliseconds can be really harmful.
I claim that "a better Internet" should support real-time conferencing;
so I'd rather concentrate our efforts on the VoIP side. (YMMV)

> standing queues?  Is it 1 packet?  Or, 10?  Or, 25msec?  Or, ...

   Obviously, opinions will differ. Thus, I see bikeshedding ahead!

> then the followon "why?" gets me!

   With a better question, I claim the "why" becomes easier.

   "Greedy TCP" aims for quick completion time. Please note: this says
rather too little about our standing-queue question. We can, of coure,
play the "ceteris paribus" game, but ceteris never _are_ paribus. Quick
completion time is in fact much better served by finding capacity
quickly than by bigger queues.

   VoIP aims for the network to become "invisible". (Pun acknowledged!)
Differences in RTT are our biggest enemy. Even packet loss we could
route-around with redundancy. IMHO, it is long-since settled that spare
capacity is the cheapest way to reduce jitter.

   (There are _many_ other use cases, which I won't go into here!)

---

From: Mark Allman <[email-masked]>
Date: Thu, 12 Nov 2015 09:27:58 -0500

>    This presumes that "fully utilize" is a reasonable goal. I dissent.

If you want a world that is persistently over-provisioned then,
well, OK.  But, I think in the real world that is impossible---both
in economic terms and because there are times when we just want to
go as fast as possible and you can never over-provision that.

>    (I take it to mean you want "1 packet".)

Sure.  In practical terms.  As long as you have at least one packet
in the queue then you are able to transmit the instant the link
becomes available and hence you can get full utilization.

> the Internet better -- not how to make the ISP business more

This list is about congestion control.  So, if your model is that
everything always needs to be over-provisioned relative to the load
then there will be no congestion and so we won't have to control
it.

> And, in fact, the difference between a target of one-packet
> standing queue and zero-packet is probably not worth much
> argument.

Math-wise, I agree.

But, I commented because I think the distinction is more important
than the slight delay a 1 packet queue will add.  It is the
difference between aiming in a direction that allows us to be able
to utilize a link and aiming in a direction that never will.

allman

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 13 Nov 2015 06:29:01 +0000

Hi

Thanks. I agree on the “widely deployed” scope. One reason why I wrote up the draft is the statement “It is not designed for wireless networks.” In draft-ietf-tcpm-cubic. The man reason why cubic is less apt for e.g. LTE is that AQM seems to be widely “undeployed”. Certain vendors offer AQM solutions but that does not mean that AQM is enabled, this means that the cubic congestion avoidance algorithm is quite inefficient. Still, even with AQM there are reason to believe that cubic (or even HyStart for that matter) is suboptimal, but is not for certain that the cubic algorithm is bad, it may as well be that elements like packet pacing is needed.

If, for some reason it becomes desired to implement very LTE specific algorithms then I would say that these suit better implemented in proxies as is done already today.
I hope that proposed draft can provide with some guidelines for algorithm designers, possibly new congestion control algorithms.
Whether or not it is possible to come up with very specific details around e.g. optimal standing queues is a future question that at least I don’t have the full answer to.
Finally, I don’t rule out that certain findings will find its way back to LTE enhancements and/or valuable input for 5G stack optimizations.

/Ingemar

From: Matt Mathis [mailto:[email-masked]]
Sent: den 10 november 2015 21:28
To: Scharf, Michael (Michael)
Cc: Ingemar Johansson S; Black, David; Michael Welzl; David Ros; [email-masked]; Karen E. E. Nielsen; [email-masked]
Subject: Re: [iccrg] [tsvwg] [tcpm] FW: New Version Notification for draft-johansson-cc-for-4g-5g-00.txt

Thanks,
--MM--

Privacy matters!  We know from recent events that people are using our services to speak in defiance of unjust governments.   We treat privacy and security as matters of life and death, because for some users, they are.

On Tue, Nov 10, 2015 at 11:32 AM, Scharf, Michael (Michael) <[email-masked]<mailto:[email-masked]>> wrote:
If a congestion control algorithm is used for TCP, the following sentence in the TCPM charter applies:

Michael

> From: iccrg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Ingemar
> Johansson S
> Cc: David Ros; [email-masked]<mailto:[email-masked]>; Karen E. E. Nielsen;
> [email-masked]<mailto:[email-masked]>; Ingemar Johansson S
> > From: Black, David [mailto:[email-masked]<mailto:[email-masked]>]
> > Cc: David Ros; Ingemar Johansson S; [email-masked]<mailto:[email-masked]>; Black, David;
> > [email-masked]<mailto:[email-masked]>
> > > From: Michael Welzl [mailto:[email-masked]<mailto:[email-masked]>]
> > > Cc: David Ros; Ingemar Johansson S; [email-masked]<mailto:[email-masked]>
> > > > On 05 Nov 2015, at 23:11, Black, David <[email-masked]<mailto:[email-masked]>>
> wrote:
> > > >
> > > > From: tsvwg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of David
> Ros
> > > > Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Michael Welzl
> > > <[email-masked]<mailto:[email-masked]>> wrote:
> > > >
> > > >
> > > > Hi
> > > >
> > > > From: Bob Briscoe [mailto:[email-masked]<mailto:[email-masked]>]
> > > > From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>]
> > > until the htmlized version and diff are available at
> tools.ietf.org<http://tools.ietf.org>.
> > > >
> > > > The IETF Secretariat
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> [email-masked]<mailto:[email-masked]>
_______________________________________________
iccrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/iccrg

---

From: Dave Taht <[email-masked]>
Date: Fri, 13 Nov 2015 09:12:59 +0100

On Tue, Nov 10, 2015 at 9:27 PM, Matt Mathis <[email-masked]> wrote:
> If you propose to try to put link technology specific hacks^H^H^H^H features
> in TCP, you are very likely to hear objections from unexpected places.
>
> answered in a uniform, transport protocol independent way (and eventually in
> all end systems), where it will be applied to *all* link types.

On some other thread somewhere we'd settled on a minimum of 1 packet.

However, as many of these link types do packet aggregation, an optimum
minimum is 1-2 aggregates, which can be of variable size based on link
technology - and contention.

While 802.11n wifi, for example can have 5.7ms of
max airtime per station, and 42 packets outstanding, later standards can have
many, many more packets outstanding, and a single station eating 5.7ms of
airtime is not desirable in more highly contended networks.

I do look forward to both the LTE and satellite industries describing
more fully their existing requirements and capabilities to the ietf.

> This is not a trivial question: I can think of some cases where optimal for

No kidding!

> improve both LTE and transport protocols.

+10! Everybody at layer 2 and layer 3 please lock themselves in a room
to sort this out! A metric ton of pizza required.

> On Tue, Nov 10, 2015 at 11:32 AM, Scharf, Michael (Michael)
> <[email-masked]> wrote:
>>
>> > tools.ietf.org.

---

From: grenville armitage <[email-masked]>
Date: Fri, 13 Nov 2015 19:43:03 +1100

On 11/13/2015 17:29, Ingemar Johansson S wrote:
	[..]
> Whether or not it is possible to come up with very specific details
> around e.g. optimal standing queues is a future question that at
> least I don’t have the full answer to.

And the meta-question, what is the optimal standing length of a reply email on a busy thread ;)

(/me starts in wonder at the prior email's ~100KB of superfluous, untrimmed glory.)

cheers,
gja

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 17 Nov 2015 07:46:22 +0000

Limiting my answer to only LTE scope (which I know better than other access types): 
Some kind of LTE rule for downlink transmission (network to terminal) is 
"Enough data to fill transport block completely the next time a bearer is scheduled", problem here is that there are two variables that are unknown to the transmitting endpoint.
1) Size of transport block, depends on resource availability and channel quality
2) Next scheduling event, depends on system load.

Given this, only one packet in queue is likely to give an under-utilized link, which will give reduced peak throughput. The severity of this of course depends. If the system (LTE radio access) load is low, then the loss in peak throughput may be considerable, if on the other hand the system load is high then it may not be much of a difference as other users may benefit from the under-utilization.

/Ingemar

> From: Mark Allman [mailto:[email-masked]]
> Sent: den 12 november 2015 15:28
> To: John Leslie
> Cc: [email-masked]; Matt Mathis
> Subject: Re: [iccrg] "Optimal target standing queue"
> 
> 
> If you want a world that is persistently over-provisioned then, well, OK.  But,
> I think in the real world that is impossible---both in economic terms and
> because there are times when we just want to go as fast as possible and you
> can never over-provision that.
> 
> Sure.  In practical terms.  As long as you have at least one packet in the queue
> then you are able to transmit the instant the link becomes available and
> hence you can get full utilization.
> 
> > On this list, we're supposed to be talking about research to make the
> > Internet better -- not how to make the ISP business more profitable.
> 
> This list is about congestion control.  So, if your model is that everything
> always needs to be over-provisioned relative to the load then there will be
> no congestion and so we won't have to control it.
> 
> But, I commented because I think the distinction is more important than the
> slight delay a 1 packet queue will add.  It is the difference between aiming in
> a direction that allows us to be able to utilize a link and aiming in a direction
> that never will.
> 
> allman
> 
>