Subject: Re: [Gen-art] Gen-art LC review of draft-ietf-aqm-recommendation-08

From: Elwyn Davies <[email-masked]>
Date: Wed, 07 Jan 2015 23:40:37 +0000

(Copied to aqm mailing list as suggested by WG chair).
Hi.

Thanks for your responses.  Just a reminder... I am not (these days, 
anyway) an expert in router queue management, so my comments should not 
be seen as deep critique of the individual items, but things that come 
to mind as matters of general control engineering and areas where I feel 
the language needs clarification  - that's what gen-art is for,

As a matter of interest it might be useful to explain a bit what scale 
of routing engine you are thinking about in this paper.  This is because 
I got a feeling from your responses to the buffer bloat question that 
you are primarily thinking big iron here.  The buffer bloat phenomenon 
has tended to be in smaller boxes where the AQM stuff may or may not be 
applicable.  I don't quite know what your target is here - or if you are 
thinking over the whole range of sizes.  The responses below clearly 
indicate that you have some examples in mind (Codel, for example which I 
know nothing about except (now) that it is an AQM WG product) and I 
don't know what scale of equipment these are really relevant to.

Some more responses in line.

Regards,
Elwyn

On 05/01/15 20:32, Fred Baker (fred) wrote:
>
>> On Jan 5, 2015, at 1:13 AM, [email-masked] wrote:
>>
>> Fred, I've applied the minor edits.
>>
>> I have questions to you on the comments blow (see GF:) before I
>> proceed.
>>
>> Gorry
>
> Adding Elwyn, as the discussion of his comments should include him -
> he might b able to clarify his concerns. I started last night to
> write a note, which I will now discard and instead comment here.
>
>>> I am the assigned Gen-ART reviewer for this draft. For background
>>> on Gen-ART, please see the FAQ at
>>>
>>> <http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.
>>>
>>> Please resolve these comments along with any other Last Call
>>> comments you may receive.
>>>
>>> Document: draft-ietf-aqm-recommendation-08.txt Reviewer: Elwyn
>>> Davies Review Date: 2014/12/19 IETF LC End Date: 2014/12/24 IESG
>>> Telechat date: (if known) -
>>>
>>> Summary:  Almost ready for BCP.
>>>
>>> Possibly missing issues:
>>>
>>> Buffer bloat:  The suggestions/discussions are pretty much all
>>> about keeping buffer size sufficiently large to avoid burst
>>> dropping.  It seems to me that it might be good to mention the
>>> possibility that one can over provision queues, and this needs to
>>> be avoided as well as under provisioning.
>>>
>> GF: I am not sure - this to me depends use case.
>
> To me, this is lily-gilding. To pick one example, the Cisco ASR 8X10G
> line card comes standard from the factory with 200 ms of queue per
> 10G interface. If we were to implement Codel on it, Codel would try
> desperately to keep the average induced latency less than five ms. If
> it tried to make it be 100 microseconds, we would run into the issues
> the draft talks about - we're trying to maximize rate while
> minimizing mean latency, and due to TCP's dynamics, we would no
> longer maximize rate. If 5 ms is a reasonable number (and for
> intra-continental terrestrial delays I would think it is), and we set
> that variable to 10, 50, or 100 ms, the only harm would be that we
> had some probability of a higher mean induced latency than was really
> necessary - AQM would be a little less effective. In the worst case,
> (suppose we set Codel's limit to 200 ms), it would revert to tail
> drop, which is what we already have.
>
> There are two reasonable responses to this. One would be to note that
> high RTT cases, even if auto-tuning mostly works, manual tuning may
> deliver better results or tune itself correctly more quickly (on a
> 650 ms RTT satcom link, I'd start by changing Codel's 100 ms trigger
> to something in the neighborhood of 650 ms). The other is to simply
> say that there is no direct harm in increasing the limits, and there
> may be value in some use cases. But I would also tend to think that
> anyone that actually operates a network already has a pretty good
> handle on that fact. So I don't see the value in saying it - which is
> mostly why it's not there already.
My take on this would be "make as few assumptions about your audience as 
possible, and write them down".  Its a generally interesting topic and 
would interest people who are not deeply skilled in the art - as well as 
potentially pulling in some new researchers!
>
>>> Interaction between boxes using different or the same algorithms:
>>> Buffer bloat seems to be generally about situations where chains
>>> of boxes all have too much buffer.  One thing that is not
>>> currently mentioned is the possibility that if different AQM
>>> schemes are implemented in various boxes through which a flow
>>> passes, then there could be inappropriate interaction between the
>>> different algorithms.  The old RFC suggested RED and nothing else
>>> so that one just had one to make sure multiple RED boxes in
>>> series didn't do anything bad.  With potentially different
>>> algorithms in series, one had better be sure that the mechanisms
>>> don't interact in a bad way when chained together - another
>>> research topic, I think.
>>>
>> GF: I think this could be added as an area for continued research
>> mentioned in section 4.7. At least I know of some poor
>> interactions between PIE and CoDel on particular paths - where both
>> algorithms are triggered. However, I doubt if this is worth much
>> discussion in this document? thoughts?
>>
>> Suggest: "The Internet presents a wide variety of paths where
>> traffic can experience combinations of mechanisms that can
>> potentially interact to influence the performance of applications.
>> Research therefore needs to consider the interactions between
>> different AQM algorithms, patterns of interaction in network
>> traffic and other network mechanisms to ensure that multiple
>> mechanisms do not inadvertently interact to impact performance."
>
> Mentioning it as a possible research area makes sense. Your proposed
> text is fine, from my perspective.
>
Yes. I think something like this would be good.   The buffer bloat 
example is probably an extreme case of things not having AQM at all and 
interacting badly.  It would maybe be worth mentioning that any AQM 
mechanism has also got to work in series with boxes that don't have any 
active AQM - just tail drop.  Ultimately, I would say this is just a 
matter of control engineering principles:  You are potentially making a 
network in which various control algorithms are implemented on different 
legs/nodes and the combination of transfer functions could possibly be 
unstable.  Has anybody applied any of the raft of control theoretic 
methods to these algorithms?  I have no idea!

> I start by questioning the underlying assumption, though, which is
> that bufferbloat is about paths in which there are multiple
> simultaneous bottlenecks. Yes, that occurs (think about paths that
> include both Cogent and a busy BRAS or CMTS, or more generally, if
> any link has some probability of congesting, math sophomore
> statistics course maintained that any pair of links has the product
> of the two probabilities of being simultaneously congested), but I'd
> be hard-pressed to make a statistically compelling argument out of
> it. The research and practice I have seen has been about a single
> bottleneck.
Please don't fixate on buffer bloat!
>
>>> Minor issues: s3, para after end of bullet 3:
>>>> The projected increase in the fraction of total Internet
>>>> traffic for more aggressive flows in classes 2 and 3 could pose
>>>> a threat to the performance of the future Internet.  There is
>>>> therefore an urgent need for measurements of current conditions
>>>> and for further research into the ways of managing such flows.
>>>> This raises many difficult issues in finding methods with an
>>>> acceptable overhead cost that can identify and isolate
>>>> unresponsive flows or flows that are less responsive than TCP.
>>>
>>> Question: Is there actually any published research into how one
>>> would identify class 2 or class 3 traffic in a router/middle box?
>>> If so it would be worth noting - the text call for "further
>>> research" seems to indicate there is something out there.
>>>
>> GF: I think the text is OK.
>
> Agreed. Elwyn's objection appears to be to the use of the word
> "further"; if we don't know of a paper, he'd like us to call for
> "research". The papers that come quickly to my mind are various
> papers on non-responsive flows, such as
> http://www.icir.org/floyd/papers/collapse.may99.pdf or
> http://www2.research.att.com/~jiawang/sstp08-camera/SSTP08_Pan.pdf.
> We already have a pretty extensive bibliography...
>
Right either remove/alter "further" if there isn't anything already out 
there or put in some reference(s).

>>> s4.2, next to last para: Is it worth saying also that the
>>> randomness should avoid targeting a single flow within a
>>> reasonable period to give a degree of fairness.
>
> Network devices SHOULD use an AQM algorithm to determine the packets
> that are marked or discarded due to congestion.  Procedures for
> dropping or marking packets within the network need to avoid
> increasing synchronization events, and hence randomness SHOULD be
> introduced in the algorithms that generate these congestion signals
> to the endpoints.
>
>> GF: Thoughts?
>
> I worry. The reasons for the randomness are (1) to tend to hit
> different sessions, and (2) when the same session is hit, to minimize
> the probability of multiple hits in the same RTT. It might be worth
> saying as much. However, to *stipulate* that algorithms should limit
> the hit rate on a given flow invites a discussion of stateful
> inspection algorithms. If someone wants to do such a thing, I'm not
> going to try to stop them (you could describe fq_* in those terms),
> but I don't want to put the idea into their heads (see later comment
> on privacy). Also, that is frankly more of a concern with Reno than
> with NewReno, and with NewReno than with anything that uses SACK.
> SACK will (usually) retransmit all dropped segments in the subsequent
> RTT, while NewReno will retransmit the Nth dropped packet in the Nth
> following RTT, and Reno might take that many RTO timeouts.

You have thought about what I said.  Put in what you think it needs.
>
>>> s4.2.1, next to last para:
>>>> An AQM algorithm that supports ECN needs to define the
>>>> threshold and algorithm for ECN-marking.  This threshold MAY
>>>> differ from that used for dropping packets that are not marked
>>>> as ECN-capable, and SHOULD be configurable.
>>>>
>>> Is this suggestion really compatible with recommendation 3 and
>>> s4.3 (no tuning)?
>>>
>> GF: I think making a recommendation here is beyond the "BCP"
>> experience, although I suspect that a lower marking threshold is
>> generally good. Should we add it also to the research agenda as an
>> item at the end of para 3 in S4.7.?

I think you may have misunderstood what I am saying here.  Rec 3 and 
s4.3 say things should work without tuning.  Doesn't having to set these 
thresholds/algorithms constitute tuning?  If so then it makes it 
difficult to see these ECN schemes as meeting the constraints.  If you 
disagree then explain how it isn't - or suggest  that there should be 
research to see how to make ECN zero config as well.
>
> I can see adding it to the research agenda; the comment comes from
> Bob Briscoe's research.
>
> That said, any algorithm using any mechanism by definition needs to
> specify any variables it uses - Codel, for example, tries to keep a
> queue at 5 ms or less, and cuts in after a queue fails to empty for a
> period of 100 ms. I don't see a good argument for saying "but an
> ECN-based algorithm doesn't need to define its thresholds or
> algorithms". Also, as I recall, the MAY in the text came from the
> fact that Bob seemed to think there was value in it (which BTW I
> agree with). To my mind, SHOULD and MUST are strong words, but absent
> such an assertion, an implementation MAY do just about anything that
> comes to the implementor's mind. So saying an implementation MAY <do
> something> is mostly a suggestion that an implementor SHOULD think
> about it. Are we to say that an implementor, given Bob's research,
> should NOT think about giving folks the option?
>
> I also don't think Elwyn's argument quite follows. When I say that an
> algorithm should auto-tune, I'm not saying that it should not have
> knobs; I'm saying that the default values of those knobs should be
> adequate for the vast majority of use cases. I'm also not saying that
> there should be exactly one initial default; I could easily imagine
> an implementation noting the bit rate of an interface and the ping
> RTT to a peer and pulling its initial configuration out of a table.
That would be at least partially acceptable as a mode of operation.  But 
you might have a "warm-up" issue - would it work OK while the algorithm 
was working out what the RTT actually was?  And would the algorithms 
adapt autonomously (i.e., auto-tune) to close in on optimum values after 
picking initial values from the table?
>
>>> s7:  There is an arguable privacy concern that if schemes are
>>> able to identify class 2 or class 3 flows, then a core device can
>>> extract privacy related info from the identified flows.
>>>
>> GF: I don't see how traffic profiles expose privacy concerns, sure
>> users and apps can be characterised by patterns of interaction -
>> but this isn't what is being talked about here.
>
> Agreed. If the reference is to RFC 6973, I don't see a violation of
> https://tools.ietf.org/html/rfc6973#section-7. I would if we appeared
> to be inviting stateful inspection algorithms. To give an example of
> how difficult sessions are managed, RFC 6057 uses the CTS message in
> round-robin fashion to push back on top-talker users in order to
> enable the service provider to give consistent service to all of his
> subscribers when a few are behaving in a manner that might prevent
> him from doing so. Note that the "session", in that case, is not a
> single TCP session, but a bittorrent-or-whatever server engaged in
> sessions to tens or hundreds of peers. The fact that a few users
> receive some pushback doesn't reveal the identities of those users.
> I'd need to hear the substance behind Elwyn's concern before I could
> write anything.

My reaction was that if your algorithm identifies flows then you have

On 05/01/15 20:32, Fred Baker (fred) wrote:>
 >> On Jan 5, 2015, at 1:13 AM, [email-masked] wrote:
 >>
 >> Fred, I've applied the minor edits.
 >>
 >> I have questions to you on the comments blow (see GF:) before I proceed.
 >>
 >> Gorry
 >
 > Adding Elwyn, as the discussion of his comments should include him - 
he might b able to clarify his concerns. I started last night to write a 
note, which I will now discard and instead comment here.
 >
 >>> I am the assigned Gen-ART reviewer for this draft. For background on
 >>> Gen-ART, please see the FAQ at
 >>>
 >>> <http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.
 >>>
 >>> Please resolve these comments along with any other Last Call comments
 >>> you may receive.
 >>>
 >>> Document: draft-ietf-aqm-recommendation-08.txt
 >>> Reviewer: Elwyn Davies
 >>> Review Date: 2014/12/19
 >>> IETF LC End Date: 2014/12/24
 >>> IESG Telechat date: (if known) -
 >>>
 >>> Summary:  Almost ready for BCP.
 >>>
 >>> Possibly missing issues:
 >>>
 >>> Buffer bloat:  The suggestions/discussions are pretty much all about
 >>> keeping buffer size
 >>> sufficiently large to avoid burst dropping.  It seems to me that it 
might
 >>> be good to
 >>> mention the possibility that one can over provision queues, and 
this needs
 >>> to be avoided
 >>> as well as under provisioning.
 >>>
 >> GF: I am not sure - this to me depends use case.
 >
 > To me, this is lily-gilding. To pick one example, the Cisco ASR 8X10G 
line card comes standard from the factory with 200 ms of queue per 10G 
interface. If we were to implement Codel on it, Codel would try 
desperately to keep the average induced latency less than five ms. If it 
tried to make it be 100 microseconds, we would run into the issues the 
draft talks about - we're trying to maximize rate while minimizing mean 
latency, and due to TCP's dynamics, we would no longer maximize rate. If 
5 ms is a reasonable number (and for intra-continental terrestrial 
delays I would think it is), and we set that variable to 10, 50, or 100 
ms, the only harm would be that we had some probability of a higher mean 
induced latency than was really necessary - AQM would be a little less 
effective. In the worst case, (suppose we set Codel's limit to 200 ms), 
it would revert to tail drop, which is what we already have.
 >
 > There are two reasonable responses to this. One would be to note that 
high RTT cases, even if auto-tuning mostly works, manual tuning may 
deliver better results or tune itself correctly more quickly (on a 650 
ms RTT satcom link, I'd start by changing Codel's 100 ms trigger to 
something in the neighborhood of 650 ms). The other is to simply say 
that there is no direct harm in increasing the limits, and there may be 
value in some use cases. But I would also tend to think that anyone that 
actually operates a network already has a pretty good handle on that 
fact. So I don't see the value in saying it - which is mostly why it's 
not there already.
 >
 >>> Interaction between boxes using different or the same algorithms: 
Buffer
 >>> bloat seems to
 >>> be generally about situations where chains of boxes all have too much
 >>> buffer.  One thing
 >>> that is not currently mentioned is the possibility that if 
different AQM
 >>> schemes are
 >>> implemented in various boxes through which a flow passes, then 
there could
 >>> be inappropriate
 >>> interaction between the different algorithms.  The old RFC 
suggested RED
 >>> and nothing else so
 >>> that one just had one to make sure multiple RED boxes in series 
didn't do
 >>> anything bad.  With
 >>> potentially different algorithms in series, one had better be sure that
 >>> the mechanisms don't
 >>> interact in a bad way when chained together - another research topic, I
 >>> think.
 >>>
 >> GF: I think this could be added as an area for continued research
 >> mentioned in section 4.7. At least I know of some poor interactions
 >> between PIE and CoDel on particular paths - where both algorithms are
 >> triggered. However, I doubt if this is worth much discussion in this
 >> document? thoughts?
 >>
 >> Suggest:
 >> "The Internet presents a wide variety of paths where traffic can
 >> experience combinations of mechanisms that can potentially interact to
 >> influence the performance of applications. Research therefore needs to
 >> consider the interactions between different AQM algorithms, patterns of
 >> interaction in network traffic and other network mechanisms to 
ensure that
 >> multiple mechanisms do not inadvertently interact to impact 
performance."
 >
 > Mentioning it as a possible research area makes sense. Your proposed 
text is fine, from my perspective.
 >
 > I start by questioning the underlying assumption, though, which is 
that bufferbloat is about paths in which there are multiple simultaneous 
bottlenecks. Yes, that occurs (think about paths that include both 
Cogent and a busy BRAS or CMTS, or more generally, if any link has some 
probability of congesting, math sophomore statistics course maintained 
that any pair of links has the product of the two probabilities of being 
simultaneously congested), but I'd be hard-pressed to make a 
statistically compelling argument out of it. The research and practice I 
have seen has been about a single bottleneck.
 >
 >>> Minor issues:
 >>> s3, para after end of bullet 3:
 >>>>     The projected increase in the fraction of total Internet 
traffic for
 >>>>     more aggressive flows in classes 2 and 3 could pose a threat 
to the
 >>>>     performance of the future Internet.  There is therefore an urgent
 >>>>     need for measurements of current conditions and for further 
research
 >>>>     into the ways of managing such flows.  This raises many difficult
 >>>>     issues in finding methods with an acceptable overhead cost 
that can
 >>>>     identify and isolate unresponsive flows or flows that are less
 >>>>     responsive than TCP.
 >>>
 >>> Question: Is there actually any published research into how one would
 >>> identify
 >>> class 2 or class 3 traffic in a router/middle box? If so it would be
 >>> worth noting -
 >>> the text call for "further research" seems to indicate there is
 >>> something out there.
 >>>
 >> GF: I think the text is OK.
 >
 > Agreed. Elwyn's objection appears to be to the use of the word 
"further"; if we don't know of a paper, he'd like us to call for 
"research". The papers that come quickly to my mind are various papers 
on non-responsive flows, such as 
http://www.icir.org/floyd/papers/collapse.may99.pdf or 
http://www2.research.att.com/~jiawang/sstp08-camera/SSTP08_Pan.pdf. We 
already have a pretty extensive bibliography...
 >
 >>> s4.2, next to last para: Is it worth saying also that the randomness
 >>> should avoid targeting a single flow within a reasonable period to give
 >>> a degree of fairness.
 >
 >     Network devices SHOULD use an AQM algorithm to determine the packets
 >     that are marked or discarded due to congestion.  Procedures for
 >     dropping or marking packets within the network need to avoid
 >     increasing synchronization events, and hence randomness SHOULD be
 >     introduced in the algorithms that generate these congestion signals
 >     to the endpoints.
 >
 >> GF: Thoughts?
 >
 > I worry. The reasons for the randomness are (1) to tend to hit 
different sessions, and (2) when the same session is hit, to minimize 
the probability of multiple hits in the same RTT. It might be worth 
saying as much. However, to *stipulate* that algorithms should limit the 
hit rate on a given flow invites a discussion of stateful inspection 
algorithms. If someone wants to do such a thing, I'm not going to try to 
stop them (you could describe fq_* in those terms), but I don't want to 
put the idea into their heads (see later comment on privacy). Also, that 
is frankly more of a concern with Reno than with NewReno, and with 
NewReno than with anything that uses SACK. SACK will (usually) 
retransmit all dropped segments in the subsequent RTT, while NewReno 
will retransmit the Nth dropped packet in the Nth following RTT, and 
Reno might take that many RTO timeouts.

On 05/01/15 20:32, Fred Baker (fred) wrote:>
 >> On Jan 5, 2015, at 1:13 AM, [email-masked] wrote:
 >>
 >> Fred, I've applied the minor edits.
 >>
 >> I have questions to you on the comments blow (see GF:) before I proceed.
 >>
 >> Gorry
 >
 > Adding Elwyn, as the discussion of his comments should include him - 
he might b able to clarify his concerns. I started last night to write a 
note, which I will now discard and instead comment here.
 >
 >>> I am the assigned Gen-ART reviewer for this draft. For background on
 >>> Gen-ART, please see the FAQ at
 >>>
 >>> <http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.
 >>>
 >>> Please resolve these comments along with any other Last Call comments
 >>> you may receive.
 >>>
 >>> Document: draft-ietf-aqm-recommendation-08.txt
 >>> Reviewer: Elwyn Davies
 >>> Review Date: 2014/12/19
 >>> IETF LC End Date: 2014/12/24
 >>> IESG Telechat date: (if known) -
 >>>
 >>> Summary:  Almost ready for BCP.
 >>>
 >>> Possibly missing issues:
 >>>
 >>> Buffer bloat:  The suggestions/discussions are pretty much all about
 >>> keeping buffer size
 >>> sufficiently large to avoid burst dropping.  It seems to me that it 
might
 >>> be good to
 >>> mention the possibility that one can over provision queues, and 
this needs
 >>> to be avoided
 >>> as well as under provisioning.
 >>>
 >> GF: I am not sure - this to me depends use case.
 >
 > To me, this is lily-gilding. To pick one example, the Cisco ASR 8X10G 
line card comes standard from the factory with 200 ms of queue per 10G 
interface. If we were to implement Codel on it, Codel would try 
desperately to keep the average induced latency less than five ms. If it 
tried to make it be 100 microseconds, we would run into the issues the 
draft talks about - we're trying to maximize rate while minimizing mean 
latency, and due to TCP's dynamics, we would no longer maximize rate. If 
5 ms is a reasonable number (and for intra-continental terrestrial 
delays I would think it is), and we set that variable to 10, 50, or 100 
ms, the only harm would be that we had some probability of a higher mean 
induced latency than was really necessary - AQM would be a little less 
effective. In the worst case, (suppose we set Codel's limit to 200 ms), 
it would revert to tail drop, which is what we already have.
 >
 > There are two reasonable responses to this. One would be to note that 
high RTT cases, even if auto-tuning mostly works, manual tuning may 
deliver better results or tune itself correctly more quickly (on a 650 
ms RTT satcom link, I'd start by changing Codel's 100 ms trigger to 
something in the neighborhood of 650 ms). The other is to simply say 
that there is no direct harm in increasing the limits, and there may be 
value in some use cases. But I would also tend to think that anyone that 
actually operates a network already has a pretty good handle on that 
fact. So I don't see the value in saying it - which is mostly why it's 
not there already.
 >
 >>> Interaction between boxes using different or the same algorithms: 
Buffer
 >>> bloat seems to
 >>> be generally about situations where chains of boxes all have too much
 >>> buffer.  One thing
 >>> that is not currently mentioned is the possibility that if 
different AQM
 >>> schemes are
 >>> implemented in various boxes through which a flow passes, then 
there could
 >>> be inappropriate
 >>> interaction between the different algorithms.  The old RFC 
suggested RED
 >>> and nothing else so
 >>> that one just had one to make sure multiple RED boxes in series 
didn't do
 >>> anything bad.  With
 >>> potentially different algorithms in series, one had better be sure that
 >>> the mechanisms don't
 >>> interact in a bad way when chained together - another research topic, I
 >>> think.
 >>>
 >> GF: I think this could be added as an area for continued research
 >> mentioned in section 4.7. At least I know of some poor interactions
 >> between PIE and CoDel on particular paths - where both algorithms are
 >> triggered. However, I doubt if this is worth much discussion in this
 >> document? thoughts?
 >>
 >> Suggest:
 >> "The Internet presents a wide variety of paths where traffic can
 >> experience combinations of mechanisms that can potentially interact to
 >> influence the performance of applications. Research therefore needs to
 >> consider the interactions between different AQM algorithms, patterns of
 >> interaction in network traffic and other network mechanisms to 
ensure that
 >> multiple mechanisms do not inadvertently interact to impact 
performance."
 >
 > Mentioning it as a possible research area makes sense. Your proposed 
text is fine, from my perspective.
 >
 > I start by questioning the underlying assumption, though, which is 
that bufferbloat is about paths in which there are multiple simultaneous 
bottlenecks. Yes, that occurs (think about paths that include both 
Cogent and a busy BRAS or CMTS, or more generally, if any link has some 
probability of congesting, math sophomore statistics course maintained 
that any pair of links has the product of the two probabilities of being 
simultaneously congested), but I'd be hard-pressed to make a 
statistically compelling argument out of it. The research and practice I 
have seen has been about a single bottleneck.
 >
 >>> Minor issues:
 >>> s3, para after end of bullet 3:
 >>>>     The projected increase in the fraction of total Internet 
traffic for
 >>>>     more aggressive flows in classes 2 and 3 could pose a threat 
to the
 >>>>     performance of the future Internet.  There is therefore an urgent
 >>>>     need for measurements of current conditions and for further 
research
 >>>>     into the ways of managing such flows.  This raises many difficult
 >>>>     issues in finding methods with an acceptable overhead cost 
that can
 >>>>     identify and isolate unresponsive flows or flows that are less
 >>>>     responsive than TCP.
 >>>
 >>> Question: Is there actually any published research into how one would
 >>> identify
 >>> class 2 or class 3 traffic in a router/middle box? If so it would be
 >>> worth noting -
 >>> the text call for "further research" seems to indicate there is
 >>> something out there.
 >>>
 >> GF: I think the text is OK.
 >
 > Agreed. Elwyn's objection appears to be to the use of the word 
"further"; if we don't know of a paper, he'd like us to call for 
"research". The papers that come quickly to my mind are various papers 
on non-responsive flows, such as 
http://www.icir.org/floyd/papers/collapse.may99.pdf or 
http://www2.research.att.com/~jiawang/sstp08-camera/SSTP08_Pan.pdf. We 
already have a pretty extensive bibliography...
 >
 >>> s4.2, next to last para: Is it worth saying also that the randomness
 >>> should avoid targeting a single flow within a reasonable period to give
 >>> a degree of fairness.
 >
 >     Network devices SHOULD use an AQM algorithm to determine the packets
 >     that are marked or discarded due to congestion.  Procedures for
 >     dropping or marking packets within the network need to avoid
 >     increasing synchronization events, and hence randomness SHOULD be
 >     introduced in the algorithms that generate these congestion signals
 >     to the endpoints.
 >
 >> GF: Thoughts?
 >
 > I worry. The reasons for the randomness are (1) to tend to hit 
different sessions, and (2) when the same session is hit, to minimize 
the probability of multiple hits in the same RTT. It might be worth 
saying as much. However, to *stipulate* that algorithms should limit the 
hit rate on a given flow invites a discussion of stateful inspection 
algorithms. If someone wants to do such a thing, I'm not going to try to 
stop them (you could describe fq_* in those terms), but I don't want to 
put the idea into their heads (see later comment on privacy). Also, that 
is frankly more of a concern with Reno than with NewReno, and with 
NewReno than with anything that uses SACK. SACK will (usually) 
retransmit all dropped segments in the subsequent RTT, while NewReno 
will retransmit the Nth dropped packet in the Nth following RTT, and 
Reno might take that many RTO timeouts.
 >
 >>> s4.2.1, next to last para:
 >>>>     An AQM algorithm that supports ECN needs to define the 
threshold and
 >>>>     algorithm for ECN-marking.  This threshold MAY differ from 
that used
 >>>>     for dropping packets that are not marked as ECN-capable, and 
SHOULD
 >>>>     be configurable.
 >>>>
 >>> Is this suggestion really compatible with recommendation 3 and s4.3 (no
 >>> tuning)?
 >>>
 >> GF: I think making a recommendation here is beyond the "BCP" experience,
 >> although I suspect that a lower marking threshold is generally good.
 >> Should we add it also to the research agenda as an item at the end 
of para
 >> 3 in S4.7.?
 >
 > I can see adding it to the research agenda; the comment comes from 
Bob Briscoe's research.
 >
 > That said, any algorithm using any mechanism by definition needs to 
specify any variables it uses - Codel, for example, tries to keep a 
queue at 5 ms or less, and cuts in after a queue fails to empty for a 
period of 100 ms. I don't see a good argument for saying "but an 
ECN-based algorithm doesn't need to define its thresholds or 
algorithms". Also, as I recall, the MAY in the text came from the fact 
that Bob seemed to think there was value in it (which BTW I agree with). 
To my mind, SHOULD and MUST are strong words, but absent such an 
assertion, an implementation MAY do just about anything that comes to 
the implementor's mind. So saying an implementation MAY <do something> 
is mostly a suggestion that an implementor SHOULD think about it. Are we 
to say that an implementor, given Bob's research, should NOT think about 
giving folks the option?
 >
 > I also don't think Elwyn's argument quite follows. When I say that an 
algorithm should auto-tune, I'm not saying that it should not have 
knobs; I'm saying that the default values of those knobs should be 
adequate for the vast majority of use cases. I'm also not saying that 
there should be exactly one initial default; I could easily imagine an 
implementation noting the bit rate of an interface and the ping RTT to a 
peer and pulling its initial configuration out of a table.
 >
 >>> s7:  There is an arguable privacy concern that if schemes are able to
 >>> identify class 2 or class 3 flows, then a core device can extract
 >>> privacy related info from the identified flows.
 >>>
 >> GF: I don't see how traffic profiles expose privacy concerns, sure users
 >> and apps can be characterised by patterns of interaction - but this 
isn't
 >> what is being talked about here.
 >
 > Agreed. If the reference is to RFC 6973, I don't see a violation of 
https://tools.ietf.org/html/rfc6973#section-7. I would if we appeared to 
be inviting stateful inspection algorithms. To give an example of how 
difficult sessions are managed, RFC 6057 uses the CTS message in 
round-robin fashion to push back on top-talker users in order to enable 
the service provider to give consistent service to all of his 
subscribers when a few are behaving in a manner that might prevent him 
from doing so. Note that the "session", in that case, is not a single 
TCP session, but a bittorrent-or-whatever server engaged in sessions to 
tens or hundreds of peers. The fact that a few users receive some 
pushback doesn't reveal the identities of those users. I'd need to hear 
the substance behind Elwyn's concern before I could write anything.
 >
 >> s4.7, para 3:
 >>> the use of Map/Reduce applications in data centers
 >>> I think this needs a reference or a brief explanation.
 >> GF: Fred do you know a reference or can suggest extra text?
 >
 > The concern has to do with incast, which is a pretty active research 
area (http://lmgtfy.com/?q=research+incast). The paragraph asks a 
question, which is whether the common taxonomy of network flows (mice vs 
elephants) needs to be extended to include references to herds of mice 
traveling together, with the result that congestion control algorithms 
designed under the assumption that a heavy data flow contains an 
elephant merely introduce head-of-line blocking in short flows. The word 
"lemmings" is mine.
 >
 > I know of at least four papers (Microsoft Research, CAIA, Tsinghua, 
and KAIST) submitted to various journals in 2014 on the topic. It's 
also, at least in part, the basis for the DCLC RG. The only ones we 
could reference, among those, would relate to DCTCP, as the rest have 
not yet been published.
 >
 > Again, I'd like to understand the underlying issue. I doubt that it 
is that Elwyn doesn't like the question as such. Is it that he's looking 
for the word “incast” to replace "map/reduce"?
 >
 >> --- The edits below have been incorporated in the XML for  v-09 ---
 >>> Nits/editorial comments:
 >>> General: s/e.g./e.g.,/, s/i.e./i.e.,/
 >>>
 >>> s1.2, para 2(?) - top of p4: s/and often necessary/and is often 
necessary/
 >>> s1.2, para 3: s/a > class of technologies that/a class of technologies
 >>> that/
 >>>
 >>> s2, first bullet 3: s/Large burst of packets/Large bursts of packets/
 >>>
 >>> s2, last para: Probably need to expand POP, IMAP and RDP; maybe provide
 >>> refs??
 >>>
 >>> s2.1, last para: s/open a large numbers of short TCP flows/may open a
 >>> large number of short duration TCP flows/
 >>>
 >>> s4, last para: s/experience occasional issues that need moderation./can
 >>> experience occasional issues that warrant mitigation./
 >>>
 >>> s4.2, para 6, last sentence: s/similarly react/react similarly/
 >>>
 >>> s4.2.1, para 1: s/using AQM to decider when/using AQM to decide when/
 >>>
 >>> s4.7, para 3:
 >>>> In 2013,
 >>> "At the time of writing" ?
 >>>
 >>> s4.7, para 3:
 >>>> the use of Map/Reduce applications in data centers
 >>> I think this needs a reference or a brief explanation.
 >

 >>> s4.2.1, next to last para:
 >>>>     An AQM algorithm that supports ECN needs to define the 
threshold and
 >>>>     algorithm for ECN-marking.  This threshold MAY differ from 
that used
 >>>>     for dropping packets that are not marked as ECN-capable, and 
SHOULD
 >>>>     be configurable.
 >>>>
 >>> Is this suggestion really compatible with recommendation 3 and s4.3 (no
 >>> tuning)?
 >>>
 >> GF: I think making a recommendation here is beyond the "BCP" experience,
 >> although I suspect that a lower marking threshold is generally good.
 >> Should we add it also to the research agenda as an item at the end 
of para
 >> 3 in S4.7.?
 >
 > I can see adding it to the research agenda; the comment comes from 
Bob Briscoe's research.
 >
 > That said, any algorithm using any mechanism by definition needs to 
specify any variables it uses - Codel, for example, tries to keep a 
queue at 5 ms or less, and cuts in after a queue fails to empty for a 
period of 100 ms. I don't see a good argument for saying "but an 
ECN-based algorithm doesn't need to define its thresholds or 
algorithms". Also, as I recall, the MAY in the text came from the fact 
that Bob seemed to think there was value in it (which BTW I agree with). 
To my mind, SHOULD and MUST are strong words, but absent such an 
assertion, an implementation MAY do just about anything that comes to 
the implementor's mind. So saying an implementation MAY <do something> 
is mostly a suggestion that an implementor SHOULD think about it. Are we 
to say that an implementor, given Bob's research, should NOT think about 
giving folks the option?
 >
 > I also don't think Elwyn's argument quite follows. When I say that an 
algorithm should auto-tune, I'm not saying that it should not have 
knobs; I'm saying that the default values of those knobs should be 
adequate for the vast majority of use cases. I'm also not saying that 
there should be exactly one initial default; I could easily imagine an 
implementation noting the bit rate of an interface and the ping RTT to a 
peer and pulling its initial configuration out of a table.
 >
 >>> s7:  There is an arguable privacy concern that if schemes are able to
 >>> identify class 2 or class 3 flows, then a core device can extract
 >>> privacy related info from the identified flows.
 >>>
 >> GF: I don't see how traffic profiles expose privacy concerns, sure users
 >> and apps can be characterised by patterns of interaction - but this 
isn't
 >> what is being talked about here.
 >
 > Agreed. If the reference is to RFC 6973, I don't see a violation of 
https://tools.ietf.org/html/rfc6973#section-7. I would if we appeared to 
be inviting stateful inspection algorithms. To give an example of how 
difficult sessions are managed, RFC 6057 uses the CTS message in 
round-robin fashion to push back on top-talker users in order to enable 
the service provider to give consistent service to all of his 
subscribers when a few are behaving in a manner that might prevent him 
from doing so. Note that the "session", in that case, is not a single 
TCP session, but a bittorrent-or-whatever server engaged in sessions to 
tens or hundreds of peers. The fact that a few users receive some 
pushback doesn't reveal the identities of those users. I'd need to hear 
the substance behind Elwyn's concern before I could write anything.
 >
 >> s4.7, para 3:
 >>> the use of Map/Reduce applications in data centers
 >>> I think this needs a reference or a brief explanation.
 >> GF: Fred do you know a reference or can suggest extra text?
 >
 > The concern has to do with incast, which is a pretty active research 
area (http://lmgtfy.com/?q=research+incast). The paragraph asks a 
question, which is whether the common taxonomy of network flows (mice vs 
elephants) needs to be extended to include references to herds of mice 
traveling together, with the result that congestion control algorithms 
designed under the assumption that a heavy data flow contains an 
elephant merely introduce head-of-line blocking in short flows. The word 
"lemmings" is mine.
 >
 > I know of at least four papers (Microsoft Research, CAIA, Tsinghua, 
and KAIST) submitted to various journals in 2014 on the topic. It's 
also, at least in part, the basis for the DCLC RG. The only ones we 
could reference, among those, would relate to DCTCP, as the rest have 
not yet been published.
 >
 > Again, I'd like to understand the underlying issue. I doubt that it 
is that Elwyn doesn't like the question as such. Is it that he's looking 
for the word “incast” to replace "map/reduce"?
 >
 >> --- The edits below have been incorporated in the XML for  v-09 ---
 >>> Nits/editorial comments:
 >>> General: s/e.g./e.g.,/, s/i.e./i.e.,/
 >>>
 >>> s1.2, para 2(?) - top of p4: s/and often necessary/and is often 
necessary/
 >>> s1.2, para 3: s/a > class of technologies that/a class of technologies
 >>> that/
 >>>
 >>> s2, first bullet 3: s/Large burst of packets/Large bursts of packets/
 >>>
 >>> s2, last para: Probably need to expand POP, IMAP and RDP; maybe provide
 >>> refs??
 >>>
 >>> s2.1, last para: s/open a large numbers of short TCP flows/may open a
 >>> large number of short duration TCP flows/
 >>>
 >>> s4, last para: s/experience occasional issues that need moderation./can
 >>> experience occasional issues that warrant mitigation./
 >>>
 >>> s4.2, para 6, last sentence: s/similarly react/react similarly/
 >>>
 >>> s4.2.1, para 1: s/using AQM to decider when/using AQM to decide when/
 >>>
 >>> s4.7, para 3:
 >>>> In 2013,
 >>> "At the time of writing" ?
 >>>
 >>> s4.7, para 3:
 >>>> the use of Map/Reduce applications in data centers
 >>> I think this needs a reference or a brief explanation.
 >
potentially helped a bad actor to pick off such flows or get to know who 
is communicating in a situation that currently it would be very 
difficult to know as the queueing is basically flow agnostic.  OK this 
fairly way out but we have seen some pretty serious stuff apparently 
being done around core routers according to Snowden et al.
>
>> s4.7, para 3:
>>> the use of Map/Reduce applications in data centers I think this
>>> needs a reference or a brief explanation.
>> GF: Fred do you know a reference or can suggest extra text?
>
> The concern has to do with incast, which is a pretty active research
> area (http://lmgtfy.com/?q=research+incast). The paragraph asks a
> question, which is whether the common taxonomy of network flows (mice
> vs elephants) needs to be extended to include references to herds of
> mice traveling together, with the result that congestion control
> algorithms designed under the assumption that a heavy data flow
> contains an elephant merely introduce head-of-line blocking in short
> flows. The word "lemmings" is mine.
>
> I know of at least four papers (Microsoft Research, CAIA, Tsinghua,
> and KAIST) submitted to various journals in 2014 on the topic. It's
> also, at least in part, the basis for the DCLC RG. The only ones we
> could reference, among those, would relate to DCTCP, as the rest have
> not yet been published.
>
> Again, I'd like to understand the underlying issue. I doubt that it
> is that Elwyn doesn't like the question as such. Is it that he's
> looking for the word “incast” to replace "map/reduce"?

I was just looking for somebody to define the jargon - As far as I am 
concerned at this moment "incast" would be just as "bad" since it would 
produce an equally blank stare followed by a grab for Google.
>
>> --- The edits below have been incorporated in the XML for  v-09
>> ---
>>> Nits/editorial comments: General: s/e.g./e.g.,/, s/i.e./i.e.,/
>>>
>>> s1.2, para 2(?) - top of p4: s/and often necessary/and is often
>>> necessary/ s1.2, para 3: s/a > class of technologies that/a class
>>> of technologies that/
>>>
>>> s2, first bullet 3: s/Large burst of packets/Large bursts of
>>> packets/
>>>
>>> s2, last para: Probably need to expand POP, IMAP and RDP; maybe
>>> provide refs??
>>>
>>> s2.1, last para: s/open a large numbers of short TCP flows/may
>>> open a large number of short duration TCP flows/
>>>
>>> s4, last para: s/experience occasional issues that need
>>> moderation./can experience occasional issues that warrant
>>> mitigation./
>>>
>>> s4.2, para 6, last sentence: s/similarly react/react similarly/
>>>
>>> s4.2.1, para 1: s/using AQM to decider when/using AQM to decide
>>> when/
>>>
>>> s4.7, para 3:
>>>> In 2013,
>>> "At the time of writing" ?
>>>
>>> s4.7, para 3:
>>>> the use of Map/Reduce applications in data centers
>>> I think this needs a reference or a brief explanation.
>

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 8 Jan 2015 01:20:29 +0000

> On Jan 7, 2015, at 3:40 PM, Elwyn Davies <[email-masked]> wrote:
> 
> Thanks for your responses.  Just a reminder... I am not (these days, anyway) an expert in router queue management, so my comments should not be seen as deep critique of the individual items, but things that come to mind as matters of general control engineering and areas where I feel the language needs clarification  - that's what gen-art is for,
> 
> As a matter of interest it might be useful to explain a bit what scale of routing engine you are thinking about in this paper.  This is because I got a feeling from your responses to the buffer bloat question that you are primarily thinking big iron here.  The buffer bloat phenomenon has tended to be in smaller boxes where the AQM stuff may or may not be applicable.

If Dave Taht replies, he’ll mention OpenWRT. Personally, I am thinking about ISP interconnect points and Access equipment such as BRAS and CMTS, which are indeed big iron, but also small CPEs such as might implement Homenet technologies and OpenWRT.

> I don't quite know what your target is here - or if you are thinking over the whole range of sizes.  The responses below clearly indicate that you have some examples in mind (Codel, for example which I know nothing about except (now) that it is an AQM WG product) and I don't know what scale of equipment these are really relevant to.
> 

> My take on this would be "make as few assumptions about your audience as possible, and write them down".  Its a generally interesting topic and would interest people who are not deeply skilled in the art - as well as potentially pulling in some new researchers!

I’m still not entirely sure what you’d like to have said. Is this a one sentence “setting limits looser than necessary is neither harmful nor helpful”, or a treatise on the topic?

> Yes. I think something like this would be good.   The buffer bloat example is probably an extreme case of things not having AQM at all and interacting badly.  It would maybe be worth mentioning that any AQM mechanism has also got to work in series with boxes that don't have any active AQM - just tail drop. Ultimately, I would say this is just a matter of control engineering principles:  You are potentially making a network in which various control algorithms are implemented on different legs/nodes and the combination of transfer functions could possibly be unstable.  Has anybody applied any of the raft of control theoretic methods to these algorithms?  I have no idea!

Well, PIE basically came out of control theory (a basic equation that describes a phase-locked loop), and I believe that Van and Kathy will say something similar about Codel. But that’s not a question for this paper, it’s a question for the various algorithmic papers.

> Please don't fixate on buffer bloat!

?

AQM is absolutely fixated on buffer bloat. We have called it a lot of things over the years, none of them very pleasant, but the fundamental issue in RFC 2309 and V+K’s RED work was maximizing throughput while minimizing queue occupancy.

> Right either remove/alter "further" if there isn't anything already out there or put in some reference(s).

OK, Gorry. You have two papers there to refer to. There are more, but two should cover it.

>>>> reasonable period to give a degree of fairness.

The text from the spec is:

> You have thought about what I said.  Put in what you think it needs.

Well, I didn’t think it needed a lot of saying. :-) How about this?

Network devices SHOULD use an AQM algorithm to measure local congestion and determine which packets to mark or drop to manage congestion. In general, dropping or marking multiple packets from the same sessions in the same RTT is ineffective, and can negative consequences. Also, dropping or marking packets from multiple sessions simultaneously has the effect of synchronizing them, meaning that subsequent peaks and troughs in traffic load are exacerbated. Hence, AQM algorithms should randomize dropping and marking in time, to desynchronize sessions and improve overall algorithmic effectiveness.

> I think you may have misunderstood what I am saying here.  Rec 3 and s4.3 say things should work without tuning.  Doesn't having to set these thresholds/algorithms constitute tuning?  If so then it makes it difficult to see these ECN schemes as meeting the constraints.  If you disagree then explain how it isn't - or suggest  that there should be research to see how to make ECN zero config as well.

Well, I think you may have misunderstood the statement in the draft. The big problem with RED, mandated in RFC 2309, was that it couldn’t be deployed without case by case tuning. We’re trying to minimize that. 

Recommendation 3 is:

   3.  The algorithms that the IETF recommends SHOULD NOT require
       operational (especially manual) configuration or tuning.

The title of section 4.3, which is the section explaining recommendation 3, is "AQM algorithms deployed SHOULD NOT require operational tuning”. That’s not “MUST NOT”; if there is a case, there is a case. It goes on to say:

   o  SHOULD NOT require tuning of initial or configuration parameters.
      An algorithm needs to provide a default behaviour that auto-tunes
      to a reasonable performance for typical network operational
      conditions.  This is expected to ease deployment and operation.
      Initial conditions, such as the interface rate and MTU size or
      other values derived from these, MAY be required by an AQM
      algorithm.

   o  MAY support further manual tuning that could improve performance
      in a specific deployed network. 

We’re looking for “reasonable performance in the general case”, and allowing for the use of knobs to adjust that in edge cases.

Let me give you a case in point. Codel and PIE both work quite well without tuning in intra-continental applications, which is to say use cases in which RTT is on the order of tens of milliseconds. The larger the RTT is, the longer it takes to adjust, but it gets there.

One problem we specifically saw with both algorithms as delays got larger - especially satcom - is most easily explained using Codel. Codel starts a season of marking/dropping when an interface has been continually transmitting for 100 ms, meaning that the queue never emptied in 100 ms. At that point, any packet that has been delayed for more than 5 ms has some probability of being marked or dropped. TCP, in slow start, works pretty hard to keep the buffer full - as full as it can make it. So now I’m moving a large file (iperf, but you get the idea), and manage to fill the queue with 50 ms of data and hold it there for 50 ms, with the probable effect of dropping a packet near the tail of that burst. Now, assume that I am on a geosynchronous satellite, so one way delay is on the order of 325 ms. Codel, dropping a packet 1/3 of the way into getting that going, had a horrible time even filling the link. PIE has the same issue, but the underlying mechanism is different. If I’m on that kind of link, I’m going to either turn AQM off or look for a way to tune it.

> That would be at least partially acceptable as a mode of operation.  But you might have a "warm-up" issue - would it work OK while the algorithm was working out what the RTT actually was?  And would the algorithms adapt autonomously (i.e., auto-tune) to close in on optimum values after picking initial values from the table?

Again, we have quite a bit of testing experience there, and the places the “warm-up” issue comes in are also the cases where RTT is large. For reasons 100% unrelated to this (it was a comment on ISOC’s so-called Measurement Project), yesterday I sent 10 pings, using IPv4 and IPv6 if possible, to the top 1000 sites in the Alexa list. Why 301 of them didn’t respond, I don’t know; I suspect it has something to do with the name in the Alexa list, like dont-go-here.com. But:

699 reachable by IPv4
median average RTT(IPv4): 0.064000
average loss (IPv4): 0.486409
average minimum RTT(IPv4): 111.923004
average RTT(IPv4): 116.361664
average maximum  RTT(IPv4): 124.621014
average standard deviation(IPv4): 4.043735

118 reachable by IPv6
median average RTT(IPv6): 28.748000
average loss (IPv6): 0.000000
average minimum RTT(IPv6): 67.267008
average RTT(IPv6): 69.580873
average maximum  RTT(IPv6): 74.213856
average standard deviation(IPv6): 2.133907

On a 100 ms timescale, the algorithms we’re discussing get themselves sorted out.

> [big chunk of text that I think was there by mistake dropped out]
> potentially helped a bad actor to pick off such flows or get to know who is communicating in a situation that currently it would be very difficult to know as the queueing is basically flow agnostic.  OK this fairly way out but we have seen some pretty serious stuff apparently being done around core routers according to Snowden et al.

Again, the algorithms I’m aware of are not sensitive to the user or his/her address; they are sensitive to his/her behavior. But again, unless we’re talking about fq_*, we’re not identifying flows, and with fq_* we’re only identifying them in the sense of WFQ. If, to pick something out of the air, PIE is dropping every 100th packet with some probability, and the traffic has 1000 “mouse” flows competing with a single “elephant”, I have a 1000:1 probability of hitting the elephant. That’s not because I don’t like the user or have singled him out in some way; it’s because he’s sending the vast majority of the traffic. So I don’t see the issue you’re driving at.

> I was just looking for somebody to define the jargon - As far as I am concerned at this moment "incast" would be just as "bad" since it would produce an equally blank stare followed by a grab for Google.

If a researcher had that reaction, his or her research wasn’t particularly relevant to data center operations. People working with Hadoop or other such applications are familiar with this in detail.

Imagine that I walked into Times Square on the evening of 31 December, and using a bull-horn asked everyone present to simultaneously lift their bull-horns and tell me how they were feeling. Imagine that they did. You now know what incast is and why it’s a problem. That’s what map/reduce applications do - they simultaneously open or use sessions to thousands of neighboring computers, expecting a reply from each.

But fine. You mentioned Google, so I asked Google about “data center incast”. I first asked just about “incast”. Merriam-Webster told me that it was in their unabridged dictionary but not the free dictionary. But asking about data center incast, I got pages and pages of papers on the topic. 

Gorry, one that might be worth pointing to would be http://www.academia.edu/2160335/A_Survey_on_TCP_Incast_in_Data_Center_Networks

---

From: [email-masked]
Date: Thu, 8 Jan 2015 09:40:27 -0000

I have made some suggested changes in-line below, please see if these
may resolve these issues.

Gorry

>> applicable.
> If Dave Taht replies, he’ll mention OpenWRT. Personally, I am thinking
> about ISP interconnect points and Access equipment such as BRAS and CMTS,
> which are indeed big iron, but also small CPEs such as might implement
> Homenet technologies and OpenWRT.
>
GF: I'm not sure this helps in the BCP - but has to be considered when you
define algorithms for methods - hence the recommendation for wide
applicability, but I tend to think the details need to go into any AQM
evaluation guidelines or the specs themselves.
---

>> I don't quite know what your target is here - or if you are thinking
>> over the whole range of sizes.  The responses below clearly indicate
>> that you have some examples in mind (Codel, for example which I know
>> nothing about except (now) that it is an AQM WG product) and I don't
>> know what scale of equipment these are really relevant to.
>>
> I’m still not entirely sure what you’d like to have said. Is this a one
> sentence “setting limits looser than necessary is neither harmful nor
> helpful”, or a treatise on the topic?
>
GF:
The text block I think this relates to is:
     "It is expected that the principles and guidance
      are also applicable to a wide range of environments, but may require
      tuning for specific types of link/network (e.g. to accommodate the
      traffic patterns found in data centres, the challenges of wireless
      infrastructure, or the higher delay encountered on satellite Internet
      links). "

- We could for instance add one or two more explicit examples here.
e.g. based on the text above... do we need examples also?
----
> Well, PIE basically came out of control theory (a basic equation that
> describes a phase-locked loop), and I believe that Van and Kathy will say
> something similar about Codel. But that’s not a question for this paper,
> it’s a question for the various algorithmic papers.
>
GF: I wonder if this is probably something that needs to go into specific
algorithm specs, not the BCP on the topic?
----
> AQM is absolutely fixated on buffer bloat. We have called it a lot of
> things over the years, none of them very pleasant, but the fundamental
> issue in RFC 2309 and V+K’s RED work was maximizing throughput while
> minimizing queue occupancy.
>
GF: I think the papers provide background, but I'm not sure they capture
the possibilities well - I suspect there is much more out there in products
patents and labs...The words further research, I think need to be read as
"encourage" research,

I suggest it may be best simply to remove /further/ in
this specific sentence.
----
> Network devices SHOULD use an AQM algorithm to measure local congestion
> and determine which packets to mark or drop to manage congestion. In
> general, dropping or marking multiple packets from the same sessions in
> the same RTT is ineffective, and can negative consequences. Also, dropping
> or marking packets from multiple sessions simultaneously has the effect of
> synchronizing them, meaning that subsequent peaks and troughs in traffic
> load are exacerbated. Hence, AQM algorithms should randomize dropping and
> marking in time, to desynchronize sessions and improve overall algorithmic
> effectiveness.
>
I agree on drop, but not mark. Because, multiple CE marks in the same RTT
could have usage with ECN - e.g. DCTCP. so I'd rather not prejudge what
may happen in this space - as the WG starts to work on new ECN methods.

I suggest:

"Network devices SHOULD use an AQM algorithm to measure local congestion
and determine which packets to mark or drop to manage congestion.

In general, dropping multiple packets from the same sessions in the same RTT
is ineffective, and can reduce throughput. Also, dropping or marking packets
from multiple sessions simultaneously can have the effect of synchronizing
them,
resulting in increasing peaks and troughs in the subsequent traffic load.
Hence, AQM algorithms SHOULD randomize dropping and marking in time,
to desynchronize sessions and improve overall algorithmic effectiveness."

- Is this useful?

----
> Well, I think you may have misunderstood the statement in the draft. The
> big problem with RED, mandated in RFC 2309, was that it couldn’t be
> deployed without case by case tuning. We’re trying to minimize that.
>
> Recommendation 3 is:
>
> The title of section 4.3, which is the section explaining recommendation
> 3, is "AQM algorithms deployed SHOULD NOT require operational tuning”.
> That’s not “MUST NOT”; if there is a case, there is a case. It goes on to
> say:
>
> We’re looking for “reasonable performance in the general case”, and
> allowing for the use of knobs to adjust that in edge cases.
>
> Let me give you a case in point. Codel and PIE both work quite well
> without tuning in intra-continental applications, which is to say use
> cases in which RTT is on the order of tens of milliseconds. The larger the
> RTT is, the longer it takes to adjust, but it gets there.
>
> One problem we specifically saw with both algorithms as delays got larger
> - especially satcom - is most easily explained using Codel. Codel starts a
> season of marking/dropping when an interface has been continually
> transmitting for 100 ms, meaning that the queue never emptied in 100 ms.
> At that point, any packet that has been delayed for more than 5 ms has
> some probability of being marked or dropped. TCP, in slow start, works
> pretty hard to keep the buffer full - as full as it can make it. So now
> I’m moving a large file (iperf, but you get the idea), and manage to fill
> the queue with 50 ms of data and hold it there for 50 ms, with the
> probable effect of dropping a packet near the tail of that burst. Now,
> assume that I am on a geosynchronous satellite, so one way delay is on the
> order of 325 ms. Codel, dropping a packet 1/3 of the way into getting that
> going, had a horrible time even filling the link. PIE has the same issue,
> but the underlying mechanism is different. If I’m on that kind of link,
> I’m going to either turn AQM off or look for a way to tune it.
>
GF: I don't know what to change.
----
> Again, we have quite a bit of testing experience there, and the places the
> “warm-up” issue comes in are also the cases where RTT is large. For
> reasons 100% unrelated to this (it was a comment on ISOC’s so-called
> Measurement Project), yesterday I sent 10 pings, using IPv4 and IPv6 if
> possible, to the top 1000 sites in the Alexa list. Why 301 of them didn’t
> respond, I don’t know; I suspect it has something to do with the name in
> the Alexa list, like dont-go-here.com. But:
>
> 699 reachable by IPv4
> Again, the algorithms I’m aware of are not sensitive to the user or
> his/her address; they are sensitive to his/her behavior. But again, unless
> we’re talking about fq_*, we’re not identifying flows, and with fq_* we’re
> only identifying them in the sense of WFQ. If, to pick something out of
> the air, PIE is dropping every 100th packet with some probability, and the
> traffic has 1000 “mouse” flows competing with a single “elephant”, I have
> a 1000:1 probability of hitting the elephant. That’s not because I don’t
> like the user or have singled him out in some way; it’s because he’s
> sending the vast majority of the traffic. So I don’t see the issue you’re
> driving at.
>
GF: I don't know what to change (if any).
---
> If a researcher had that reaction, his or her research wasn’t particularly
> relevant to data center operations. People working with Hadoop or other
> such applications are familiar with this in detail.
>
> Imagine that I walked into Times Square on the evening of 31 December, and
> using a bull-horn asked everyone present to simultaneously lift their
> bull-horns and tell me how they were feeling. Imagine that they did. You
> now know what incast is and why it’s a problem. That’s what map/reduce
> applications do - they simultaneously open or use sessions to thousands of
> neighboring computers, expecting a reply from each.
>
> But fine. You mentioned Google, so I asked Google about “data center
> incast”. I first asked just about “incast”. Merriam-Webster told me that
> it was in their unabridged dictionary but not the free dictionary. But
> asking about data center incast, I got pages and pages of papers on the
> topic.
>
> Gorry, one that might be worth pointing to would be
> http://www.academia.edu/2160335/A_Survey_on_TCP_Incast_in_Data_Center_Networks
>
I suggest:

"        <t>Traffic patterns can depend on the network deployment
        scenario, and Internet research therefore needs to consider
        the implications of a diverse range of application interactions.
        At the time of writing (in 2015), an obvious example of further
        research is the need to consider the many-to-one communication
        patterns found in data centers, known as incast <xref
        target=“REN12”></xref>, (e.g. produced by
        Map/Reduce applications). Research also needs to consider the need
        to extend our taxonomy of
        transport sessions to include not only "mice" and "elephants", but
        "lemmings"? Where ”Lemmings" are flash crowds of "mice" that the
network
        inadvertently tries to signal to as if they were elephant flows,
        resulting in head of line blocking in data center applications.</t>"

----

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 9 Jan 2015 21:30:04 +0000

Your suggestions are all fine with me. When it comes to you not knowing what to change, I don’t know either. That ball’s in Elwyn’s court.

> On Jan 8, 2015, at 1:40 AM, [email-masked] wrote:
> 
>> If Dave Taht replies, he'll mention OpenWRT. Personally, I am thinking
>> I'm still not entirely sure what you'd like to have said. Is this a one
>> sentence "setting limits looser than necessary is neither harmful nor
>> helpful", or a treatise on the topic?
>> 
> GF:
>> something similar about Codel. But that's not a question for this paper,
>> it's a question for the various algorithmic papers.
>> 
>> issue in RFC 2309 and V+K's RED work was maximizing throughput while
>> Well, I didn't think it needed a lot of saying. :-) How about this?
>> 
>> big problem with RED, mandated in RFC 2309, was that it couldn't be
>> deployed without case by case tuning. We're trying to minimize that.
>> 
>> Recommendation 3 is:
>> 
>> 3, is "AQM algorithms deployed SHOULD NOT require operational tuning".
>> That's not "MUST NOT"; if there is a case, there is a case. It goes on to
>> say:
>> 
>> We're looking for "reasonable performance in the general case", and
>> I'm moving a large file (iperf, but you get the idea), and manage to fill
>> but the underlying mechanism is different. If I'm on that kind of link,
>> I'm going to either turn AQM off or look for a way to tune it.
>> 
>> "warm-up" issue comes in are also the cases where RTT is large. For
>> reasons 100% unrelated to this (it was a comment on ISOC's so-called
>> possible, to the top 1000 sites in the Alexa list. Why 301 of them didn't
>> respond, I don't know; I suspect it has something to do with the name in
>> On a 100 ms timescale, the algorithms we're discussing get themselves
>> sorted out.
>> 
>> Again, the algorithms I'm aware of are not sensitive to the user or
>> we're talking about fq_*, we're not identifying flows, and with fq_* we're
>> traffic has 1000 "mouse" flows competing with a single "elephant", I have
>> a 1000:1 probability of hitting the elephant. That's not because I don't
>> like the user or have singled him out in some way; it's because he's
>> sending the vast majority of the traffic. So I don't see the issue you're
>> driving at.
>> 
>>>> looking for the word "incast" to replace "map/reduce"?
>>> 
>> If a researcher had that reaction, his or her research wasn't particularly
>> now know what incast is and why it's a problem. That's what map/reduce
>> But fine. You mentioned Google, so I asked Google about "data center
>> incast". I first asked just about "incast". Merriam-Webster told me that
>        target="REN12"></xref>, (e.g. produced by
>        "lemmings"? Where "Lemmings" are flash crowds of "mice" that the
> network