Subject: [ledbat] missing minutes

From: Murari Sridharan <[email-masked]>
Date: Wed, 13 Jan 2010 17:17:22 +0000

All, I was fairly sure that the merged minutes from Alissa and Eric's notes were posted right after Hiroshima before I left. I am attaching the raw notes. There are several action items from the meeting that we need to act on, primarily a document update. In Eric's notes the action items are marked with a ****.

I will try to post the merged notes that I have to the tools site.

Thanks

Subject: LEDBAT minutes
From: Eric Burger <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Date: Fri, 13 Nov 2009 11:04:30 +0900

LEDBAT IETF 76, 13 November 2009

Stanislav Shalunov, Murari Chairs

Eric Burger, Scribe

=====

Summary Action items. Items are marked with “****” in the narrative.

LEDBAT document:

Document why no framing in LEDBAT document

Need to document need for real randomness in algorithm. Both in terms  
of implementation (don’t count on disk rotation latency, etc.) and  
theory (non-convergence).

Document late-comers ADVANTAGE

Explain where constants come from. Give a single, specific value with  
a range.

Recommendations document:

Split out DIFFSERV into a different document (if people care about  
DIFFSERV)

Be absolutely sure to clarify difference between practices that work  
around bugs versus practices that are good to follow. Specifically,  
approaches to work around broken HTTP implementations (e.g., no  
pipelining)

Please send text; formulate things in terms of “It would be in your  
interest NOT to do xxx” instead of “You SHOULD do yyy.”

Low Extra Delay Background Transport: Stanislav Shalunov

=====

Framing

-----

Rich Woundy: Transport level may or may not have transport  
identifiers. Depending on transport protocol, you will need to be able  
to identify stream.

Slav: need it for fine-grain monitoring. However, "this is the last  
latency I have seen" should be good enough. Reording should not be  
much of an issue.

Rich: worth highlighting order issue. If you get stuff in the order  
3-2-1, you will think packet 3 has great latency, 2 has normal  
latency, 1 has awful latency.

Slav: if measurement error is small, should not impact algorithm. If  
lots of variance, then ledbat algorithm has failed.

Murari: if I am doing framing, why would I make it sequence-number  
specific?

Slav: not a stream-domain protocol.

Lars: Not doing framing, but should note how to frame: good, bad,  
recommended. Should be in the document. Not normative, but  
suggestions. ****

Late comer's (non)-advantage

-----

Lars: implementation depends on side effects on user's operating  
system, and looks like only might work by serendipity. It seems to be  
just a hack that might not work all the time.

Slav: if your machine is so good, then you can introduce noise /  
random fluctuations in inter-packet transmission time.

Stewart: As world moves to flash disk, high-performance CPUs, timers  
are more and more precise. Cannot depend on the end system introducing  
random noise.

Murari: problem is non-existent on many systems.

Rich: Assumption about a backoff time is random. Need to clearly state  
expectation that things back off with a random component. The random  
component is an implementation detail. ****

Lars: Better to have something explicit in the document.

Slav: Need to flush queue to figure out RTT on second stream to really  
know.

Lars: That is too heavy weight (Slav clearly agrees)

Bob Briscoe: Late-comer's advantage: please add it to the draft as  
described on the list.  Disadvantage: late-comer does not know base  
delay. Advantage: if it does know base delay, other flows will not  
necessarily get out of the way of the new flow. Draft says randomness  
fixes this, but that means randomness needs to be ensured ****

LNS

Fairness

-----

Bob: Need to write down assumptions. Random walks with a lot of flows  
competing with each other; do not see how ones will a small amount of  
bandwidth will ever get large or vice versa. Need to state why  
randomness helps and importance of true randomness in the algorithm.  
****

Andrew: randomness protocol adds needs an estimator to know what  
randomness is available so it knows how much randomness to add.

Bob: May be we see randomness in networks because today’s protocols  
have randomness built in. If on a ledbat-only network, no randomness.  
E.g., data center

Slav: experience is randomness still there.

Stuart: every network does NOT have randomness. Slav: every network I  
have TRIED has randomness, but not necessarily true for all networks.  
Stuart: MacOS 8.6 (?) had this problem; throughput cut in half. No  
randomness, so everyone had the same collisions repeating.

Bob: just need to note this in spec.

Parameter Values

-----

Bob: very uncomfortable about target delay at 25ms

Murali: if we are not strict (MUST), then bad actors can set values  
better for themselves but worse for the network

Lars: bad actors can always chose, e.g., low TARGET values

Lars: This is the "Magic Numbers" slide. Numbers do not seem to have  
any justification. OK for experimental protocol. Need a lot of data on  
what we think will be eventual PS solution.

Slav: can get rid of the NOISE_FILTER number; almost always will be 1  
packet, which means it is not considered in the calculation. Today,  
these are ranges because we do NOT know what the value should really  
be. That is why we are doing the experiment.

Jana: these values came from somewhere. How about: “Here is a  
mechanism, and here is a constant. Here are the implications of moving  
the values.” Should explain all of the constants this way. **** How  
can I see if the protocol works if different implementations use  
different parameters. Where is the failure, if any? Implementers will  
just copy and paste from spec, so why not give preferred values?

Lars: “It should be this number, but here is the range.”

Andrew: Bottom 5 numbers are consequences of the choice of the first  
number. However, the first number seems to be pulled out from nowhere.

Slav: Yes, it really IS a magic number.

Implementation

-----

Rich: Has anyone done an ns2 or ns3 model.

Slav: some told in private, but nothing public.

Murali: one implementation known of.

General Questions

-----

Jana: Do we have data we can look at data from BitTorrent  
implementation?

Slav:  Can distribute test executable for people. Not open source.

Jana: Not using TCP mechanisms. There should be discussion about  
burstiness considerations.

Slav: spent lots of time looking at rate-based, window-based, etc.  
methods. Not sure how much should be in the spec. Since spec is for  
implementers, not researchers. Do not need it.

Lars: actually, you do need to figure out, e.g., when to send  
feedback. Perhaps “Framing Considerations”?

Jana: Need to have some mechanism for spreading out (not bursting) data.

Slav: Thought about it a lot. There may be something in their  
implementation, but not there today. Discuss off-line.

Bob: Note: we do not know that BitTorrent implementation of ledbat is  
ledbat.

Slav: Feel free to write an implementation.

LEDBAT Applications, Practices, and Recommendations. (Reinaldo Penno)

=====

Lars: What do you mean by a control connection? Is it a limited  
application connection?

Reinaldo: It is not a bulk data connection.

Vijay: Control connection could be, e.g., SIP; data could be RTP. HTTP  
does not have that distinction. FTP does. Control connection exchanges  
parameters to setup bulk data transfer connection.

Lars: this is not the obvious place to discuss DIFFSERV. Might want to  
split out to another document. ****

Recommendations

-----

Bob: Scope of draft is about application design. One aspect is whether  
you give user control. Not in document. Could be important to document.

Six connections: “A number between 4 and 8.” Jana: How does  
application know first connections are not saturated.

Reinaldo: Because the application does not see packet loss.

Stuart: Need to look at studies, but on his MacBook he can saturate  
his GE interface with a single TCP connection. TCP fast retransmit  
works better with one connection than with multiple connections.

Reinaldo: multiple connections good for head-of-line blocking.  
Described in the draft.

Jana: purpose of multiple connections was for head-of-line blocking,  
not for getting more-than-fair network bandwidth. Might get more  
bandwidth, but not guaranteed and not the purpose. Should not be a  
consideration for this work group.

Slav: Set your Firefox browser to 32 connections per server. Looks  
like win is from parallizing latency from server response.

Bob: Would it be better to say things like “It would be in your  
interest NOT to xxx” instead of saying “You SHOULD do yyy.” ****

Jana: +1. Need to wordsmith carefully, as implementors can misconstrue  
document to say, “I can always open 6 connections.”

Reinaldo: Please send text ****

Concurrent Connections

----

Stuart: Is question how to generically deal with performance, or how  
to deal with HTTP servers that cannot do pipelined requests? Good HTTP  
guidance, but not general guidance. You get better performance with a  
server that does pipelining with a single connection than multiple  
connections.

Slav: Does no harm

Murali: about documenting current practices.

Stuart: Important to highlight HTTP issue, not an IP issue.

Reinaldo: Important to document why using multiple connections for HTTP.

Stuart: Useful to describe what is being done to work around bugs  
versus truly best practices. ****

Bob: Often confusion about problems with half-open connections on  
operation systems.

Murali: Anyone do not want to adopt this document as a WG document?

RESULT: silence

Murali: Anyone want to adopt this?

RESULT: consensus

Jana: +1 to Stuart about being careful distinguishing current practice  
versus recommendations ****

Goal: have parameter values by Anaheim (IETF 77). Please review pseudo  
code. Discuss on list.

At least one implementation report by Anaheim.

Also: do not have implications of opening multiple connections on  
devices with low power constraints.

Subject: Raw notes from LEDBAT
From: Alissa Cooper <[email-masked]>
To: Murari Sridharan <[email-masked]>, Stanislav Shalunov <[email-masked]>
Date: Fri, 13 Nov 2009 10:58:02 +0900

Here you go.
-Alissa

------------

LEDBAT

Low Extra Delay Background  Transport (LEDBAT), Stas Shalunov
http://tools.ietf.org/id/draft-ietf-ledbat-congestion-00.txt

Stas
TCP congestion control designed to maximize throughput, but not
address latency
if you ignore one characteristic, may be fine, may not be
algorithms can be absed on either characteristic or a combination
TCP based on throughput
Stas approach: keep latency fixed

draft has new name: draft-ietf-ledbat-congestion
were several open issues, the one non-trivial one was addressed: non-
congestive constraint

non-congestive constraint:
app may be rate-limited itself
in this case, congestion may not be encoutered for an extended period,
queuing delay is zero
congestion control would continue to climb forever
if congestion is later encountered, app will have to start from a
point where it thinks it can send much more than it should
not only will delay be high, but app will encounter loss --> sub-optimal
sender should never make assumptions to deviant from what it has done
in the past
proposed solution: flight size
flight size (fs) = amnt of outstanding data (sent but not yet
acknowledged by the recipient)
fs only meaninful to the sender
fs is effective window *right now*
keep congestion window on a leash attached to fs
cwnd <= ALLOWED_INCREASE + TETHER*fs where TETHER is your leeway
parameter
tether needs to allow for at least one more packet to allow you to
probe, so can't be < 2
but tether < 2 may be too generous once you've ramped up, so we add
additive constant (ALLOWED_INCREASE)
so tether must be > 1, Stas recommends at least 25% increase
ALLOWED_INCREASE and TEHTER are constant
A_I must be at least 1 packet, but not more than 3 packets
TETHER must be > 1, should be between 1.25 and 2
fs is implemented in all BitTorrent products that use uTP

known document issues:
framing is out of scope -- discussed during charter discussion
need to capture discussion about late comer's advantage (lack thereof)
need to discuss whether we have fairness between ledbat flows
need to discuss choice of parameter values, e.g., when there are
ranges, what tradeoffs are between either end of range
check for loose ends in pseudocode and spec -- need new eyes on this

framing out of scope:
wire representatin is out of scope
but need to be able to carry two quantities: time stamp and delay
without time stamp, can't measure one-way delay
recipient needs to be able to convey info about one-way delay to sender
two quantities go in opposite directions (sender->recipient and vice
versa)
current BT implementation uses UDP, proprietary framing
could use this with TCP, but embedding two quantities is not easy in
TCP header
Rich Woundy: transport level may ahve apcket IDs or it may not, so
transport protocol may have to declare which packet it's talking about
Stas: you could do that for assurance against extra re-ordering, but
you don't have to, can just imply the delay you last saw (delay in
each direction is likely similar), so this is probably not a requirement
Rich: may not be a req, just worth pointing out
Murari: out of order should not matter
Rich: worried about delay seesaw-ing (3,2,1 over and over getting
mixed up)
Stas: re-order can be from sender to receiver or vice versa; former
will result in measurement noise and if it's small, should be fine (if
large, delay-based protocol will fail anyway); in latter, packet IDs
could add extra value; BT implementation has the extra value of ACKs
Murari: if I'm framing anyway, why wouldn't I make it sequence number-
specific?
Stas: but this doesn't need to be a stream transmission protocol
Rich: but it may not be TCP. but this is fine.
Lars Eggert: point is that we're not doing framing, but can add
considerations for what framing protocol needs to do (not reqs, reqs
are too heavy)
Stas/Murari: agreed

late comer's (non) advantage:
first ledbat connection builds up a target worth of queue, reaches
steady state
second connection comes in, starts measuring delay, doesn't know base
delay
firs connection sees more larger queue, second connection is at
advantage of being able to get an extra target's worth of packets
two reasons why this doesn't happen in the protocol
1) dips in queue are enough to measure base delay; dips come from
machine hiccups
2) even without dips, once first connection backs off, second
connection is now exclusive network user and can't be deceived
Lars: worried about this. for the good operation of this mechanism we
depend on side effects of the computer, or we say even when it goes
wrong it doesn't go wrong for long. can we find a mechanism that gives
the second connection a better example quicker? e.g., every 5 seconds
stop sending (bad suggestion)
Stas: or every minute might be feasible. implementors might know what
kind of machines you're going to run on.
Stuart Cheshire: what if you have a dedicatd special-purpose device
with flash-based memory and no disk? data rate can be absolutely
constant. can't expect randomness. have discussion of how much
randomness is necessary.
Murari: position that ICCRG has taken is that if your protocol is only
delay-based, they've been very particular about this.
Rich: perhaps we cna say there's an assumption of a back-off time but
implementation can be based on some randomness. how you back off
periodically is not important, just that it happens.
Stas: agreed.
Murari: can do even better than randomness probably.
Lars: having something in the mechanism would be better bc even as
implementor you don't know where it will get deployed.
Stas: not much choices exists in design space. need to transmit info
about base delay through the queue. must empty the queue, and to that
you shut off for a RT time.
Murari: have to do something periodically.
Lars: isn't that too heavyweight? something else we can do? jitter the
delay measurement?
Bob Briscoe: late comer's disadvantage: has been discussed on list.
should be added to the draft. late comer's disavantage = other flows
don't get out of the way for new flows. they see increase in delay as
does the new flow.
Stas: this is the fairness issue, coming up next.

fairness between ledbat flows:
how do multiple ledbat connections share between themselves?
mechanism converges, so there's no probing
convergence time is proportional to rate at which bottleneck is sending
fairness controller in XCP used to inform this mechanism: add random
noise to delay measurement before using it, and suppose that value of
radnomness is fixed fraction of target delay
something proportional to this fraction will now be distributed each
RT time between connections
Bob: we need to write down assumptions. with lots of flows competing
with each other, no guarantee that random walk will even things out.
you only get that when they bounce off either edge.
Stas: with constrained random walk, you converge.
Andrew McGregor: if bottleneck queue is adding randomness
Stas: that's good for redistribution.
Andrew: other suggestion is that if randomness is too low,
deliberately increase it
Stas: have not seen a situation where there isn't enough randomness.
reluctant to add an untested code path. suggested mechanism is too
smart.
Bob: echoing Stuart, we may see randomness bc protocols are designed
to put randomness in. what about when protocol is running against
itself? what about using ledbat in data center?
Stuart: every network does not have randomness. shipped a Mac OS
update with this exact problem over ISDN in France. bottleneck was
very precisely clocked, TCP safety margin went to zero, triggered
retransmission, every packet got sent twice, performance cut in half
for 10,000s customers.
Stas: understood. should add aritficial randomness.
Bob: we are doing this as experimental, so don't have to prove all of
it ahead of time.

choice of parameter values:
current reqs may be unecessarily strict, target and gain both fixed
target must be 25 ms or more (and should be < ??)
gain must be 1 mss/rtt, but what if you ramp up slower?
Bob: very uncomfortable of target delay of 25 ms. once we standardize,
if we want to get rid of those delays.
Stas: get rid of 8 s of delay first, then we can get rid of last 25 ms.
Bob: just need to think about how we can get rid of 25 ms.
Murari: if we're not strict, we create unfairness.
Stas: yes, allow people to set values that are bad for them if they
compete with standard values.
Lars: not sure if there will be competition between ledbat flows, but
you could win by changing target.
Stas: in BT, there is competition, several connections share same
bottleneck (home uplink). 4-5 uploading connections split it up based
onr andom walk.

2 <= base history <= 10: higher values give better measurements of
base delay if neither of the other factors is active
but there may be situations to deviate from sweet spot (perfect
clocks, no rerouting)

1 packet <= noice filter <= cwnd/2 (should): at low end, you get
faster reaction and lower delay in control loop -> more stable,
converge faster.
probably not a good idea to make noice filter more than 1 cwnd (1 RTT)
bc you will double your response time.

Lars: I hate magic numbers. these are all magic numbers. would like to
elminate some of them or understand better what they should be.
eventually we will try to make this a proposed standard, so we will
need a lot of data on what these should be set for general internet
conditions. might be better to exclude cases where you say "under
controlled environment" -- charter discusses deployment on general
internet.
Stas: that's why we use SHOULD. they're not banned, they're not
required. we could get rid of the noise filter, bc in majority of
situations you want it to be 1 packet. but others can't really be
removed.
Lars: it's not only magic numbers, it's also magic ranges. might need
experimentation for some of them.
Stas: I can't seriously say that I know what the tether value should
really be, that's why it has a range. within the ranges, all values
would work on internet deployment.
Jana Iyengar: one point of experimental protocol is to see if it
works, how can we specify a range and see if it works? they will all
be using different parameters.
Murari: so you want us to pick some value?
Stas: SHOULD value that is not a range?
Jana: lay down a consideration of the extremes, and then give
suggestions of what constants are likely to work. instead of using
SHOULD (not as strong).
Lars: Jana is saying, it SHOULD be X, and here's what happens when you
go lower or higher.
Stas: need something between SHOULD and MUST
Lars: SHOULD REALLY
Andrew: target is a magic number and all the others are consequences
of target. but why 25 ms?
Stas: we had extensive discussion of this at last meeting.

careful proofreading:
need implementors to read (esp section 5)
best way to read is to implement
Rich: has anyone done an NS 2 or NS 3 model?
Murari: I will send report to the mailing list of the one that I know
of.
Rich: might be interesting to take the model and run it against
different network topologies.
Vijay Gurbani: have implemented BT in NS 3. doesn't have docsis but
does have a wireless model.
Stas: and don't need to simulate in context of BT, because that has
been done already.

Q&A
Jana: can we look at data about BT?
Stas: Bt carries most of the congestive data on the internet. it's a
real implementation.
Jana: would like to see some data to decide if it works.
Stas: client is not open source, but if people need text executable,
Stas can give that out.
Rich: can we send that to the mailing list?
Stas: I will investigate. but you can get it privately.

Jana: no discussion of burstiness in algorithm.
Stas: tried rate-based vs window-based, but not sure how much of it
should be in a spec. can discuss what has been considered and whether
it worked, but target audience of the draft is implementor, not
researcher.
Lars: there are implications on framing. how often do you send the
feedback?
Murari: may be more than framing.
Jana: was talking about burstiness of bulk application. if delay is
distributed across multiple queues, that's a reason to spread your
data out. pacing or something. not bursting. need to know what amount
of burst is reasonable.
Stas: discuss offline.
Jana: I think it should go in the draft.

Bob: we don't know if the BT implementation is a ledbat implementation
bc it's not open source.
Stas: go ahead.

LEDBAT Practices and Recommendations for Managing Multiple Concurrent
TCP Connections, Reinaldo Penno
http://tools.ietf.org/id/draft-penno-ledbat-app-practices-recommendations-01.txt

draft was written because of concern about apps opening multiple TCP
connections
new draft has recommendations section

comments from IETF74 onwards:
differentiate between multiple control and data connections
DiffServ may or may not be in scope
rule of thumb about how many connections are too much?
reassembly at app layer. not in spec right now

multiple control vs data connections:
new section 3
http can open many data connections
protocols on top of http open multiple control and data connections
BT, Skype use multiple control and data connections
Lars: what does control connection mean?
Vijay: control connection might be the signaling part in SIP; HTTP
doesn't have the differentiator
Lars: it's not about one being about xfer and the other not.
Vijay: FTP is another example. control connection is there to exchange
meta data, data connection to exchange file.

DiffServ:
section 6.1: recommendation to use RFC 3662 to mark packets for bulk
transfer
WG input necessary to decide if this should be in there
Lars: this document is not an obvious place to put this. could spin it
out into stand-alone document or whether the WG will take it on.

how many connections are needed:
new section 6.3: many recent studies on parallel TCP connections
can't make recs for control connections bc they make the protcol work

disadvantage due to reassembly on top of TCP at app layer:
some studies exist about this, need to add text about this in next rev

Bob: draft is about application design. one aspect of design is
whether you give the user control over this. should be mentioned.

recommendations:
rec-1: bulk data xfer should be DiffServ marked

rec-2: window should be scaled

rec-3: apps should only open more than 6 connections to download the
same object if the first hop link is not saturated.
browser usually open more than 4, some studies show going up to 8. 6
is a compromise between 4 and 8.
Jana: for app to open 6 connections, it should know link is not
saturated, how does it know?
Reinaldo: app does not see packet drops, so knows link is not saturated.
all the studies have constraints. congestion can be near the server.
there are lots of reasons to use multiple connections, listed in the
draft.
Stuart: GB ethernet can be saturated with one connection. browser
designers thought TCP fairness was based on flow fairness, not
realizing that one pipeline can give better performance.
Reinaldo: a lot depends on head of line blocking and whether one
persistent connection will slow down everything else or not. TCP
usually goes up to 75% of a link.
Stas: if you type about:config into Firefox and search for connections
and there's a network.http.max-connections and value of default is 15.
can change that number to 32 and it will go faster for you.
Jana: browsers do not open multiple connections for bandwidth, but for
head of line blocking. badnwidth is a side effect.
Bob: maybe better than SHOULD is say it's "unlikely to be in your
interest" for designers.
Murari: purely for throughput reasons, we should not recommending
opening multiple connections.
Jana: wary of IETF saying "apps can open 6 connections"
Reinaldo: suggest text, please.
Firefox, Internet Explorer
Cloud Four: has 4 domains, asks people to access the page from major
mobile browsers
some browsers open 40 connections (Opera), next version down to 5
connections
Stuart: Safari opens multiple connections because some percentage of
webs ervers can't handle pipelines requests. this might be good HTTP
guidance, but not sure if it's good for broader applications guidance.
Stas: internet is not blowing up because of multiple connections. it's
the de facto practice.
Stuart: important to distinguish HTTP servers from others
Reinaldo: draft has lots of discussion of why browsers do multiple
connections
Bob: lots of confusions about OS limits on half-open connections.
Windows limit used to be 10. RFC 5290 says that these limits exist and
it doesn't say they are for half-connections. draft should distinguish
between connections and half connections.

rec-4: ?

to dos:
bi-directional HTTP recommendations?
reassembly at app level

Murari: hum: don't adopt document?
result: no hums
hum: adopt as WG doc?
result: consensus in favor

Jana: need to be careful about observing what happens and saying what
we recommend
Reinaldo: can split into subsections what we observe today and what
future recs are

Wrap-up: Murari
goal for Anaheim IETF: finalize key pieces of the algorithm
want to have implementation report on the agenda for the next meeting
app practices doesn't discuss power cosndierations for constrained
devices (mobile)

---

From: Murari Sridharan <[email-masked]>
Date: Wed, 13 Jan 2010 18:09:10 +0000

The minutes have been uploaded as well.

Thanks

From: [email-masked] [mailto:[email-masked]] On Behalf Of Murari Sridharan
Sent: Wednesday, January 13, 2010 9:17 AM
To: [email-masked]
Subject: [ledbat] missing minutes

Thanks

---

From: Murari Sridharan <[email-masked]>
Date: Thu, 21 Jan 2010 09:27:18 +0000

Document authors, we need updated documents based on the minutes before we can have further discussions. Can you please provide an ETA for the same?

From: [email-masked] [mailto:[email-masked]] On Behalf Of Murari Sridharan
Sent: Wednesday, January 13, 2010 10:09 AM
To: [email-masked]
Subject: Re: [ledbat] missing minutes
Importance: Low

Thanks

Thanks