Subject: [p2pi] discussing P2PI-related standardization in Dublin

From: Lars Eggert <[email-masked]>
Date: Fri, 6 Jun 2008 15:58:11 +0200

Hi,

the recent workshop on P2P infrastructure (P2PI) has
resulted in several suggestions for future IETF work. Two of
the suggested work items are related to core TSV topics, and
a third has at least strong ties to TSV (but also to other
areas).

We'd be interested to hear from the community if we should
schedule a BOF in Dublin to discuss whether the IETF should
take on some or all of these, and if so, whether TSV would
be the right place for this work.

I'll be CC'ing this email to several lists. Please direct
your replies to the TSV area list ([email-masked]), so
that the discussion happens in one place.

The working title for the BOF is "Techniques for Advanced
Networking Applications" or TANA. The high-level idea is
that the IETF would develop guidelines, recommendations and
mechanisms for advanced applications that make intensive use
of Internet communication in a way that goes beyond the
capabilities offered by the IETF's current standard
end-to-end transport protocols. Such network-centric
applications include those that transmit delay-sensitive
traffic and those that transmit delay-insensitive traffic,
both using either peer-to-peer approaches or more
traditionally structured communication patterns. The work
would focus on broadly applicable techniques that address
the needs of large subsets of these class of network-centric
applications in a fashion that is independent of any
particular application.

The following three work items were identified as potential
new work for the IETF at the P2PI workshop. The first two
would be pretty clearly in scope for the TSV area, whereas
the third has some significant inter-area relations:

(1) A congestion control algorithm for
     less-than-best-effort "background" transmissions, i.e.,
     an algorithm that attempts to scavenge otherwise idle
     bandwidth for its transmissions in a way that minimizes
     interference with regular best-effort traffic. Among the
     desired features of such an algorithm are the ability to
     maintain short standing queues, the capability to quickly
     yield to regular best-effort traffic that uses standard
     congestion control, the ability to utilize explicit
     congestion notification (ECN), active queue management
     (AQM), and/or end-to-end differentiated services
     (DiffServ) where available, as well as effective
     operation in situations where some or all of these
     mechanisms are not available. In addition to specifying a
     congestion control algorithm, the work may also include
     specifications of how the algorithm is used in specific
     UDP-based protocols. (Application of the algorithm to
     other transport protocols is expected to occur in the
     working groups that maintain those protocols.) This work
     item has relations to IETF WGs that deal with congestion
     control, such as TCPM, DCCP and TSVWG, as well as the
     IRTF's congestion control research group (ICCRG).

(2) A guidelines document that discusses the tradeoffs
     surrounding the use of many concurrent transport
     connections to one peer and/or to different peers.
     Standard Internet congestion control results in a state
     where different transport connections end up sharing
     bottleneck capacity, and when an application uses several
     transport connections to transfer through a bottleneck,
     it tends to end up with a larger fraction of the
     bottleneck's loaded resource than if it would be using
     fewer connections. Note that although capacity is the
     most commonly considered bottleneck resource, middlebox
     state table entries are a second important resource for
     end system communication. Other resource types may exist,
     and the guidelines are expected to comprehensively
     discuss them.

(3) A mechanism that lets an application that can
     transfer data from or to several potential peers (i.e.,
     servers, caches, end systems) select a subset of peers
     for efficient transmission in a way that is aligned with
     the dynamic interconnection structure of the network
     operators that provide connectivity to those peers.
     Application designers, network equipment vendors and
     network operators will need to collaborate on this work
     item to define what kinds of dynamic interconnection
     information is useful to applications, how to obtain it,
     and how to provide it to applications, resulting in a
     generic mechanism that is broadly applicable to many
     current and future applications. This work item has
     obvious interactions with the IETF's Application, Routing
     and Operations areas. In order to make this effort more
     manageable, it may make sense to work out the
     requirements for such a solution first, before discussing
     individual proposals or breaking up the work into pieces
     for specification in different areas or WGs.

Please send your thoughts on these three topics.

These three work items aren't necessarily the only ones that
the IETF could think about taking on; they merely appeared
to be the most concrete ones coming out of the P2PI
workshop. We could think about discussing additional work
items in Dublin - please write up a short description
(similar to the ones above), so we can discuss them.

Lars

_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi

---

From: Lars Eggert <[email-masked]>
Date: Fri, 6 Jun 2008 16:40:36 +0200

Hi,

related to the work item below, I've just heard that some folks had  
already been discussing the submission of a more targeted BOF  
proposal. Depending on the details of that proposal (which I haven't  
seen yet), work item (3) might be subsumed by it.

Lars

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Fri, 06 Jun 2008 11:36:17 -0500

>      an algorithm that attempts to scavenge otherwise idle
[...]

I ran across the following references today that I thought
may be pertinent to congestion control in DHT algorithms.
I do not think they operate on the scavenger principle,
nor do I remember them as references in the P2Pi workshop
set of papers.

Fabius Klemm, Jean-Yves Le Boudec, Karl Aberer: Congestion
Control for Distributed Hash Tables. The 5th IEEE International
Symposium on Network Computing and Applications (IEEE NCA06),
Cambridge, MA, USA, July 24 - 26, 2006 (available online at
http://lsirpeople.epfl.ch/klemm/Congestion%20Control%20for%20Distributed%20Hash%20Tables.pdf)

Fabius Klemm, Jean-Yves Le Boudec, Dejan Kostic, Karl Aberer:
Improving the Throughput of Distributed Hash Tables Using
Congestion-Aware Routing. International Workshop on Peer-to-Peer
Systems (IPTPS'07), Bellevue, WA, USA, February 26.-27, 2007
(available online at
http://lsirpeople.epfl.ch/klemm/Improving%20the%20Throughput%20of%20Distributed%20Hash%20Tables%20Using%20Congestion-Aware%20Routing.pdf)

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
2701 Lucent Lane, Rm. 9F-546, Lisle, Illinois 60532 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
WWW:   http://www.alcatel-lucent.com/bell-labs
_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi

---

From: "Peterson, Jon" <[email-masked]>
Date: Mon, 9 Jun 2008 16:40:07 -0400

The proposal for the ALTO BoF (which roughly covers (3)) is now in the
BoF tracker. You can read about it at its home page:

http://alto.tilab.com/

ALTO proposes to explore a slightly broader space than merely P2P
file-sharing applications, since discovery of nearby peers has equal
applicability to many real-time communications protocols (for example,
the discovery of a TURN server to assist the establishment of a VoIP
session). Although RAI proposes to sponsor a BoF on this subject, we're
not yet persuaded that RAI would be the optimal home for any ongoing
work efforts related to this space - we hope that discussions
surrounding the BoF could elucidate the proper division of labor and
approach to this problem.

I do believe this work effort is sufficiently distinct from the
transport issues discussed in (1) and (2) of Lars' original mail to
merit a separate effort. I do, however, whole-heartedly support TSV
taking on (1) and (2) in a forum like TANA. 

Jon Peterson
NeuStar, Inc.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Lars Eggert
Sent: Friday, June 06, 2008 7:41 AM
To: TSV Area
Cc: [email-masked]; [email-masked] IAB; [email-masked]; IESG ((E-mail));
P2PSIP Mailing List
Subject: Re: discussing P2PI-related standardization in Dublin

Hi,

Lars

---

From: "Livingood, Jason" <[email-masked]>
Date: Mon, 9 Jun 2008 17:01:41 -0400

Exceptionally minor comments inline.

> take on some or all of these, and if so, whether TSV would be 
> the right place for this work.

A BoF or two would be good, IMHO.  Some of this probably fits into TSV
too, so confiming that this seems on the right track to me at least.

> Networking Applications" or TANA. 

So far we have TANA and ALTO, plus referring some items into TSV, which
again, seems appropriate and useful.  I do not have anything else to add
to this list for now; perhaps discussion @ IETF-72 will reveal further
needs.

Jason

---

From: Enrico Marocco <[email-masked]>
Date: Thu, 12 Jun 2008 20:33:06 +0200

[...]
> we hope that discussions surrounding the BoF could elucidate the proper
> division of labor and approach to this problem.

To follow up on Jon's thoughts and to reflect discussions that happened
during the MIT workshop and on the mailing list, Vijay and I are going
to update the problem statement draft the ALTO BoF proposal is based on
(draft-marocco-alto-problem-statement-00).

As a prelude to focus the ALTO BoF, it would be great to start
coalescing community agreement on ensuring that the scope of the problem
is well defined and understood.

Essentially, the problem that ALTO will attempt to solve is this:
applications (P2P and otherwise) that make intensive use of the network
bandwidth or want to make decisions to optimize some function (see use
cases) may be interested in choosing all or some of their peers based on
recommendations provided by some network entity -- colloquially called
an oracle -- that has more of a global view on the peers in the network,
their topological distances, and their capabilities than the application
itself does.

In thinking about the problem, it may help to consider the following:

+ Use cases: current version of the draft identifies four use cases for
  ALTO solutions (file-sharing, VoIP, p2p streaming and DHTs) but others
  come to mind (e.g. cache/http-mirror selection, CDN).  At the MIT
  workshop, Ted Hardie coined the term "unattended consequences" of
  peer-to-peer nodes where traffic runs constantly for as long as the
  resources to that node are popular.  Video feeds to remote stations
  were identified as a category.

+ Third-party services: even if topology information is directly
  available to network operators, nothing should prevent third-parties
  to provide peer selection services alternatively to or in competition
  with ISPs.  An incomplete list of proposed solutions for providing
  such services is online at http://alto.tilab.com/resources.html;

+ Type(s) of information that are likely provided by the oracle (note
  that a third-party oracle may not have access to some of this
  information):
  * Given a list of peer candidate IP addresses, sort them according to
    preferences (topological proximity, bandwidth constraints, etc.) and
    return to querying application;
  * Ability for an application to determine its network location and
    proximity to other nodes;
  * Support for cost and charging information for application so that
    applications can make appropriate trade-offs;
  * Capabilities and characteristics of peers (i.e., last hop bandwidth,
    etc.);
  * Network policies;
  * ...

+ Type(s) of information that is provided to the oracle for rendering an
  equitable decision:
  * List of peer candidate IP addresses;
  * Parameter to optimize (cost, delay, ...);
  * ...

This is mainly what we have captured so far; however, feedback,
suggestions and contributions are not only welcome, but at this time
even required for the successful setting of a BOF.

Besides, the draft currently identifies two elements for which
standardization work would be required, namely a discovery mechanisms
for locating oracles and a signaling protocol for querying them.  Do
people think the core solution for ALTO should include anything else?

-- 
Ciao,
Enrico

---

From: stefano previdi <[email-masked]>
Date: Fri, 13 Jun 2008 10:52:29 +0200

Enrico,

I mostly agree mostly with everything below. My suggestion (as a routing guy
more than an apps guy) is also to scope what could be extended/enhanced in
routing protocols in order to improve localization mechanisms (whatever they
are). 

As an example, we have extended ISIS routing protocol in order to carry some
application related information (at this stage is an empty container) and
probably these kind of enhancements may have a use in localization/oracle
services.

s.

> From: Enrico Marocco <[email-masked]>
> 
> Peterson, Jon wrote:

---

From: John Leslie <[email-masked]>
Date: Fri, 13 Jun 2008 11:00:58 -0400

Enrico Marocco <[email-masked]> wrote:
>... 

   I think we could reasonably expand this, so as to provide more for the
oracle to work from.

   There are two well-known cases:

1. We want a well-known range of bandwidth for a possibly well-known
   length of time.

2. We want to transfer a well-known number of bytes within a fairly
   well-known period of time.

   In the first case, "No path will work" is a useful answer, and we
should have a way of saying so.

   In the second case, an estimate of how long the transfer _would_
take is very useful information, and we should have a way of passing
that back. (Users will typically abort transfers that aren't making
enough progress; and giving them information to abort sooner would be
a good thing.)

   (I really like calling this an "oracle" service, because it must
necessarily make predictions about the future. However poor those
predictions may prove, the oracle _can_ have a better (statistical)
basis for them than either end-point.)

   Latency is also possibly an issue in the first case: it's probably
a good idea to report expected latency for each path. (Optimizing for
it may or may not be useful.)

--
John Leslie <[email-masked]>
_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Fri, 13 Jun 2008 10:54:10 -0500

stefano previdi wrote:
> Enrico,
> 
> I mostly agree mostly with everything below.

Stefano: Thank you.

> My suggestion (as a routing guy more than an apps guy) is also to
> scope what could be extended/enhanced in routing protocols in order
> to improve localization mechanisms (whatever they are).

I think the issue of scoping is something that we definitely
have to work on.

I also suspect that ALTO solutions deployed by network operators
will probably make use of information carried in routing protocols
and, if it turns out that extensions to such protocols may be
beneficial, they should be specified in coordination with ALTO, but
probably in the Routing area.  At the current time, the viewpoints
of the ADs appear to be that ALTO will be either in RAI or Apps.

> As an example, we have extended ISIS routing protocol in order to 
> carry some application related information (at this stage is an empty
> container) and probably these kind of enhancements may have a use in
> localization/oracle services.

It would be great if you could describe this in a draft.  One of
the points we made in our workshop paper was the need for layer
cooperation.  Your existing work would go a long way to provide
some illumination of feasibility in that area.

Ciao.

- vijay

---

From: Lars Eggert <[email-masked]>
Date: Fri, 13 Jun 2008 19:10:23 +0300

Hi,

>  that a third-party oracle may not have access to some of this

I have a few comments about the kinds of information that the oracle  
might provide to the app.

The high level bit is that I believe that although it will be possible  
to provide information that changes on relatively large timescales  
(such as information about topological proximity), it will be both  
very difficult and potentially problematic to share performance- 
related information that changes on short timescales in the same  
fashion.

>    proximity to other nodes;

These are all OK, except for the bit about "bandwidth constraints".  
The available bandwidth along a path changes very rapidly, on  
timescales roughly O(RTT), because bandwidth consumption is regulated  
by the congestion control loops of the flows sharing a path. This  
pretty much guarantees that any such information an oracle could  
maintain will be stale by the time that an oracle can provide it to an  
application. So this information is unlikely to be useful to an app;  
and I can even construct cases where using this stale information  
leads to traffic oscillations, which is problematic.

Modulo this concern about stuff that varies on short timescales, a  
simple mechanism that lets an app query an oracle about the things in  
the two bullets above would be useful.

I'm MUCH less convinced of the usefulness of the shopping list of  
things in the following bullets. For a lot of the things below, it's  
unclear to me why an application would want them and where an oracle  
would get them from. I don't think they should be part of any  
standardization effort at this time. (Looking at them from a research  
perspective is fine.)

>    applications can make appropriate trade-offs;

Can you be a bit more explicit about whose costs this is about? Who is  
paying whom, on what timescales and based on what resources?

>  * Capabilities and characteristics of peers (i.e., last hop  

The bandwidth example is IMO again problematic (and it's not a peer  
characteristic, it's a path characteristic.) Also, why would an oracle  
know anything about the capabilities or characteristics of the peers?  
Are the peers supposed to register with the oracle? That goes far  
beyond what I thought this BOF was about.

>  * Network policies;

Such as?

>  * Parameter to optimize (cost, delay, ...);

Why is it useful to query the oracle for delay? An app can get a  
relatively accurate and more importantly timely RTT sample by sending  
a few packets to a peer. That practice causes no issues for operators,  
since the traffic volume is so low.

Lars

---

From: Lars Eggert <[email-masked]>
Date: Fri, 13 Jun 2008 19:12:57 +0300

> enough progress; and giving them information to abort sooner would be

There is NO way for the oracle to figure this out - this depends  
entirely on the current load along the path, which is changing on  
short timescales under the control of the end-to-end congestion  
control loop.

Lars

---

From: Lars Eggert <[email-masked]>
Date: Fri, 13 Jun 2008 19:39:35 +0300

Hi,

> seen yet), work item (3) might be subsumed by it.

this BOF proposal has now materialized in form of ALTO (http://alto.tilab.com/ 
), a BOF proposal targeted at APP and RAI.

It's hence somewhat likely that we may have a TANA BOF in TSV that  
focuses on the work items (1) and (2) from my original email, and a  
second ALTO BOF hosted by APP and RAI.

(Note: this email is *not* an IESG approval of either the TANA or ALTO  
BOF - those will come from the responsible ADs if and when they come.)

ALTO is currently scoped a bit more broadly than the TANA work item  
(3) was, and some of the broader proposals in the ALTO space could  
lead to significant transport issues. A fraction of the ALTO  
proponents argues to exchange information related to end-to-end  
performance (bandwidth, RTT, congestion, transfer time estimates,  
etc.) through the oracle, for example.

I'd like to ask transport folks to participate in the discussion on  
ALTO on the [email-masked] mailing list, in order to discuss the  
transport implications of the various ALTO ideas.

Lars

---

From: Enrico Marocco <[email-masked]>
Date: Sat, 14 Jun 2008 21:44:17 +0200

> might provide to the app.

Lars, thanks for your comments.

> the two bullets above would be useful.

Ok.

> paying whom, on what timescales and based on what resources?

Sure.  This was one bullet point that was made in Prof. Schulzrinne's 
paper.  I quote the text from the paper:

    Cost-of-bandwidth information

    Relying on cooperative applications to be “nice” is probably
    insufficient unless there is an incentive. There are at least three
    possibilities, namely either volume limiting, rate limiting or
    volume-based charging. For example, a network could offer a tiered
    tariff that allows unlimited bandwidth within the same AS, 100 kb/s
    rate-limited but unlimited prioritized DiffServ to anywhere and
    volume-limited bulk traffic to any AS. There are several ways to
    convey this information to applications, allowing them to make
    appropriate trade-offs. For example, a network could provide an
    information service with a list of service descriptors, similar to
    the RSVP TSpec. Alternatively, a protocol such as GIST could discover
    the cost of particular destination.  The service classes may change
    over time. For example, a network may offer "midnight specials" when
    the network is idle during off-hours, encouraging bulk downloads
    during that time.

> characteristic, it's a path characteristic.)

Yes, you are right in characterizing this as a path characteristic.

 >                                              Also, why would an oracle

No, we are not suggesting that peers register with the oracle.  This is 
more along the point of trying to generate some discussion.  What if a 
peer is willing to publish its capabilities, either for other peers or 
for an oracle.  Is this something that is worth considering, if not for 
a standardization aspect, at least as a research problem?

> Such as?

Along the same line as mentioned above in the "Cost-of-bandwidth" 
information.

> since the traffic volume is so low.

Well, the use case I had in mind here was the selection of media relays 
in real-time communication systems, when the list of candidate peers may 
have hundreds or thousands of elements (think of Skype).  In such a 
case, provided the oracle can elaborate lists and not on single 
instances only, a one query would replace tons of RTT measurements.

On a side note, is it possible for RTT to be small on a high latency 
link one hop away when compared to a slightly higher RTT on low latency 
links six hops away?  That is, the time it takes for a peer to download 
a large content from the peer with the lower RTT is actually higher than 
the time it takes to do the same from the peer with higher RTT.

-- 
Ciao,
Enrico

---

From: John Leslie <[email-masked]>
Date: Fri, 13 Jun 2008 18:57:27 -0400

> related information that changes on short timescales in the same  

   That's an implementation issue.

   We should do protocol design before we do implementation.

   Besides, it's only a couple orders of magnitude "difficult." It might
be quite practical five years from now.

>>    preferences (topological proximity, bandwidth constraints, etc.)  
>>    and return to querying application;
> by the congestion control loops of the flows sharing a path.

   Absolutely true.

> This pretty much guarantees that any such information an oracle could  
> application.

   Not as true.

   The application isn't particularly interested in one instantaneous
snapshot: it's interested in what's likely to be the case over an
extended period (or how long it is likely to take to transfer a known
amount of data. Statistical summaries are much more useful: and the
oracle is in a good position to provide those.

> So this information is unlikely to be useful to an app;  

   Traffic oscillations are indeed problematic. But can you construct
a case where statistical damping isn't sufficient to guard against
that?

> standardization effort at this time.

   We should be standardizing a protocol, not an implementation.

>>    applications can make appropriate trade-offs;

   Clearly, applications have every reason to consider cost/charging
information provided by the supplier the user will have to pay.

> paying whom, on what timescales and based on what resources?

   I'm not sure we can answer any of those, but it would be good for
the protocol to carry the answers to those questions.

>>  * Capabilities and characteristics of peers (i.e., last hop  
>>    bandwidth, etc.);
> 

   The peer might impose its own limits, beyond the limits of the path.
It might (or might not) be appropriate to standardize a way of
advertising these.

> Such as?

   Good question: I can imagine several, but don't know if we could
define a good set of metrics.

>> + Type(s) of information that is provided to the oracle for  
>>   rendering an equitable decision:
> Why is it useful to query the oracle for delay?

   Because instantaneous delay isn't a good measure. An oracle will be
in a better position to know minimum latency, plus normal range for
minimal congestion.

---

From: Richard Bennett <[email-masked]>
Date: Fri, 13 Jun 2008 17:16:27 -0700

Do we have a date for the proposed BOF in Dublin?

RB

-- 
Richard Bennett

---

From: Lars Eggert <[email-masked]>
Date: Sat, 14 Jun 2008 11:24:26 +0300

> Do we have a date for the proposed BOF in Dublin?

Not yet. I've approved TANA (with work items 1 and 2 from the original  
email), but ALTO needs the approval of the ADs for the areas it was  
submitted to.

> Lars Eggert wrote:
>> (Note: this email is *not* an IESG approval of either the TANA or  
>> ALTO  BOF - those will come from the responsible ADs if and when  
>> they come.)

Lars

---

From: Lars Eggert <[email-masked]>
Date: Mon, 16 Jun 2008 11:41:58 +0300

Hi,

On 2008-6-14, at 22:44, ext Enrico Marocco wrote:
> Sure.  This was one bullet point that was made in Prof.  
> Schulzrinne's paper.  I quote the text from the paper:
>

thanks - that does make it clearer. I agree that it's interesting to  
think through the possibilities here, but I don't see anything we  
could immediately think about standardizing here.

> >                                              Also, why would an  
> oracle
> No, we are not suggesting that peers register with the oracle.  This  
> is more along the point of trying to generate some discussion.  What  
> if a peer is willing to publish its capabilities, either for other  
> peers or for an oracle.

It's more natural IMO for peers to advertise their capabilities to  
each other as part of the P2P protocol, rather than going through the  
oracle.

As for peers wanting to advertise capabilities to the oracle, sure,  
that's a possibility.

>  Is this something that is worth considering, if not for a  
> standardization aspect, at least as a research problem?

Definitely interesting from a research perspective, but not ready for  
standardization IMO.

> Well, the use case I had in mind here was the selection of media  
> relays in real-time communication systems, when the list of  
> candidate peers may have hundreds or thousands of elements (think of  
> Skype).  In such a case, provided the oracle can elaborate lists and  
> not on single instances only, a one query would replace tons of RTT  
> measurements.

I think I remember reading some papers on choosing the best option out  
of a small number of random candidate peers will result in a choice  
that is reasonably good. (Obviously you wouldn't want to probe all the  
candidates.) So the potential for savings is not large. In order for  
the oracle to have a reasonably good view of delays between peers, I'd  
need to perform a similar amount of probing, or it needs to arrange  
peers in some virtual coordinate system and hope that that's accurate  
enough.

> link one hop away when compared to a slightly higher RTT on low  
> latency links six hops away?

I think you mean transmission time when you say latency (RTT *is*  
latency in both ways).

>  That is, the time it takes for a peer to download a large content  
> from the peer with the lower RTT is actually higher than the time it  
> takes to do the same from the peer with higher RTT.

That is definitely possible. For example, low-RTT peers may be behind  
other DSL boxes within your ISP, whereas the university with the big  
pipe is a bit further away in RTT terms than those peers - who'd you  
like to transfer from? I'd expect the correlation between RTT and  
transmission time to be pretty weak (but have no data to back this up.)

Lars

---

From: Lars Eggert <[email-masked]>
Date: Mon, 16 Jun 2008 12:11:45 +0300

Hi,

> be quite practical five years from now.

I think we're talking about two different things. I understood the  
proposal for the oracle to maintain performance-related information to  
be for actual, current load of a path (modulo quantization effects and  
communication latency). That has all sorts of timing- and accuracy- 
related issues.

If I understand your email correctly (based on the quoted section  
below), you see the oracle as maintaining more coarse-grained,  
statistical load data about paths. Things like "this path is half idle  
on Saturday night." I agree that that information would be easier to  
provide. But I question whether such load data can be useful to  
applications (see below).

> oracle is in a good position to provide those.

Assuming for a moment we had an oracle that had such information, the  
only benefit to apps would be to not even try to establish connections  
to peers that had a projected low throughput. That's a minor  
optimization at best. Many P2P systems will drop that peer after a  
short time anyway.

(And if the aggregate traffic to such peers would conflict with ISP  
interests, they'd rank the peer low in the oracle because of that  
anyway, independent of traffic statistics)

> a case where statistical damping isn't sufficient to guard against

Who would dampen what? The apps are just using information provided by  
the oracle and see no oscillations. The oracle could observe that  
traffic shifts, but how does it differentiate if that is due to the  
information it provides or whether it is independent of that? And even  
if it could, what does it then do? Longer statistical aggregation  
decreases the frequency of oscillations, but also means that the load  
data will be even less indicative of current loads.

>   We should be standardizing a protocol, not an implementation.

Agreed, but spending cycles defining a protocol to exchange  
information that an oracle has no chance of gathering or that we  
aren't really sure what an app would do with isn't very practical.

> information provided by the supplier the user will have to pay.

Agreed; *if* the charging isn't a flat fee.

> It might (or might not) be appropriate to standardize a way of

See my email to Enrico; IMO such advertisements between peers should  
be handled inside a specific P2P protocol rather than by a protocol- 
agnostic oracle.

> in a better position to know minimum latency, plus normal range for

Clarification: When you say "delay", it sounds you mean "projected  
transmission time"? Because I think Enrico actually meant RTT. (See my  
email to him on why I don't think RTTs are useful for an oracle to  
maintain.)

Lars

---

From: stefano previdi <[email-masked]>
Date: Mon, 16 Jun 2008 11:23:25 +0200

> From: "Vijay K. Gurbani" <[email-masked]>
> 
> stefano previdi wrote:
>> Enrico,
>> 

Well, I clearly won't start a debate here (on where to land the WG) ;-)
As long as we have a good channel with the routing wg's it's ok to me.
 
> some illumination of feasibility in that area.

ok. I'll try to come up with some documented ideas...

thanks.

s.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 16 Jun 2008 08:31:04 -0500

I don't actually think I'm good at channeling ADs, but for what it's 
worth...

My understanding of what Lars is concerned about, is the idea of using an 
oracle to provide guidance about path characteristics on a timescale of less 
than many, many RTTs. He's concerned that a transmitter can respond to "good 
news" about a path from an oracle more quickly than the oracle can recognize 
that a transmitter has rapidly ramped up its transmit rate and start telling 
senders what it's seeing.

My understanding of the discussion between John and Lars in this thread is 
that one key question is whether guidance about path selection has any 
relationship with congestion avoidance, detection, and recovery, or not.

If I tell Lars "this path is usually uncongested on Saturday night", and he 
starts a large transfer on Saturday night, what does he do when he 
encounters congestion?

- ignore it and keep transmitting as if the path was really uncongested 
(wrong answer!),

- look for another path (but the guidance was that this is the best path 
most of the time), or

- stay on this path and do TCPish loss-based bandwidth probing (even if the 
problem is that Magnus ALSO saw "this path is usually uncongested on 
Saturday night" and also selected this path for a large transfer - and this 
is even funnier if Magnus was congesting all the other paths LAST Saturday 
night, and decided to switch to this path "because it was uncongested")

It seems helpful to discuss what the strategy is before we get very far into 
how the strategy might work...

Thanks,

Spencer

---

From: John Leslie <[email-masked]>
Date: Mon, 16 Jun 2008 13:16:03 -0400

Lars Eggert <[email-masked]> wrote:
>>> it will be both very difficult and potentially problematic to share
>>> performance-related information that changes on short timescales
>>> in the same fashion.
>>
>> Besides, it's only a couple orders of magnitude "difficult." It  
>> might be quite practical five years from now.
> 
> communication latency).

   I understood the proposal to be maintaining performance-related
information without specification of the time period it reflects.

> That has all sorts of timing- and accuracy-related issues.

   Either way, yes... except that some characteristics like maximum
bandwidth and minimum latency are pretty safe to report.

> statistical load data about paths.

   Actually, I was more concerned that a protocol should support
reporting such things, not wishing to get into how such information
might be divined.

> Things like "this path is half idle on Saturday night."

   (Actually, that's a bad example: while an oracle might be able to
divine that parts of a path are in fact idle from 1800 to 2400 GMT on
Saturdays, it couldn't reasonably say that about the _whole_ path.
It _could_ say "latency above N milliseconds" indicates congestion.")

> I agree that that information would be easier to provide. But I
> question whether such load data can be useful to applications
> (see below).
> 
> to peers that had a projected low throughput.

   It's more a matter of which connection(s) to try _first_.

> anyway, independent of traffic statistics)

   Oracles will not _only_ be maintained by ISPs. ISP oracles will be
used _only_ if the users perceive a benefit from using them.

>>> So this information is unlikely to be useful to an app;

   I don't believe you've made your case for that...

> Who would dampen what?

   Damping is inherent in statistical averaging.

> The apps are just using information provided by the oracle and see
> no oscillations. The oracle could observe that traffic shifts, but
> how does it differentiate if that is due to the information it
> provides or whether it is independent of that?

   It _should_ assume it is independent.

> And even if it could, what does it then do?

   Unless it's the ISPs oracle, it should do nothing different. If it
_is_ the ISPs oracle, we're getting pretty far into implementation...

> Longer statistical aggregation decreases the frequency of oscillations,
> but also means that the load data will be even less indicative of
> current loads.

   Yes. (For many reasons, the p2p applications are likely to be in a
better position to deal with instantaneous congestion.)

> transmission time"? Because I think Enrico actually meant RTT.

   I was intentionally vague, but you guessed right: I think in terms
of one-direction latency. P2P applications _could keep good enough
clocks to make effective use of this.

---

From: John Leslie <[email-masked]>
Date: Mon, 16 Jun 2008 13:39:53 -0400

Spencer Dawkins <[email-masked]> wrote:
> 
> My understanding of what Lars is concerned about, is the idea of using
> an oracle to provide guidance about path characteristics on a timescale
> of less than many, many RTTs. He's concerned that a transmitter can
> respond to "good news" about a path from an oracle more quickly than
> the oracle can recognize that a transmitter has rapidly ramped up its
> transmit rate and start telling senders what it's seeing.

   For a number of reasons, oracles _could_ be in a better position to
know path characteristics. Congestion along a path tends to be concentrated
at relatively few points along the path. Peers know what they're seeing
(instantaneously) over the whole path, but generally lack any understanding
about subpaths.

> My understanding of the discussion between John and Lars in this thread
> is that one key question is whether guidance about path selection has any 
> If I tell Lars "this path is usually uncongested on Saturday night", and
> he starts a large transfer on Saturday night, what does he do when he 
> encounters congestion?
> 

   Agreed.

> - look for another path (but the guidance was that this is the best path 

   Abandoning the path based on instantaneous congestion may be wrong.
Trying an alternate as an additional option may be wise.

> - stay on this path and do TCPish loss-based bandwidth probing (even if
>   the problem is that Magnus ALSO saw "this path is usually uncongested
>   on Saturday night" and also selected this path for a large transfer

   It is reasonable to expect P2P applications to learn how to use more
than one remote peer at a time. Statistical information is never a
guarantee that you'll see exactly the predicted situation.

   (An ISPs oracle could, in principle, downgrade a recommendation after
it has been issued to some number of customers. However, that is an
implementation detail, and I don't mean to recommend it.)

>   - and this is even funnier if Magnus was congesting all the other
>   paths LAST Saturday night, and decided to switch to this path
>   "because it was uncongested")

   The world is, indeed, stranger than we _can_ imagine...

> It seems helpful to discuss what the strategy is before we get very far
> into how the strategy might work...

   P2P applications need not all use the same strategy for identifying
congestion, nor even for responding to it. However, it is arguably
"TCP-friendly" to switch half your bandwidth to an alternate path when
congestion is experienced...

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 16 Jun 2008 13:17:28 -0500

Hi, John,

> (instantaneously) over the whole path, but generally lack any 

I agree that oracles could be in a better position to know path 
characteristics.

What I thought Lars was concerned about was not what the oracle could know, 
but how quickly the oracle could share this knowledge with transmitters. If 
someone told Lars that we expect the oracle to adjust its suggestions every 
30 minutes, he'd probably be happier than if someone told Lars that we 
expect the oracle to adjust its suggestions every 30 milliseconds... and I'm 
not sure that Lars knows which is more representative of what people are 
thinking.

Thanks,

Spencer

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 17 Jun 2008 22:29:46 -0400

> could immediately think about standardizing here.

I do see a need for standardization here, namely conveying the current  
bandwidth price or total usage. (More complicated models are feasible,  
but I think something pretty simple would do the job and avoid getting  
into business models.)

You'd want to get something like

"Your current charge for priority X is Y currency units/byte, valid  
until time T."

"You have used B bytes out of L bytes for priority X."

"Your max. rate for priority X is S bits/second."

This covers the major knobs that a provider can turn (quotas, rate  
limits and metering).

> transmission time to be pretty weak (but have no data to back this  

Given that TCP throughput is inversely proportional to RTT, one would  
indeed expect a correlation. See

http://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html

for a random related page. Obviously, capacity matters where a single  
flow is likely to occupy the whole pipe.

Henning

---

From: "Stanislav Shalunov" <[email-masked]>
Date: Tue, 17 Jun 2008 21:05:26 -0700

> "Your max. rate for priority X is S bits/second."

It might also be interesting to know the differences in provider costs
between different destinations.

> indeed expect a correlation.

The small differences in Internet RTTs are drowned out by the huge
extra delay indiced by TCP, and so wouldn't matter even in random or
periodic loss model that this statement implies.

In reality, of course, TCP throughput is related to bottleneck bitrate.

-- 
Stanislav Shalunov
http://shlang.com/

---

From: Lars Eggert <[email-masked]>
Date: Wed, 18 Jun 2008 08:56:55 +0300

Hi,

On 2008-6-18, at 5:29, ext Henning Schulzrinne wrote:
> I do see a need for standardization here, namely conveying the  
> current bandwidth price or total usage. (More complicated models are  
> feasible, but I think something pretty simple would do the job and  
> avoid getting into business models.)
>

that seems reasonable - thanks for clarifying.

> Given that TCP throughput is inversely proportional to RTT, one  
> would indeed expect a correlation.

TCP bandwidth is proportional to the RTT when different flows *share  
the same bottleneck*. But that isn't what matters here. Knowing the  
RTT to a potential peer will not allow an app to draw any conclusions  
about projected throughput, because the app knows neither the  
bottleneck capacity nor the RTTs of all other flows going through that  
bottleneck. Plus, the theoretical result is only valid when all flows  
are in steady state.

Lars

---

From: Lars Eggert <[email-masked]>
Date: Mon, 23 Jun 2008 17:51:24 +0300

Hi,

sorry it took me a while to respond.

>> transmit rate and start telling senders what it's seeing.

Spencer: sort of, yes. Basically, I'm concerned that this builds a  
high-layer control loop that acts on the same information on (as I  
understand it) roughly the same timescales. That's not necessarily  
problematic, but there's at least a ;arge potential for interactions.

> (instantaneously) over the whole path, but generally lack any  

John: But what matters for a given flow are the characteristics of the  
whole path. Why would sub-paths matter? Also, how would the oracle  
learn about the congestion levels of each router, and on what  
timescales is this information accurate?

> guarantee that you'll see exactly the predicted situation.

True, but unless the information is accurate much more often than not,  
why bother designing a mechanism to provide it? Apps will just ignore  
it then.

I question whether we actually can design an oracle that is right much  
more often than wrong with regards to performance-related information.  
Trying to answer this question as a research effort is very  
interesting, doing standardization around it is IMO premature.

Lars

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 23 Jun 2008 11:10:34 -0500

For what it's worth...

> why bother designing a mechanism to provide it? Apps will just ignore  it 
> then.

This was actually the stake through the heart of TRIGTRAN, if anyone 
remembers that conversation - TCPs are complicated enough already, so if you 
have to treat non-loss indications as advisory, and you can figure out what 
the right answer is in a small number of RTTs without using the other 
indications anyway, the thought was that there wasn't enough reason for 
implementers to find TRIGTRAN interesting.

> Trying to answer this question as a research effort is very  interesting, 
> doing standardization around it is IMO premature.

You may very well be correct... :-)

Spencer

---

From: John Leslie <[email-masked]>
Date: Mon, 23 Jun 2008 12:45:23 -0400

Lars Eggert <[email-masked]> wrote:
>> For a number of reasons, oracles _could_ be in a better position
>> to know path characteristics. Congestion along a path tends to be  
>> concentrated at relatively few points along the path. Peers know
>> what they're  seeing (instantaneously) over the whole path, but
>> generally lack any understanding about subpaths.
> 
> John: But what matters for a given flow are the characteristics of
> the whole path. Why would sub-paths matter?

   A good question, actually...

   Sub-paths matter because they're what gets congested.

   P2P applications have good reason to want to know (instantaneous)
congestion, and what the likelihood is that the congestion will get
better or worse soon.

   Oracles _could_ give them "This is what congestion looks like"
information. This could be a big win.

> Also, how would the oracle learn about the congestion levels of each
> router, and on what timescales is this information accurate?

   Oracles, pretty much by definition, have access to characteristics
over a _long_ time period. _How_ they learn congestion levels of each
router is not in scope of this work. (Certainly there are standard
tools ISPs use to divine congestion levels.)

   It's not important for us to say what timescales Oracles should
use, we merely want to specify how they report their timescales.

> True, but unless the information is accurate much more often than not,  

   This shouldn't be a difficult goal -- unless we do a poor job of
defining a protocol to accurately communicate _what_ has been measured,
on what timescale.

> why bother designing a mechanism to provide it? Apps will just ignore  

   ... giving other Oracles a shot at the market...

> more often than wrong with regards to performance-related information.  

   It really isn't hard!

   Oracles can almost trivially find maximum bandwidth and minimum
latency along subpaths, and unless the routing changes, these will
stay rather constant.

   The apps, given just that much information, can make a much better
judgment of instantaneous congestion than they could without it.

   Anything an Oracle could do about predicting future trends in
congestion would be icing on the cake, hardly even necessary to
prove their worth.

> interesting, doing standardization around it is IMO premature.

   There's certainly room for a lot of research; but standardizing
the _interface_ between oracle and P2P app would be a _major_
benefit to that research, while giving some immediate benefit to
the apps and the ISPs.

---

From: "Robert Snively" <[email-masked]>
Date: Mon, 23 Jun 2008 13:15:01 -0700

One thing I have not seen directly addressed is that congestion
is only a second-order symptom of the demand profile.  It indicates what
managed to get into the network, not what people really wanted to pass
through the network.  How do we address that broader question?

Bob

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
John Leslie
Sent: Monday, June 23, 2008 9:45 AM
To: Lars Eggert
Cc: [email-masked]
Subject: Re: [p2pi] Refining the ALTO problem statement

   It really isn't hard!

---

From: "Stanislav Shalunov" <[email-masked]>
Date: Mon, 23 Jun 2008 16:58:39 -0700

> congestion, and what the likelihood is that the congestion will get

This is the function of transport.  P2P apps, by doing e2e
measurements, already have pretty decent information about state of
congestion, and use it just the way you'd expect.

What P2P apps don't have, and what ALTO should provide, is new
information -- not whether a path is congested, but whether a path is
preferred by the ISP.  Some simple notion of cost or preferred ASes or
IP prefixes or whatnot.

This is information that the routing layer has and that the overlay
routing in P2P has no access to.

---

From: Laird Popkin <[email-masked]>
Date: Thu, 26 Jun 2008 13:54:48 -0400 (EDT)

I agree with Stanislav's main point (ALTO should focus on providing new information). We've had extensive email chains listing such opportunities, so no need to repeat them here.

That being said, there's some value in knowing about congestion more precisely than we do now. In particular, it would be good to know when:

1) The user's internet connection is saturated (and in which direction) so that the application as a whole can reduce bandwidth to de-saturate the link. Many p2p apps do this now, but the techniques for attempting to indirectly detect 'edge' link saturation are inconsistent in their effectiveness. If the edge router could tell app's the precise situation, that would help app's avoid saturating links while maximizing performance.

2) A specific TCP stream to a specific peer is saturated, in which case we should slow down that stream and pull more data from elsewhere. Re-ECN looks like it'll address the latter scenario. What's the relationship (if any) between Re-ECN and ALTO?

- Laird Popkin, CTO, Pando Networks
  mobile: 646/465-0570

---

From: stefano previdi <[email-masked]>
Date: Thu, 26 Jun 2008 20:16:02 +0200

> From: Laird Popkin <[email-masked]>
> 
> I agree with Stanislav's main point (ALTO should focus on providing new
> information). We've had extensive email chains listing such opportunities, so
> no need to repeat them here.
> 
> That being said, there's some value in knowing about congestion more precisely
> than we do now. In particular, it would be good to know when:
> 
> 1) The user's internet connection is saturated (and in which direction) so
> that the application as a whole can reduce bandwidth to de-saturate the link.
> Many p2p apps do this now, but the techniques for attempting to indirectly
> detect 'edge' link saturation are inconsistent in their effectiveness. If the
> edge router could tell app's the precise situation, that would help app's
> avoid saturating links while maximizing performance.

it's a matter of trade-off in terms of complexity for retrieving the
information at that granularity level you need it and the improved
correctness/usefulness of it. IOW: what you get now may be good enough
already.

> 2) A specific TCP stream to a specific peer is saturated, in which case we
> should slow down that stream and pull more data from elsewhere. Re-ECN looks
> like it'll address the latter scenario. What's the relationship (if any)
> between Re-ECN and ALTO?

I wonder if we have a granularity problem here. A given app/p2p client may
be interested into path characteristics for the flow is going to
originate/receive.

However, the provider infrastructure may not (easily/consistently) deliver
this level of visibility (measuring the impact of a few Mb flow over a
multi-Gb backbone).
 
> routing in P2P has no access to.

I definitely subscribe to this.

s.

---

From: Stanislav Shalunov <[email-masked]>
Date: Thu, 26 Jun 2008 14:21:27 -0700

Knowledge about congestion is a transport topic.  Improvements in  
transport are possible, and, I would argue, necessary.  Congestion  
information is useful on entirely different timescales and is a very  
different problem.  An uncongested path may well be congested seconds  
later.  An inexpensive path is likely to remain such tomorrow and even  
a week later.

Congestion pricing and variations of ECN to achieve it is a very  
interesting topic, but not an ALTO topic.

-- 
Stanislav Shalunov
http://shlang.com

On Jun 26, 2008, at 10:54 AM, Laird Popkin wrote:

> I agree with Stanislav's main point (ALTO should focus on providing  
> new information). We've had extensive email chains listing such  
> opportunities, so no need to repeat them here.
>
> That being said, there's some value in knowing about congestion more  
> precisely than we do now. In particular, it would be good to know  
> when:
>
> 1) The user's internet connection is saturated (and in which  
> direction) so that the application as a whole can reduce bandwidth  
> to de-saturate the link. Many p2p apps do this now, but the  
> techniques for attempting to indirectly detect 'edge' link  
> saturation are inconsistent in their effectiveness. If the edge  
> router could tell app's the precise situation, that would help app's  
> 2) A specific TCP stream to a specific peer is saturated, in which  
> case we should slow down that stream and pull more data from  
> elsewhere. Re-ECN looks like it'll address the latter scenario.  
> What's the relationship (if any) between Re-ECN and ALTO?
>

---

From: Lars Eggert <[email-masked]>
Date: Fri, 27 Jun 2008 10:14:15 +0300

On 2008-6-27, at 0:21, ext Stanislav Shalunov wrote:
> different problem.  An uncongested path may well be congested  
> seconds later.  An inexpensive path is likely to remain such  
> tomorrow and even a week later.
>

It will surprise nobody that I fully agree with Stas.

Lars

>> That being said, there's some value in knowing about congestion  
>> more precisely than we do now. In particular, it would be good to  
>> know when:
>>
>> router could tell app's the precise situation, that would help  
>> app's avoid saturating links while maximizing performance.
>>