Subject: Re: [p2pi] My notes from today's workshop

From: "Paul Jessop" <[email-masked]>
Date: Mon, 2 Jun 2008 20:05:16 +0100

Spencer,

Thanks for circulating this. Very helpful - it captures several points
that I missed in my own notes.

I have a comment that I've like to see included after the "Not dealing
with illegal traffic" line:

Paul Jessop (RIAA) suggested that this restricted focus was not in
accordance with the meeting call which was much broader in scope. He
suggested that IETF could address technical issues associated with the
ability to exclude illegal content which would itself reduce network
congestion.

This was rejected, principally on the grounds that IETF is an
international standards body and it is impossible to determine whether
content is illegal or not at any particular place.

I think I've captured that exchange accurately.

Thanks,

Paul

Paul Jessop

CTO

RIAA

________________________________

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Spencer Dawkins
Sent: 28 May 2008 21:51
To: [email-masked]
Subject: [p2pi] My notes from today's workshop

Here's what I captured...

Spencer

3.1           Welcome/Note Well/Intro Slides

Goals:

*         To think about the Comcast/BitTorrent problem.

*         To think about what the IETF might do to work on the problem 

*         Focus on what's actionable for the IETF, engineering, not
research or politics.

"Not dealing with illegal traffic" - Cullen asked us to focus on the
problems that we still have when all the traffic is legal.

3.2           Service Provider Perspective (Comcast)

Presenters are Jason Livingood and Richard Woundy from Comcast
(including attributions because they're presenting a sponsor's
perspective0.

View that this is a two-way street - applications becoming more
network-friendly, networks becoming more application-friendly. 

Comcast has committed to implement a protocol-agnostic network
management technique (by yearend).

Comcast was managing Vonage customer traffic, and customers were
complaining. Not all the complaints reflected this management (Comcast
and Vonage both experienced problems with transit providers), but that
wasn't obvious to customers.

Most ISPs started managing P2P traffic about 2-3 years ago.  When
Comcast started managing its traffic, they stopped getting complaints
from Vonage and from P2P users. The recent concerns have involved a
suspicion that Comcast was degrading Vonage VoIP service in an attempt
to favor Comcast's own VoIP products.

Current DOCSIS can experience congestion independently on upstream and
downstream and experience congestion independently across DOCSIS
domains.

We talked about some DOCSIS details, but remember that we have to solve
problems for a broad class of access networks, not just for DOCSIS.

Slide point: "How 'TCP flow fairness' applies to the DOCSIS upstream is
an open research question".

DOCSIS also has constant-bandwidth "service flows" (so you don't need to
request-to-send every packet). A Canadian cable company charges extra to
set up a second service flow for Vonage, but other cable companies
haven't followed this lead because of concerns about "net neutrality". 

We did note that if you have enough bandwidth, you don't need to
prioritize traffic, but this doesn't help when traffic levels increase
suddenly ("network capacity increases are not instantaneous"), and
capacity increases can be consumed quickly.

Not all customers are similar (even at the DOCSIS domain level) -
college populations use more P2P, VoIP/streaming video are more diurnal
than bulk file distribution.

Comcast would like to avoid protocol-specific "cat and mouse" detection
and mitigation, and would like to avoid capacity increases that benefit
only 5 percent of heavy users.

DOCSIS 3.0 adds capacity but doesn't solve the capacity problem. Comcast
will deploy DOCSIS 3.0 in 20 percent of network by yearend. This more
than doubles bandwidth available per subscriber, but the conflict is
between adding capacity and supporting applications that are designed to
maximize bulk bandwidth consumption.

There are Internet2 techniques for client-to-network signaling that
haven't been standardized in IETF yet...

Caching content in the network can reduce TCP RTTs and reduce the
peak-to-trough differences of individual flows.

Network management trials begin in June - management based on traffic
characteristics, not on specific protocols. Algorithm - mark all flows
as priority and then mark specific flows as "best-effort" when
congestion approaches "the Near Congestion State".

Comcast hasn't figured out how to notify users that they have been
marked as best-effort yet.

BT is planning a similar mechanism on their DSL network, but they are
planning on varying the bandwidth, not changing QoS markings. Comcast
thinks customers are more concerned about changing markings and don't
want to penalize high-bandwidth users when the network is NOT congested
("it's a high-bandwidth service, people should use it").

Henning is concerned about mechanisms that have 8 independent tweakable
parameters...

Comcast does still care about protocol-specific management mechanisms,
when the carrier is providing the application, or when the traffic is
illegal, etc.

3.3           Application Designer Perspective (BitTorrent)

Presenter is Stanislov Shalunov.

BitTorrent is willing to make changes based on outcomes at this
conference.

BitTorrent sees two problems - RTT measured in seconds, inefficient
overlay routing.

Peer selection is random, except that there is some bias towards
established peers. Trackers randomize the peer list, and peers ask the
resulting randomized list what peers they know about. Rarest-first
ensures best reliability and helps peers have pieces of files to
"trade". This also gives peer diversity.

The best peers can be farther away - a peer with a big pipe, a long way
away, can drive you at your downlink speed, but a peer on your own cable
modem subnet can only drive you at its uplink speed.

Danny's comment - restated for minutes: "If you have more localized
distribution via caching, or location techniques, or whatever, then
burst rates (i.e., peak <> trough ratios) may actually be larger and you
may increase HFC contention (note: HFC in particular, v. DSL, etc..)"

Overlay efficiency - Stanislov  is experimenting with trying peers on
same AS, or on a set of ASes (but then you need the set), or in a list
of IP prefixes (but then you need the list), or based on some cost
minimization model (but then you need costs).

If you have more than 20 peers in an AS, you can probably trade files
happily, and 57 percent of transit traffic could be reduced with this
modification. If there aren't enough local peers, make one a cache (and
please choose a fast one), or 10, or 100.

Stanislov's experience is that they can do 30-percent hit rate with 1
TByte cache, 80-percent with 100 TBytes - and 1 TByte fits into a
device.

Dave Oran asked what the required bandwidth for an 80-percent hit rate
of 1 TByte is - he is concerned that this architecture may reveal
bandwidth bottlenecks onto and off the cache device itself.

Stanislov hopes the IETF can work in these areas:

*         Cache discovery protocol

*         Network information for smart peer selection

*         Experimental congestion control

*         BCP on P2P pain

Stanislov doesn't think we're ready to do congestion control for P2P yet
- do a framework, but don't standardize the mechanism yet.

Lars says there's a TSV committee looking at new congestion control
mechanisms, but the participants are all from the long-delay community.

Current BitTorrent cache discovery protocol is DNS-based, with an SRV
record based on the reverse DNS lookup of your IP address.

Dave Oran asked for input from DSL and Metro-Ethernet network operators,
because he sees the network characteristics being radically different,
and the problem he sees on deployed networks is under-provisioned
aggregation network links.

3.4           Lightning Talks & General Discussion

3.4.1            Robb Topolski

This talk covers about a third of Robb's submitted paper.

Robb believes that the common accusations about P2P file-sharing aren't
actually true - for example, it's very rare for any applications to
rate-limit, so almost all applications will use all available bandwidth.

We need to know a lot more about the problems we're facing (including
"where congestion actually is") in order to solve problems, instead of
simply dealing with effects.

3.4.2            Nick Weaver

The problem with P2P filesharing is that it shifts cost, doesn't save
costs. It increases aggregate costs, and it's transport-inefficient.

Could easily build a BitTorrent cache today - all the components are in
place. But it's less efficient than HTTP, and it's lawsuit-bait for
pirated content. 

3.4.3            Leslie Daigle

"The open, decentralized nature must be preserved."

The problems aren't about P2P technology itself. There's an assumption
that network operators won't/can't adjust the network to meet demand (as
P2P supernodes move).

What are we trying to do:

*         Make P2P networks work better?

*         Make applications and networks more aware of each other?

*         Solve the general-case of bandwidth-intensive applications?

*         Deal with in-network issues or impact of cross-network
traffic?

3.5           Localization and Caches

3.5.1            Laird Popkin and Haiyong Xie - P4P 

Average P2P bit travels 5-7 metro hops in Verizon networks -
network-oblivious P2P applications may not be network-efficient. Routing
information is hidden, TCP rate control is coarse-grained. 

P4P goal is to design a framework that enables better cooperation
between providers and applications.

P4P tracker provides static topologies, policies, provider capabilities,
virtual cost...

Virtual costs can be distributed based on provider preferences, but they
are talking to providers about 5-minute refresh windows (for scaling
reasons). Costs are provided by the providers (not currently tied to
routing, but could be if someone wants to provide a feed from routing).

Trials at Verizon and Telefonica showed significant increases in
intra-ISP traffic, including increase in "local" traffic based on IP
prefixes. This was a large swarm (3500 peers at the peak, 300 after a
few days).

Possible IETF work - "trackerless P2P" (DNS), lookup mechanisms to find
AS-PIDs and trackers for a given ASN.

Jon asked for information about what IETF protocols are being modified.

Henning said it would be nice to have everyone work together, but IETF
is slow... P4P Forum doesn't think they are doing a standard, they think
they are validating a concept.

3.5.2            Yu-Shun Wang - P2PI Traffic Localization - Microsoft

Lots of reasons to localize - apply QoS policy, throttling, admission
control - but localization makes the user experience "non-uniform".

Using multi-layer trackers (global, local, anything in between).

Want to be "local" - but may have to accommodate traffic conditions,
load balancing, policy, etc. Need the right metrics - the policy might
be NOT to localize.

Most of focus is on topology matching between link layer and application
layer. Caches are easy (long-lived peers) but caches have to be in the
right place in order to gain benefits.

Looking at tracker redirection/delegation. Prefer off-path for
reliability and avoiding performance bottlenecks.

Eric Burger asked what incentive people have to deploy a mechanism like
this.

Jon liked this paper/presentation because it seemed easy to chop into
engineering-sized parts.

Bob raised a concern about ISP liability if they are helping pirates
choose optimal paths for particular swarms, but if you provide
information for all users in all swarms, this isn't scalable.

3.5.3            Vinay Aggrawa - ISP-aided Neighbor Selection - DT

Vinay is talking about something that's similar to P4P - having a
provider supply an oracle so that any peer can find its "local
neighbors".

Henning is concerned about oracles that dump traffic off-net, and beggar
their neighbors.

Dave Oran thought the oracle would also know who was talking to who now
- and this approach doesn't have that information yet.

People are thinking that 9000 nodes is a big swarm - for legal content
with advertising, it probably isn't. We need to think about much larger
swarms.

Could just run oracle as web server/UDP server at well-known location
(similar to BIND). Open and simple solution, open for all overlays.

Oracle service doesn't cache content at ISP, doesn't participate in file
sharing.

Are ISPs willing to accept approaches like oracles that rely on
interaction with clients? They've resisted this in the past.

3.5.4            Cullen - IETF work on localization?

(Cullen had a slide of possible work from listening to presentations and
questions/answers)

Leslie - still need to decide what problem we're trying to solve...

EKR - people talked about making clients less aggressive, moving data
into caches so it doesn't consume uplink capacity, and choosing better
peers. All of Cullen's list was about localization. Doesn't seem that
localization would be important to Comcast, and I'm terrified of the
legal issues involved in caching.

Would be useful for nodes to find their own locations in a topology -
find own AS numbers, etc.

Henning - providing information that customers can't verify, can lead to
undesirable situations (like "hot-potato" in routing).

Standardize a protocol to delegate from global trackers to local
trackers? Lisa says lots of HTTP servers have their own mirror mapping
information distribution mechanisms (proprietary). It's possible this
would be reusable, but possible that the two problem spaces are just too
different.

Lisa also had concerns about layer-nine concerns with inter-ISP
localization issues.

Why not look at the entire problem, and not just parts by parts? We did
IM as an application, and Google is starting to use the standardized
versions.  Jon thinks that IM came into the IETF in just the way he's
thinking of for P2P - what are the parts we need?

Lars - we have transport technologies that aren't seeing deployment.
Maybe incentives have changed, but something would have to change in
order for new transport stuff to be deployed. Localized changes, at the
point that benefits from the change, do get deployed.

Lisa is concerned that localization is one of the technologies that can
work in practice, in a limited environment, but can't be standardized -
because of issues about trust, accuracy, etc...

3.6           New Approaches to Congestion

3.6.1            Bob Briscoe

Bob is presenting Diffserv weighted scheduling versus weighted
congestion points. 

Diffserv is typically used without an API, so the application itself
does nothing about Diffserv.

Diffserv isn't designed to work at the granularity of individual users.

When a light user sends to a heavy user, the light user is using up
Diffserv limits in his own network, only for the traffic to be marked
down in the heavy user's network.

Bob is concerned that we're going to end up in the same place as ATM -
with too many traffic classes.

Bob proposes a "congestion harm (cost) metric". 

Is this anything like Comcast's proposal earlier in the workshop?
Important to take what ISPs are doing now into account as we plan years
into the future.

ECN is helpful but ingress policer can't see congestion on the rest of
the path. 

Bob is asking the IETF/IRTF to set up a longer-term design team for
congestion.

3.6.2            Marcin Matusze

This presentation is based on conversations with about 20 ISPs.

ISP customer acquisition costs are much higher than monthly revenue, and
cost structure varies (fixed versus mobile). It's difficult to change
contracts, and 5 percent of users generate 75 percent of the traffic.

Mobile network operators have more tools (volume-based accounting, fixed
pricing up to a limit). Most tools aren't widely used.

BT is starting to use deep packet inspection, and Bob says that's 10
times more cost-effective than adding bandwidth.

Finnish ISPs were very concerned about P2P traffic in 2004, but much
less concerned today.

3.7           Quality of Service

3.7.1            Mary Barnes - QoS-enabled traffic prioritization

Many service providers are implementing deep packet inspection, but this
isn't viable long-term.

Diffserv can be implemented today but doesn't solve all the problems,
and provisioning/managing still has to look at the events that cause
peak usage.

Why hasn't Diffserv been deployed in the past? Maybe the problem is big
enough now? Bob says data is showing that the amount of P2P traffic in
the world is decreasing...

People are using QoS - just not on the Internet. More than 50 percent of
Cisco's enterprise users have turned Diffserv on - actually higher than
that, for enterprise VoIP or video users. There's a real problem with
DSLAMs with very limited processing capabilities - Dave Oran made the
point that the problems in layer-two aggregation networks are severe and
overlooked.

3.7.2            Henning Schulzrinne - encouraging bandwidth efficiency

Average TV consumption in North America for HDTV = 972 GBytes per month
- reasonable upper limit.

Goal is to minimize OVERALL cost of content delivery.  Focusing only on
HTTP efficiency misses the point. 

Transit bandwidth trends changed in 2004 - from a steady decline to very
little change since 2004.

Cost for sharing content is a step function - moving from the
neighborhood to a national distribution requires different
infrastructure.

FIOS TV architecture is very different from the CATV architecture
discussed this morning. 

We have a general discovery problem (network topology, STUN, HELD, LoST,
SIP local network configuration...). We have a variety of solutions
(DHCP, DNS, anycast...)

If you're volume-based, you need application-visible charging indication
to allow people to make rational decisions. Can the IETF help here?

Misbehaving applications expose user to risk in a charging situation.

3.8           Conclusions & Wrap

Generalized "talking to your local infrastructure" mechanism keeps
coming up, over and over.

Indication of charging would be machine-parsed for any "rational actor"
- person or machine.

Eric is concerned that we're identifying so many topics that are
application-specific and hard-to-too-hard to generalize into a transport
mechanism. 

Jon thinks P4P "virtual costs" makes sense as something we can extract
out.

Do we need more than "this will be expensive/this will be cheap"?

"I'd be happy to tag low-priority traffic if that made any difference to
me" - if ISPs are putting bandwidth caps in place and scavenger-class
traffic didn't count, that would make a difference.

Congestion control isn't going to fix the cost issue - if we aren't
going to grow a network to accommodate increased traffic, we're just
deciding what order the packets queue in. The IETF needs to decide
what's about economics and what's about technology.

Can we stop whining about P2P using too many TCP connections? 

Is it possible to please all the parties? What if benefits to users and
benefits to ISPs are just incompatible?

We need to be very cautious about matching up network protocols and
economic models - this usually ends badly (micropayments, etc). Focus on
technical topics (congestion, etc). 

Congestion arises when there's a scarcity, and you allocate resources
based on cost (which may be money, or might be pain). Congestion isn't
different from cost - ISPs know that. Just because cost proposals
haven't worked in the past doesn't mean there's no way to address cost.

Greg - worth IETF attention, momentum around caching, momentum around
optimizing peer selection. Good to think about whether we bring all of
the work in at once. On deep packet inspection, as a vendor, unless you
want to trust host markings, first-hop routers have to inspect packets
and that's not cheap.

Cullen - Microsoft API will default-mark voice traffic above router
traffic in priority. Unless we get our act together we're headed for a
train wreck.

Danny - IETF might want to consider guidelines on application
interaction with transport issues (multiple ports, etc). Mobile networks
have all kinds of differential pricing (nights and weekends), but other
providers have no hooks into charging for things like distance (even
though it might cost the provider more to retrieve a file from Japan).

Lars - a lot of the things we're talking about aren't just P2P - IPTV
(for example) might have many of the same issues. What are the other
heavy bandwidth usages of the network?

Bob - congestion isn't cost until it becomes part of the contract that
you have with a provider - and providers can't see congestion outside
the provider's own network.

Eric - in economics, pricing is a protocol and tells you when you should
spend money. We need to remember that P2P is about content, too -
networks that were engineered for small-request/large-response
applications are now carrying servers from grandmothers.

Stanislov - opening multiple TCP connections isn't the problem - delay
is the same for P2P and Facebook. (This got responded to: this is only
the case when you're the only user on the link - but other applications
open multiple links, too. Quoting Dave Oran - "there are non-evil
reasons to open multiple TCP connections and you can't tell them
apart").

_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi