Subject: [P2PSIP] Should RELOAD impose a size restriction for storage?

From: "Narayanan, Vidya" <[email-masked]>
Date: Fri, 14 Nov 2008 01:04:28 -0800

When an overlay is composed of heterogeneous nodes, it is unrealistic to expect all nodes to have the ability to store and serve all data.  Now, a node may always reject the request to store something.  However, the robustness of a peer-to-peer network is affected when a portion of the nodes start running out of storage capability for critical data.  Critical data is, of course, quite overlay specific.  What qualifies as critical data may actually be different for different overlays.   

It is not readily possible for a storing node to tell what type of data it is being requested to store - however, it may be reasonable for overlays to have a guidance on the upper bound on the size of a single data item.  This should be configurable for overlays and may even be advertised as part of the overlay configuration.  This allows participating nodes to be composed of heterogeneous capabilities and still maintain a reasonable level of robustness in the overlay.  Nodes may choose to accept data items larger than this upper bound - however, the data owner must be prepared for a rejection of the store request when the data is larger than the bound.  

Thoughts? 

Vidya
_______________________________________________
P2PSIP mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2psip

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 14 Nov 2008 13:37:29 -0500

Longer term, the limit would indeed be configured as part of an  
overlay policy, which is part of the "to do" list. I'd like  
implementations and protocols to be as liberal as possible, since the  
same code may run in a LAN with TB disks, where a gigabyte data object  
barely move the bandwidth meter, to a sensor network on long-range RF  
links, where anything above a few hundred bytes would kill the network.

Henning

On Nov 14, 2008, at 4:04 AM, Narayanan, Vidya wrote:

>
> When an overlay is composed of heterogeneous nodes, it is  
> unrealistic to expect all nodes to have the ability to store and  
> serve all data.  Now, a node may always reject the request to store  
> something.  However, the robustness of a peer-to-peer network is  
> affected when a portion of the nodes start running out of storage  
> capability for critical data.  Critical data is, of course, quite  
> overlay specific.  What qualifies as critical data may actually be  
> different for different overlays.
>
> It is not readily possible for a storing node to tell what type of  
> data it is being requested to store - however, it may be reasonable  
> for overlays to have a guidance on the upper bound on the size of a  
> single data item.  This should be configurable for overlays and may  
> even be advertised as part of the overlay configuration.  This  
> allows participating nodes to be composed of heterogeneous  
> capabilities and still maintain a reasonable level of robustness in  
> the overlay.  Nodes may choose to accept data items larger than this  
> upper bound - however, the data owner must be prepared for a  
> rejection of the store request when the data is larger than the bound.
>
> Thoughts?
>
> Vidya

---

From: "Bruce Lowekamp" <[email-masked]>
Date: Fri, 14 Nov 2008 15:17:14 -0500

I think there are two aspects to this policy.

First, the existing usages aren't intended to store large amounts of
data.  The SIP usage stores a dictionary of AoR->URI mappings.  It
currently doesn't specify a limit on size or allow a configuration to
specify one.  I don't know if that would be useful.  Similarly the
certificate and turn server usage store very small amounts of data.
So I don't see size as a big problem with the current usages assuming
the CA hasn't been compromised in some way, facilitating a DoS attack.

But you're right that as soon as we add a usage for voicemail, etc, we
run into this issue.  I think the big question there is what support
needs to be in the base protocol to allow usages/specific overlays to
specify size limits and what needs to be in the usages.  Possibly we
could have an extension that specifies handling for large objects that
then all usages requiring large objects require.  That would keep the
base spec simple while specifying a single way to store large objects.

To Henning's point, I think there are some really interesting
questions of what part of this need to be specified as policy on a
per-overlay basis and what needs to be in the specs.  For example, if
the protocol intends to support an overlay where peers may exist that
are unable to store all of the resources for which they are
responsible, then the protocol needs to specify how to handle that
(indirect pointer or something) even if other overlays don't need it.

Bruce

Bruce

On Fri, Nov 14, 2008 at 1:37 PM, Henning Schulzrinne
<[email-masked]> wrote:
> Longer term, the limit would indeed be configured as part of an overlay
> policy, which is part of the "to do" list. I'd like implementations and
> protocols to be as liberal as possible, since the same code may run in a LAN
> with TB disks, where a gigabyte data object barely move the bandwidth meter,
> to a sensor network on long-range RF links, where anything above a few
> hundred bytes would kill the network.
>
> Henning
>
>> When an overlay is composed of heterogeneous nodes, it is unrealistic to
>> expect all nodes to have the ability to store and serve all data.  Now, a
>> node may always reject the request to store something.  However, the
>> robustness of a peer-to-peer network is affected when a portion of the nodes
>> start running out of storage capability for critical data.  Critical data
>> is, of course, quite overlay specific.  What qualifies as critical data may
>> actually be different for different overlays.
>>
>> It is not readily possible for a storing node to tell what type of data it
>> is being requested to store - however, it may be reasonable for overlays to
>> have a guidance on the upper bound on the size of a single data item.  This
>> should be configurable for overlays and may even be advertised as part of
>> the overlay configuration.  This allows participating nodes to be composed
>> of heterogeneous capabilities and still maintain a reasonable level of
>> robustness in the overlay.  Nodes may choose to accept data items larger
>> than this upper bound - however, the data owner must be prepared for a

---

From: "Das, Saumitra" <[email-masked]>
Date: Fri, 14 Nov 2008 13:21:39 -0800

There is another angle to this point to think about and that is whether a given overlay can actually support a given volume of data. Apart from real storage problems (e.g. in smaller devices), overlays are also limited by the bandwidth requirements to move data around due to churn even if every node has a terabyte disk on it. The overlay size restriction could also be used to limit data entry into the overlay based on the observed churn characteristics of the specific overlay.

This paper (High availability, scalable storage, dynamic peer neetworks: pick two, proc. of HOTOS 2003) argues that cross sytem bandwidth is the main limiting factor in the dynamic overlays that we expect to form. It shows that the supported storage of a gnutella overlay is a small fraction of the real storage capaiblities of the nodes in that overlay due to cross sytem bandwidth constraints in maintaining the stored data.

Thanks,
Saumitra

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henning Schulzrinne
Sent: Friday, November 14, 2008 10:37 AM
To: Narayanan, Vidya
Cc: [email-masked]
Subject: Re: [P2PSIP] Should RELOAD impose a size restriction for storage?

Henning

---

From: "Narayanan, Vidya" <[email-masked]>
Date: Fri, 14 Nov 2008 14:06:35 -0800

All good points.  I agree that for the immediate set of services we envision, size is potentially a non-issue - although, one question here is that if no such size verification is needed, could it be abused?  But, even outside of that, I think it would be good to have the hooks to support this for future usages such as voicemail.  

>From having the hooks point of view, having the size be part of the overlay policy is one thing.  The main thing I think is to define the node behavior when a store request exceeds the size in the policy - this probably requires an error code defined.  I don't think we need much else. 

Vidya

> From: Bruce Lowekamp [mailto:[email-masked]] 
> Sent: Friday, November 14, 2008 12:17 PM
> To: Henning Schulzrinne
> Cc: Narayanan, Vidya; [email-masked]
> First, the existing usages aren't intended to store large 
> amounts of data.  The SIP usage stores a dictionary of 
> AoR->URI mappings.  It currently doesn't specify a limit on 
> size or allow a configuration to specify one.  I don't know 
> if that would be useful.  Similarly the certificate and turn 
> server usage store very small amounts of data.
> So I don't see size as a big problem with the current usages 
> assuming the CA hasn't been compromised in some way, 
> facilitating a DoS attack.
> 
> But you're right that as soon as we add a usage for 
> voicemail, etc, we run into this issue.  I think the big 
> question there is what support needs to be in the base 
> protocol to allow usages/specific overlays to specify size 
> limits and what needs to be in the usages.  Possibly we could 
> have an extension that specifies handling for large objects 
> that then all usages requiring large objects require.  That 
> would keep the base spec simple while specifying a single way 
> to store large objects.
> 
> questions of what part of this need to be specified as policy 
> on a per-overlay basis and what needs to be in the specs.  
> For example, if the protocol intends to support an overlay 
> where peers may exist that are unable to store all of the 
> resources for which they are responsible, then the protocol 
> needs to specify how to handle that (indirect pointer or 
> something) even if other overlays don't need it.
> 
> Bruce
> 
> 
> Bruce
> 
> 
> unrealistic 
> >> to expect all nodes to have the ability to store and serve 
> all data.  
> >> Now, a node may always reject the request to store something.  
> >> However, the robustness of a peer-to-peer network is 
> affected when a 
> >> portion of the nodes start running out of storage capability for 
> >> critical data.  Critical data is, of course, quite overlay 
> specific.  
> >> What qualifies as critical data may actually be different 
> for different overlays.
> >>
> >> even be advertised as part of the overlay configuration.  
> This allows 
> >> participating nodes to be composed of heterogeneous 
> capabilities and 
> >> still maintain a reasonable level of robustness in the overlay.  
> >> Nodes may choose to accept data items larger than this 
> upper bound - 
> >> however, the data owner must be prepared for a rejection 
> of the store request when the data is larger than the bound.
> >>
> >> Thoughts?
> >>
> >> Vidya

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 14 Nov 2008 19:23:55 -0800

At Fri, 14 Nov 2008 01:04:28 -0800,
Narayanan, Vidya wrote:
> serve all data

That's what being a node means in this context. If you can't meet the
minimum requirements you shouldn't join the overlay as a peer,
but rather as a client.

> Now, a node may always reject the request to store

In a structured overlay, the particular set of nodes which are responsible
for storing a given data object is defined by the topology. If nodes
unilaterally refuse to store given kinds of data, this basically
breaks the underlying semantics of the overlay, with the problem
made worse by the fact that the overlay topology shifts over time.

> data it is being requested to store

On the contrary: in RELOAD data is typed by "usage", so it is fairly
clear.

> - however, it may be reasonable

We considered exactly this design during the early design phase of
RELOAD. Unfortunately, it's unstable. The problem is that the
set of nodes assigned to store a data value shifts over time.
So, I may store value X with node A, which can accept it, but then
node B takes over some of node A's space and node B can't store
the data and it just silently vanishes. In order to provide any
reasonable level of service expectation, the nodes have to
be willing to provide more or less the same level of service
as far as storage goes.

-Ekr

---

From: "Narayanan, Vidya" <[email-masked]>
Date: Fri, 14 Nov 2008 22:59:58 -0800

> From: Eric Rescorla [mailto:[email-masked]] 
> Sent: Friday, November 14, 2008 7:24 PM
> To: Narayanan, Vidya
> Cc: [email-masked]
> > to expect all nodes to have the ability to store and serve all data
> 
> That's what being a node means in this context. If you can't 
> meet the minimum requirements you shouldn't join the overlay 
> as a peer, but rather as a client.
> 

It is the minimum requirement I was talking about.  I agree that all peers must support the minimum requirement.  Turns out I didn't read the draft carefully enough - it looks like the size is part of the overlay configuration.  Sorry for missing that.  However, I don't see an error code for rejecting a store request due to its size - it may help to have one.  

> In a structured overlay, the particular set of nodes which 
> are responsible for storing a given data object is defined by 
> the topology. If nodes unilaterally refuse to store given 
> kinds of data, this basically breaks the underlying semantics 
> of the overlay, with the problem made worse by the fact that 
> the overlay topology shifts over time.
> 

What I was getting at above was that there is no way to enforce compliance to the overlay requirements in a distributed system.  I wasn't saying that a compliant peer would unilaterally refuse a conforming store request. 

> On the contrary: in RELOAD data is typed by "usage", so it is 
> fairly clear.
> 

I think there is more to discuss on this topic.  I'll take this up in a separate thread.  

> > still maintain a reasonable level of robustness in the 
> overlay.  Nodes 
> > may choose to accept data items larger than this upper bound - 
> > however, the data owner must be prepared for a rejection of 
> the store 
> > request when the data is larger than the bound.
> > 
> > Thoughts? 
> 
> We considered exactly this design during the early design 
> phase of RELOAD. Unfortunately, it's unstable. The problem is 
> that the set of nodes assigned to store a data value shifts over time.
> So, I may store value X with node A, which can accept it, but 
> then node B takes over some of node A's space and node B 
> can't store the data and it just silently vanishes. In order 
> to provide any reasonable level of service expectation, the 
> nodes have to be willing to provide more or less the same 
> level of service as far as storage goes.
> 

I think we agree on this - my concern was coming from incorrectly thinking that there is no size requirement specified for an overlay.  Designing an overlay where nodes may have variable bounds doesn't make sense and I wasn't suggesting we go there. 

Vidya

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 15 Nov 2008 10:41:56 -0800

At Fri, 14 Nov 2008 22:59:58 -0800,
Narayanan, Vidya wrote:
> It is the minimum requirement I was talking about.  I agree that all
> peers must support the minimum requirement.  Turns out I didn't read
> the draft carefully enough - it looks like the size is part of the
> overlay configuration.  Sorry for missing that.  However, I don't
> see an error code for rejecting a store request due to its size - it
> may help to have one.

Yeah, in general error codes in RELOAD aren't 100% baked.

-Ekr