Subject: Re: [icnrg] icnrg Digest, Vol 101, Issue 10

From: Ken Calvert <[email-masked]>
Date: Sat, 8 Aug 2020 16:07:06 -0400

Dave said:
> I’m starting a new thread on this, since it’s coming at some of the architectural choices from a broader point of view than what the producers/consumers of application data may need in a Manifest. The implications, if you buy into my general proposition, touch on a number of aspects of Manifest design, including extensibility tradeoffs, what goes into the base required functionality, and how we view both security and privacy issues.
> 
> Here’s the basic premise. I believe we want Manifests to be useful to any element that stores, fetches, or caches the content described by a manifest. This includes both:
> 
> Repositories (in NDN terms “Repos”) that provide non-volatile storage of content produced elsewhere and act both as:
> 
> archival storage that can survive the temporary or possibly permanent unavailability of the original producer of data
> A form of distributed cache in which content can be replicated and advertised into routing the same way the origin producer would.
> Forwarder Caches (Content Stores) that can provide various optimizations beyond simple on-demand opportunistic caching to improve latency and/or robustness to downstream consumers. This would include watching for requested Manifests and in addition to caching them, using information in the Manifest to pre-fetch content likely to be requested by one or more downstream consumers.
> 
It seems to me that there is an important difference between these two kinds of things.  I regard the first thing as being outside the fundamental infrastructure of the network, and on the same footing as any other user with respect to network service providers (NSP) - somewhat analogous to DNS servers.  The second is part of the network infrastructure, and is controlled by the NSP.

The point is that it's a lot harder to change specs that are relied on by NSPs.  We are a long, long way from this being a serious problem today, but perhaps it's worth keeping IETF history in mind.  To that end, having a NO-COST option for the application to hide the manifest structure from NSP forwarders, right from the beginning (as I think is proposed under #3 below) is very important.
> Both of these require that Manifests have, I believe at a minimum, the following characteristics:
> 
> Manifests have Name Constructors so that these non-application elements are able to form Interests to fetch the content described in the Manifest without knowing application-specific namespace conventions.
> 
> We have some generally understood annotations (or whatever we decide to call this restricted form of metadata) to give hints about:
> 
> optimal/desirable fetch order
> relative importance for availability when producer is either not reachable or down
> interdependence of pieces if partial and not total to help drive cache replacement policies
> Clear and simple binding to the “right” security context so that there is not a complex explosion of signing and encryption keys. My recommendation would to allow only two security contexts for Manifests
> 
> signed/encrypted by keys used for application data. Such data would not be available to the intermediaries descried above.
> signed/encrypted under by the keys used for the Manifest itself. Such data would be available to any/all elements that are given the keys associated with the names in the Manifest’s namespace.
> any other goop in a Manifest (e.g. general application metadata) is treated as if were any other transparently cached application data, although I argue elsewhere against trying to support general application metadata in Manifests.
> 
> So, as a top level question, do people think these are important and/or reasonable requirements to accommodate in our Manifest architecture?
> 
FWIW, I think these are reasonable.  I agree with the general position that a base manifest structure should be defined in such a way that it can be implemented in a common library, and used by many(?) applications that don't need anything more than a way to abstract from (and amortize the cost of) chunking, signing, and possibly encrypting large blobs of bits.  I think it is not useful to define a highly abstract structure that needs to be re-implemented (or "profiled") for every application - especially at this early stage of the game.

Another top-level question for me is whether there is consensus that prefetching (by network elements) is something that a base manifest spec should be designed to support?  It seems to me to be very difficult to calculate the tradeoff of benefit with the cost to applications that don't need/want it.  If it is supported, an application hint "don't prefetch" would seem to be required (without requiring the manifest itself to be encrypted end-to-end).
> I’ll note that it is of course possible to design a completely separate data structure and named elements for use by Repositories and forwarders, but that seems to me to be way more complex than just viewing manifests in this broader context.
> 

Agree with this, but IMHO it makes the "signed/encrypted by keys used for application data" option very important.

Ken

---

From: "David R. Oran" <[email-masked]>
Date: Sun, 09 Aug 2020 10:10:12 -0400

On 8 Aug 2020, at 16:07, Ken Calvert wrote:

> Dave said:
>> I’m starting a new thread on this, since it’s coming at some of 
>> the architectural choices from a broader point of view than what the 
>> producers/consumers of application data may need in a Manifest. The 
>> implications, if you buy into my general proposition, touch on a 
>> number of aspects of Manifest design, including extensibility 
>> tradeoffs, what goes into the base required functionality, and how we 
>> view both security and privacy issues.
>>
>> Here’s the basic premise. I believe we want Manifests to be useful 
>> to any element that stores, fetches, or caches the content described 
>> by a manifest. This includes both:
>>
>> Repositories (in NDN terms “Repos”) that provide non-volatile 
>> storage of content produced elsewhere and act both as:
>>
>> archival storage that can survive the temporary or possibly permanent 
>> unavailability of the original producer of data
>> A form of distributed cache in which content can be replicated and 
>> advertised into routing the same way the origin producer would.
>> Forwarder Caches (Content Stores) that can provide various 
>> optimizations beyond simple on-demand opportunistic caching to 
>> improve latency and/or robustness to downstream consumers. This would 
>> include watching for requested Manifests and in addition to caching 
>> them, using information in the Manifest to pre-fetch content likely 
>> to be requested by one or more downstream consumers.
>>
> It seems to me that there is an important difference between these two 
> kinds of things.  I regard the first thing as being outside the 
> fundamental infrastructure of the network, and on the same footing as 
> any other user with respect to network service providers (NSP) - 
> somewhat analogous to DNS servers.  The second is part of the network 
> infrastructure, and is controlled by the NSP.
>

Yes, they are certainly different in this regard. One question is 
whether that difference matters for what features would be basic to a 
manifest design. My sense is that in terms of Name Constructors and 
annotations the two things are essentially similar. Where they could 
very well differ is in the security context, in the sense that the 
application management might be willing to transfer custody to a repo 
(in the sense of allowing it to fetch and store the content) and not be 
willing to allow an NSP forwarding Interests and Data to figure out what 
to cache (although of course there is nothing to stop them from caching 
any Data packet that might fly by).

If there is a need to express a manifest in different security contexts 
it seems the same straightforward method works for this as for 
application data. You just produce two (or more) versions of the 
Manifest (which would have different names in the Manifest namespace) 
and encrypt them with different keys to be provided to the corresponding 
elements. Now, I do think this would not be common, since in the case of 
Repos I would think the application management would give the keys for 
the same encrypted version to each CDN partner they would like to have 
store the content. I’m less sure that logic applies to on-path NSPs, 
depending on how sensitive the information leaked in the Manifest is. 
Even the case of basic FLIC we will need to have security considerations 
around what gets leaked and whether there are further linkability 
vulnerabilities exhibited by Manifests that are’t present in plain old 
named data fetch.

> The point is that it's a lot harder to change specs that are relied on 
> by NSPs.  We are a long, long way from this being a serious problem 
> today, but perhaps it's worth keeping IETF history in mind.  To that 
> end, having a NO-COST option for the application to hide the manifest 
> structure from NSP forwarders, right from the beginning (as I think is 
> proposed under #3 below) is very important.

Since every ICN router is a middlebox, I suppose this applies to every 
single little protocol element of our specifications. I don’t think 
Manifests make this any worse.

>> Manifests have Name Constructors so that these non-application 
>> elements are able to form Interests to fetch the content described in 
>> the Manifest without knowing application-specific namespace 
>> conventions.
>>
>> We have some generally understood annotations (or whatever we decide 
>> to call this restricted form of metadata) to give hints about:
>>
>> Clear and simple binding to the “right” security context so that 
>> there is not a complex explosion of signing and encryption keys. My 
>> recommendation would to allow only two security contexts for 
>> Manifests
>>
>> signed/encrypted by keys used for application data. Such data would 
>> not be available to the intermediaries descried above.
>> signed/encrypted under by the keys used for the Manifest itself. Such 
>> data would be available to any/all elements that are given the keys 
>> associated with the names in the Manifest’s namespace.
>> any other goop in a Manifest (e.g. general application metadata) is 
>> treated as if were any other transparently cached application data, 
>> although I argue elsewhere against trying to support general 
>> application metadata in Manifests.
>>
>> So, as a top level question, do people think these are important 
>> and/or reasonable requirements to accommodate in our Manifest 
>> architecture?
>>
> FWIW, I think these are reasonable.  I agree with the general position 
> that a base manifest structure should be defined in such a way that it 
> can be implemented in a common library, and used by many(?) 
> applications that don't need anything more than a way to abstract from 
> (and amortize the cost of) chunking, signing, and possibly encrypting 
> large blobs of bits.  I think it is not useful to define a highly 
> abstract structure that needs to be re-implemented (or "profiled") for 
> every application - especially at this early stage of the game.
>
Agree.

> Another top-level question for me is whether there is consensus that 
> prefetching (by network elements) is something that a base manifest 
> spec should be designed to support?  It seems to me to be very 
> difficult to calculate the tradeoff of benefit with the cost to 
> applications that don't need/want it.  If it is supported, an 
> application hint "don't prefetch" would seem to be required (without 
> requiring the manifest itself to be encrypted end-to-end).
I would not bother with the stuff I’m proposing if it was not usable 
by Repos. I think “don’t prefetch” would be fine but I sort of 
assumed that if a pointer wasn’t marked for prefetch, that would be 
the semantic. Like importance, my thinking is that this would be encoded 
as a signed integer, since the desired behavior is to express the 
relative rankings of the pointers rather than some absolute directive.

>> I’ll note that it is of course possible to design a completely 
>> separate data structure and named elements for use by Repositories 
>> and forwarders, but that seems to me to be way more complex than just 
>> viewing manifests in this broader context.
>>
>
> Agree with this, but IMHO it makes the "signed/encrypted by keys used 
> for application data" option very important.
>
Yes. My preference is to avoid if possible anything in a Manifest that 
has to be signed by the keys used for application data.

> _______________________________________________
> icnrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/icnrg

DaveO