Subject: [icnrg] FLIC Interest Construction

From: Marc Mosko <[email-masked]>
Date: Mon, 3 Aug 2020 18:29:52 +0000

This is one of a couple of emails on FLIC topics from today’s call.  This email focuses on how a consumer reading a FLIC manifest constructs Interest packets for each entry it wants to retrieve.

In the -02 draft, this knowledge was called Namespaces, but we will rename that to something like NameConstructor or NameTemplate.  I’ll run with NameConstructor (NC).  It is a mechanism by which a Manifest can tell a reader what structure an interest should take.  The examples in the draft are hash naming (mostly a ccnx mode), segmented naming (which I think ndn would likely use) and single prefix naming (middle ground).

There are two main reasons to have Name Constructors specified in a manifest.  One is to help bridge the NDN and CCN worlds, so one could use a namespace preferred by the protocol (e.g. distinct names per data or nameless objects).  The other is to make a Manifest application-independent.  If a FLIC manifest is to be like a universal data organization mechanism, it needs to be explicit about how to fetch the data.  I do not think one can make a general purpose manifest that allows the sharing of content if the name construction mechanism is implicit.  Different publishers or different applications even within a publisher might use different conventions.  There needs to be a way for an application reading the FLIC manifest to know how to make the Interest names.

Note that independent of the NC, a Manifest supports specifying Locators.  The locators are used as the routing hints in NDN or in the Interest names in CCN nameless objects.   For NDN, NC vs locators is a somewhat orthogonal issue.  For CCNx, Locators are more interwoven, as one could only use Locators with Hash naming (i.e. nameless objects).

Option 1: Multiple Name Constructors assigned by HashGroup (This is the option in the draft)

A manifest can define a NC specification that has a NameConsturctor ID (NCID) and the rule to construct names, along with needed data like the actual name prefix.  Each hashgroup says which NCID it uses.

NC are not inherited from parent manifests.  Each manifest must be explicit about how the names are constructed.

The method explicitly supports multiple naming conventions within one Manifest object.  The main usecase is to allow one HashGroup to point to other Manifest objects that might be in one naming convention (e.g. segmented naming) and another HashGroup to point to data objects that might use another naming convetion (e.g. hash-based naming).

Option 2: A single NC per Manifest object

Each manifest object only uses one NC.  It would be essential Option #1, but without HashGroups getting to pick which NC they use, there would be only one NC for the whole Manifest Object.  Each manifest object must specify its NC.

Option 2a: Allow inheritance of NC

A parent manifest could specify the NC that is used by children.   The NC could change as one reads down a tree.

This option would make pre-fetching difficult.  If the top-level manifest with the NC went via one path and internal manifests via another path, the second set of routers for the internal nodes would have no way to pre-fetch or do other things because they would not understand how to construct the Interest names.

Option 3: Implicit NC

The way to construct an Interest name is implicit by the application.   If someone gives you a FLIC manifest out of the blue, you do not know what convention to use and must be able to associate the manifest with a specific application or domain to fetch its contents.

Routers could not prefetch these, unless they assume some likely naming schemes.   Or the routers would need to guess or apply ML to the interests they see to deduce the possible name construction.

Option N: Others?

---

From: [email-masked]
Date: Wed, 5 Aug 2020 23:06:00 -0700 (PDT)

> There are two main reasons to have Name Constructors specified in a 
> manifest. One is to help bridge the NDN and CCN worlds, so one could 
> use a namespace preferred by the protocol (e.g. distinct names per 
> data or nameless objects). The other is to make a Manifest 
> application-independent. If a FLIC manifest is to be like a universal 
> data organization mechanism, it needs to be explicit about how to 
> fetch the data. I do not think one can make a general purpose manifest 
> that allows the sharing of content if the name construction mechanism 
> is implicit.

great wake-up call!

"general purpose = explicit name construction" - fully agreed. We should 
add IPFS, Secure Scuttlebutt and Hypercore to the mentioned worlds.

Since the time when NDN and CCNx did not talk to each other, CCN-lite 
had to auto-detect the "packet's world" by guessing it from the first 
few bits of a packet. CCN-lite also had a shim to encapsulate the 
datagram types, separate CCNx and NDN from pure NFN. Finally a chance to 
overcome this.

IPFS uses "multihash encoding" for hash agility i.e., they call it 
"self-describing hash values", see https://multiformats.io/multihash/ 
There is also the multiformats project to cover other planets/worlds, 
see https://multiformats.io/

When it comes to assigned numbers for hash functions, see the end of 
page https://github.com/multiformats/multihash/issues/49 by which I 
mean: let's not redo this with a nth multihash encoding.

If NDN and CCNx would move in that "multi-" direction, above guesswork 
would not be necessary anymore. For example you could use a single UDP 
port for all your p2p ICN overlays, which also means that you could 
reuse other (multi-) project's p2p libraries and inject NDN or CCNx or 
DAT names found in "our" manifests. I should note that the 
self-describing packet network subproject -called multigram- has not 
even started, probably because others only consider http so far - a 
great opportunity to fill that void and align it with Name Constructors.

Best, c

On Mon, 3 Aug 2020, Marc Mosko wrote:

> 
> This is one of a couple of emails on FLIC topics from today’s call.  This email focuses
> on how a consumer reading a FLIC manifest constructs Interest packets for each entry it
> wants to retrieve.
>
> 
> 
> In the -02 draft, this knowledge was called Namespaces, but we will rename that to
> something like NameConstructor or NameTemplate.  I’ll run with NameConstructor (NC). 
> It is a mechanism by which a Manifest can tell a reader what structure an interest
> should take.  The examples in the draft are hash naming (mostly a ccnx mode), segmented
> naming (which I think ndn would likely use) and single prefix naming (middle ground).
>
> 
> 
> There are two main reasons to have Name Constructors specified in a manifest.  One is
> to help bridge the NDN and CCN worlds, so one could use a namespace preferred by the
> protocol (e.g. distinct names per data or nameless objects).  The other is to make a
> Manifest application-independent.  If a FLIC manifest is to be like a universal data
> organization mechanism, it needs to be explicit about how to fetch the data.  I do not
> think one can make a general purpose manifest that allows the sharing of content if the
> name construction mechanism is implicit.  Different publishers or different
> applications even within a publisher might use different conventions.  There needs to
> be a way for an application reading the FLIC manifest to know how to make the Interest
> names.
>
> 
> 
> Note that independent of the NC, a Manifest supports specifying Locators.  The locators
> are used as the routing hints in NDN or in the Interest names in CCN nameless objects.
>   For NDN, NC vs locators is a somewhat orthogonal issue.  For CCNx, Locators are more
> interwoven, as one could only use Locators with Hash naming (i.e. nameless objects).
>
> 
> 
> A manifest can define a NC specification that has a NameConsturctor ID (NCID) and the
> rule to construct names, along with needed data like the actual name prefix.  Each
> hashgroup says which NCID it uses. 
>
> 
> 
> The method explicitly supports multiple naming conventions within one Manifest object. 
> The main usecase is to allow one HashGroup to point to other Manifest objects that
> might be in one naming convention (e.g. segmented naming) and another HashGroup to
> point to data objects that might use another naming convetion (e.g. hash-based naming).
>
> 
> 
> Each manifest object only uses one NC.  It would be essential Option #1, but without
> HashGroups getting to pick which NC they use, there would be only one NC for the whole
> Manifest Object.  Each manifest object must specify its NC.
>
> 
> 
> This option would make pre-fetching difficult.  If the top-level manifest with the NC
> went via one path and internal manifests via another path, the second set of routers
> for the internal nodes would have no way to pre-fetch or do other things because they
> would not understand how to construct the Interest names.
>
> 
> 
> Option 3: Implicit NC
>
> 
> 
> The way to construct an Interest name is implicit by the application.   If someone
> gives you a FLIC manifest out of the blue, you do not know what convention to use and
> must be able to associate the manifest with a specific application or domain to fetch
> its contents.
>
> 
> 
> Routers could not prefetch these, unless they assume some likely naming schemes.   Or
> the routers would need to guess or apply ML to the interests they see to deduce the
> possible name construction.
>
> 
> 
> Option N: Others?
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
> 
> 
>

---

From: "David R. Oran" <[email-masked]>
Date: Thu, 06 Aug 2020 09:44:41 -0400

On 6 Aug 2020, at 2:06, [email-masked] wrote:

>> application-independent. If a FLIC manifest is to be like a 
>> universal data organization mechanism, it needs to be explicit about 
>> how to fetch the data. I do not think one can make a general purpose 
>> manifest that allows the sharing of content if the name construction 
>> mechanism is implicit.
>
> great wake-up call!
>
Thanks also. It seems we have reached some consensus on at least one 
issue, which is that the Manifest design needs to support Name 
Constructors on day one. Exactly how that gets encoded and how 
extensible it is beyond the three schemes Marc proposed is yet to be 
settled. Right?

> "general purpose = explicit name construction" - fully agreed. We 
> should add IPFS, Secure Scuttlebutt and Hypercore to the mentioned 
> worlds.
>
Sure, why not, although it might complicate any text explaining the 
applicability of the specification, as if we try to go beyond NDN and 
CCNx it will raise a lot of questions/requirements to demonstrate how 
you would apply it to the other protocols.

> datagram types, separate CCNx and NDN from pure NFN. Finally a chance 
> to overcome this.
>
Ah, yes, just like “magic numbers” in the first bytes of a Unix 
file…

> great opportunity to fill that void and align it with Name 
I see how this solves the problem of how to demultiplex multiple 
protocols arriving at the same endpoint “address”, but I don’t see 
how it tells you what a Name Constructor tells you, which is much 
richer.

I would suggest we consider this orthogonal for the moment, as there are 
a ton of IETF RFCs on the subject, for things like STUN, RTP and WebRTC, 
DoT and DoH, etc.

Unless I’m misunderstanding what you’e getting at…

>> something like NameConstructor or NameTemplate.  I’ll run with 
>> NameConstructor (NC). It is a mechanism by which a Manifest can tell 
>> a reader what structure an interest
>> manifest.  One is
>> Option 1: Multiple Name Constructors assigned by HashGroup (This is 
>> the option in the
>> draft)
>>
>>
>>
>> NC are not inherited from parent manifests.  Each manifest must be 
>> explicit about how
>> the names are constructed.
>>
>>
>>
>> The method explicitly supports multiple naming conventions within one 
>> Manifest object. The main usecase is to allow one HashGroup to point 
>> to other Manifest objects that
>> A parent manifest could specify the NC that is used by children.  
>>  The NC could change
>> as one reads down a tree.
>>
>>
>>
>> ______________________________________________
> icnrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/icnrg

DaveO