Subject: [icnrg] Some thoughts on architectural choices for Manifests

From: "David R. Oran" <[email-masked]>
Date: Wed, 05 Aug 2020 13:16:27 -0400

These are a bit stream-of-consciousness, and definitely susceptible to 
TL;DR, so treat accordingly.

I’ve been mulling over the issues we discussed at the ICNRG Interim 
around FLIC and I’d like to unpack a few things along a slightly 
different axis than Marc did in his two excellent messages.

My first thought is that the genesis of FLIC along the lines of iNodes 
in Unix was probably a really good way to crystalize the problem of 
representing big single objects that need to be chopped up into pieces. 
This is what happens when layering a file system directly onto a disk 
with fixed-size blocks. If one carries this through directly and solves 
**only** that problem, it means FLIC’s design derives from the 
following:

1. It is designed to represent the chunking of a single object, not 
enumerating a collection or other uses.

2. It has a hierarchical (tree, but possibly digraph) representation 
since the data structure itself has to fit in the same chunking limits 
as the underlying data objects.

3. It needs the ability to extend/append to cover cases where the object 
itself supports append operations.

4. It may need some indexing goop and size information if seek 
operations to particular bytes in the object is needed. If everything is 
fixed-size-per chunk and that size is know a priori, pointer array 
indexing suffices.

These seem fundamental to the model, but there are some less obvious 
implications if one adheres closely to the iNode/filesystem analogy:

- The data structure is meant to be interpreted by a **single** client 
piece of code, not directly read/written by multiple pieces of code that 
don’t know about one another or are unsure what the thing its pointers 
point to are.

- It’s a data structure meant to be used by some middle system 
software, not random applications

- It is closely bound to the particular client wanting to use it - most 
different filesystems in fact use a different format for iNodes - they 
are somewhat different for EXTx, VFS, HFS+, etc. and the “on-disk 
structures” are incompatible. The implication is that if we want 
different structures for ICN collections as the technology matures, one 
would expect the FLIC data structure to change in incompatible ways.

- modern file systems actually eschew the first-level iNode entries for 
small files whose data fits in a single disk block and instead embed the 
data directly in the iNode containing the directory entry for the file. 
This has now mixed things together quite a bit in modern file systems, 
so iNodes are’t the simple low-level thing they once were and 
directories are not so cleanly layered on iNodes as they once were. The 
performance advantages are dramatic, and one might expect pretty big 
performance gains for doing the same for ICN. This means, at a minimum, 
that supporting packaging a data object “inside” a manifest might in 
fact be pretty important. There are a bunch of ways to do this, but not 
considering this now would in my view be a mistake.

So, if we take this view for the architecture for FLIC:

- FLIC is just a convention for a private data structure that can be 
used by an application
- FLIC is for breaking down single objects; not for describing any other 
kind of collection.
- There is no expectation that a FLIC Manifest is understandable by 
multiple applications of the same data
- It’s up to application “magic” using either namespace 
conventions or some separate discovery machinery to figure out what Name 
to use in an Interest to fetch a FLIC manifest (or piece thereof).
- We don’t need extension mechanisms, since an application can just 
make up its own manifest format by modifying FLIC as needed - the code 
will just “follow” the app in the same way that iNode formats are 
closely bound to the particular filesystem built on top.

If you find the above exposition at all convincing, I think there are a 
number of possible ways forward. Let me try to enumerate them:

1. Define FLIC as limited to the above, and punt anything else for 
“later”.
2. Keep the current FLIC design, but build in some extensibility 
features (but don’t define any extensions now). This would include at 
a minimum a versioning scheme and some TLVs so additional things other 
than pointer arrays of hashes can be expressed.
3. Define FLIC as above, but instead make it an “interior” data 
structure of a more general Manifest format that we work on in parallel. 
That more general data structure would encompass the stuff we have 
currently put in the base FLIC Manifest. A good design would allow the 
FLIC manifest to either be embedded inside the general Manifest, or 
externally referenced via one of the pointers.
4. Continue down the current path, by having one general Manifest format 
that is extensible and contains the features we considered important, 
like Name Construtors and annotated pointers.

In addition to the above, I think any general Manifest format should 
allow direct embedding of content objects to handle the cases of simple 
small data (e.g. IoT sensors etc.) so fetch doesn’t require extra 
RTTs. This might be useful even in the case of FLIC, to allow the same 
primitive data object to either be fetched independently, or with 
Manifest, and the signature bound to the manifest rather than the data 
object, making re-signing independent of the original data producer 
code.

I have my own views on which of these directions we should pursue, but 
for the purposes of this email the question I’d like to ask is whether 
the taxonomy above is a good way to think about this and are there other 
options for how to architecturally represent things I haven’t thought 
of?

DaveO

---

From: [email-masked]
Date: Wed, 5 Aug 2020 19:33:03 -0700 (PDT)

Hi Dave and Marc,

great to have your "unpacking"! After the ICNRG interim meeting I had
started to write down my viewpoint, yesterday I revised it and I was
about to revise it again - maybe I should just blast it out, same
TL;DR disclaimer.

Summary: I distinguish between
- packet (data object)
- real blob (a packet's content)
- virtual blob (content as defined by a manifest packet)
Manifests are the mechanism to define virtual blobs - full stop. The
"full stop" excludes object composition.

Relating to Dave's enumeration of options, my writeup would belong to #2
("Keep the current FLIC design, but build in some extensibility
features"). It would be a very lean FLIC, though.

Regarding "direct embedding" - I also think that this is essential to
have (I had called it inlining).

Regarding Marc's comments on annotation (thanks, too!): my writeup
belongs to your group 3 ("Separate Data/Content objects within the
FLIC world") in a strict way, the reason being that the encoding size
of a manifest's annotations in general cannot be bound, hence would be
the first candidate to benefit from virtual blobs as offered by the
manifest construct. If the encoding is small enough, the "separate
data structure" can be "directly embedded". I guess from your text
that such embedding is not equal to your "option 2" (parallel groups
in one manifest), but it could be close as you wrote yourself.

So here it goes, perhaps not a full manifest manifesto, but laying out
an argument for keeping FLIC minimal and version-resistant.

best, c

---

First, I re-tell the FLIC story in new terms (no reference to inodes)
for positioning FLIC and what it achieves. Then I will look into the
"parallel datastructure" concern and how directory tree organization
should be handled. I also include a grammar for such a lean FLIC.

As a preparation I introduce "derivation functions" which take some
packets (signed data objects) and produce new packets.  The first
derivation function of interest is "make_virtual_blob(h_1,..h_n)"
whose result is exactly one packet of type manifest that contains the
given sequence of packet hash values, these packet being either a
normal data object ("real blob") or a manifest spec.

A packet with a manifest spec defines a "virtual blob" which is data
of any size. The hash name of the manifest packet becomes the name of
the virtual blob; the virtual blob inherits the signature of the
manifest packet.

   real_blob_pkt.content() = justs tap into the named data object
   virtual_blob_pkt.content() = gather all content by traversing the manifest tree

   note that for real blobs, real_blob_pkt.content() equals real_blob_pkt.payload
   (introducing the notation of field access vs a procedure)

A manifest is a claim, namely that there exists a virtual blob which
is the concatenating of all referenced content: it's a claim
(attribution to a given manifest) that can only be verified by
materializing the virtual blob. The claim made by a manifest can fail
for several reasons, one being that some of the included hash values
never had a corresponding data packet or that some of the referenced
packets have been forgotten, or that the manifest spec is broken.

A second derivation function is "get_virtual_blob_content_size(h)".
Again, it is a claim about a manifest that can only be verified by
materializing the virtual blob. But unlike the first function, this
function is not essential for defining the virtual blob, it is mere
decoration (meta data). There are many more decorations:

vblob_size_in_records
vblob_size_in_UTF8_chars
vblob_tree_depth
vblob_number_of_leaf_blobs
creation_time_in_sec

The last example stands for a decoration that cannot be verified.

My suggestion is to leave decorations outside the FLIC draft, except
for the possibility to have decorations. That is, it's ok to use a
manifest for defining resource forks. I will come back to size
limitations in just a moment.

Resource forks are a well known pattern that also map well to the
networking world, for example layered video encodings, labeling
information for a video etc. These additional resources are optional
and not needed to build the virtual blob abstraction - they enrich
it. And these additional (parallel) resources all have the problem of
potentially being desynchronized - after all they are all claims that
the consuming end has to handle with caution.

Size is one among many of such decorations. Yes, one could request
that all manifest MUST include the size resource fork. But then the
FLIC draft must explain how a virtual blob consumer should handle
manifests that lack that resource fork, or what to do when traversal
reveals that a size value was wrong. The answer will probably include
a sentence like "if one of the correctness checks fails you must
revert to deriving that property yourself", which is another way of
saying that handling decoration data is decoration-specific. So better
have a separate "FLIC decoration draft" explaining the various ways of
adding decorations, including size: do it per-manifest (= store the
sum of all referenced subtrees sizes), or as a single table at the
root manifest that keeps the (size) labels for all nodes of the tree
etc.

My other argument (beside diminishing decorations as claims) for not
being concerned about potentially non-synchronous data is that code
for producing and consuming manifests itself is constantly evolving
and is not synchronized, due to bugs or partial implementations. Even
when resource forks themselves would be synchronized with a manifest's
content, the softwares (plural) will not - the next version of a
producer software may introduce wrong decorations expected by all
existing consumer software. Said differently: Synchronization loops
pass through software - making some decorations mandatory will not
prevent desynchronization.

Note that decorations can be useful for applications _and_ for the
manifest traversal algorithms - I did not keep these two usages apart,
there is only one decorations field. As a side effect, manifests thus
provide a "decorated virtual blob" abstraction.

When it comes to the question of using manifests to express
directories or similar trees of components, we should not confound
decoration with object composition. It looks as if one could use the
(hash) table as a table for sub-component data. One argument here is
that we will bump into the size limitation of (manifest) packets
wherefore we should compose objects out of virtual blobs _and_ the
composition spec also be stored in a separate virtual blob. For
example, what if the file name is larger than fits in one packet?
It is much cleaner to store the mapping table (that maps from file
attributes like names to virtual blobs) in a virtual blob.

Potential confusion can arise when we build a directory tree where
files and directories are all stored in virtual blobs. Isn't this
a manifest tree of manifest trees? No, it is not because they operate
at different levels. When traversing a manifest tree, we extract hash
references from the manifest's fields.  When traversing directories,
we extract hash references from inside the virtual blob (defined by
that manifest). These are very different memory zones.

Here is a writeup in form of a grammar/definitions:

packet := length-limited data object (NDN, CCN)

          I assume that a data object has type information such that
          "real blob" can be distinguished from "manifest packet". Having
          a hash name and retrieving the data object will tell us what
          kind of packet it is. Anything else (e.g., tagging hash
          pointers whether they are pointing to real blobs or a manifest
          pkt is a claim and thus unreliable. It's fine to keep such
          tags as a decoration but these are not needed to implement
          the virtual blob abstraction.

r_pkt := packet with TLV for "real blob"
m_pkt := packet with TLV for "manifest"

r_pkt.payload = any                              # the "real blob"
m_pkt.payload = manifest_spec

manifest_spec := deco + cont_seq                 # two fields, must fit in one packet
deco          := None | hashval | inlined
cont_seq      := sequence of (hashval | inlined)

vblob_content(h) is defined as follows:
           if type(fetch(h)) == r_pkt:
                   fetch(h).payload
           elif type(fetch(h)) == m_pkt:
                   concat( vblob_content(j) for all j in fetch(h).payload.cont_seq )
                   # if j is inlined content, this is used instead of vblob_content(j)
           else:
                   fail

# The following would be part of a separate "FLIC decor draft":

vblob_deco(h, fork_name) is defined as follows:
           if type(fetch(h)) == r_pkt:
                   None
           elif type(fetch(h)) == m_pkt:
                   d = fetch(h).payload.deco
                   if d == None:
                        None
                   elif d is inlined:
                        d
                   else:
                        vblob_content(d).access(fork_name)
           # the mapping for fork_name to the resource, labeled as "access" above,
           # would also be part of that separate "FLIC decor draft"

# How would composition e.g., directories look? We show content access
# via a file's name and assume that the directory data structure maps
# a file name to some hash value (which names a virtual blob
# containing the file's content). This content access has no
# similarities with above content retrievals, as it calls
# vblob_content() twice:

dir_get_content(dir_hash_val, file_name):
         vblob_content( vblob_content(dir_hash_val).access(file_name) )

# we can decorate files because they are stored as virtual blobs:

dir_get_deco(dir_hash_val, file_name):
         vblob_deco( vblob_content(dir_hash_val).access(file_name) )

Composition does not require new NDN/CCN types: it's an application
level issue whether it wants to define a "type decoration", or put a
magic number (type) inside each directory virtual blob, and how it
encodes the composition spec.

Conclusions: The sole role of manifests should be to abstract away
from blob size limitations, providing transparent access to virtual
blobs (vblobs) of arbitrary size. A manifest defined as a list of hash
values, together with manifest recursion, are all the tricks we need -
plus name constructors for forwarding reasons, not discussed in this
writeup. "Decorations" (meta data) are important for speedup reasons,
so there must be a hook for them. As decorations may require the
support of vblobs themselves, the suggestion is that the hook comes in
form of a single hash value (that references a vblob), instead of
hardwiring decoration information into the manifest packet. Inlining
is still possible as an optimization. It's also possible to use the
manifest construct for simply decorating a real or virtual blob. For
example, decorations could carry additional signatures, or type
information outside NDN/CCN's TLV system. Composing objects (stored in
vblobs) is different from glueing together blobs (to create
vblobs). Similar to storing decoration data, composition should be
layered on top of the vblob abstraction using one vblob for every
component.

---

On Wed, 5 Aug 2020, David R. Oran wrote:

> 
> These are a bit stream-of-consciousness, and definitely susceptible to TL;DR, so treat
> accordingly.
> 
> I’ve been mulling over the issues we discussed at the ICNRG Interim around FLIC and I’d
> like to unpack a few things along a slightly different axis than Marc did in his two
> excellent messages.
> 
> My first thought is that the genesis of FLIC along the lines of iNodes in Unix was
> probably a really good way to crystalize the problem of representing big single objects
> that need to be chopped up into pieces. This is what happens when layering a file
> system directly onto a disk with fixed-size blocks. If one carries this through
> directly and solves only that problem, it means FLIC’s design derives from the
> following:
>
>  1.
>
>     It is designed to represent the chunking of a single object, not enumerating a
>     collection or other uses.
>
>  2.
>
>     It has a hierarchical (tree, but possibly digraph) representation since the data
>     structure itself has to fit in the same chunking limits as the underlying data
>     objects.
>
>  3.
>
>     It needs the ability to extend/append to cover cases where the object itself
>     supports append operations.
>
>  4.
>
>     It may need some indexing goop and size information if seek operations to
>     particular bytes in the object is needed. If everything is fixed-size-per chunk and
>     that size is know a priori, pointer array indexing suffices.
> 
> These seem fundamental to the model, but there are some less obvious implications if
> one adheres closely to the iNode/filesystem analogy:
>
>  * 
>
>     The data structure is meant to be interpreted by a single client piece of code, not
>     directly read/written by multiple pieces of code that don’t know about one another
>     or are unsure what the thing its pointers point to are.
>
>  * 
>
>     It’s a data structure meant to be used by some middle system software, not random
>     applications
>
>  * 
>
>     It is closely bound to the particular client wanting to use it - most different
>     filesystems in fact use a different format for iNodes - they are somewhat different
>     for EXTx, VFS, HFS+, etc. and the “on-disk structures” are incompatible. The
>     implication is that if we want different structures for ICN collections as the
>     technology matures, one would expect the FLIC data structure to change in
>     incompatible ways.
>
>  * 
>
>     modern file systems actually eschew the first-level iNode entries for small files
>     whose data fits in a single disk block and instead embed the data directly in the
>     iNode containing the directory entry for the file. This has now mixed things
>     together quite a bit in modern file systems, so iNodes are’t the simple low-level
>     thing they once were and directories are not so cleanly layered on iNodes as they
>     once were. The performance advantages are dramatic, and one might expect pretty big
>     performance gains for doing the same for ICN. This means, at a minimum, that
>     supporting packaging a data object “inside” a manifest might in fact be pretty
>     important. There are a bunch of ways to do this, but not considering this now would
>     in my view be a mistake.
> 
>  *  FLIC is just a convention for a private data structure that can be used by an
>     application
>  *  FLIC is for breaking down single objects; not for describing any other kind of
>     collection.
>  *  There is no expectation that a FLIC Manifest is understandable by multiple
>     applications of the same data
>  *  It’s up to application “magic” using either namespace conventions or some separate
>     discovery machinery to figure out what Name to use in an Interest to fetch a FLIC
>     manifest (or piece thereof).
>  *  We don’t need extension mechanisms, since an application can just make up its own
>     manifest format by modifying FLIC as needed - the code will just “follow” the app
>     in the same way that iNode formats are closely bound to the particular filesystem
>     built on top.
> 
> If you find the above exposition at all convincing, I think there are a number of
> possible ways forward. Let me try to enumerate them:
>
>  1. Define FLIC as limited to the above, and punt anything else for “later”.
>  2. Keep the current FLIC design, but build in some extensibility features (but don’t
>     define any extensions now). This would include at a minimum a versioning scheme and
>     some TLVs so additional things other than pointer arrays of hashes can be
>     expressed.
>  3. Define FLIC as above, but instead make it an “interior” data structure of a more
>     general Manifest format that we work on in parallel. That more general data
>     structure would encompass the stuff we have currently put in the base FLIC
>     Manifest. A good design would allow the FLIC manifest to either be embedded inside
>     the general Manifest, or externally referenced via one of the pointers.
>  4. Continue down the current path, by having one general Manifest format that is
>     extensible and contains the features we considered important, like Name Construtors
>     and annotated pointers.
> 
> In addition to the above, I think any general Manifest format should allow direct
> embedding of content objects to handle the cases of simple small data (e.g. IoT sensors
> etc.) so fetch doesn’t require extra RTTs. This might be useful even in the case of
> FLIC, to allow the same primitive data object to either be fetched independently, or
> with Manifest, and the signature bound to the manifest rather than the data object,
> making re-signing independent of the original data producer code.
> 
> I have my own views on which of these directions we should pursue, but for the purposes
> of this email the question I’d like to ask is whether the taxonomy above is a good way
> to think about this and are there other options for how to architecturally represent
> things I haven’t thought of?
> 
> DaveO
> 
> 
>

---

From: "David R. Oran" <[email-masked]>
Date: Thu, 06 Aug 2020 16:12:08 -0400

I’m still digesting this. Consider my comments more as questions, 
although all of you who know me also are aware of my predilection to 
couch questions as objections.

I also admit to not really understanding it completely after a couple of 
read-throughs. Some of this may be due to having to purge my translation 
cache of my usual understanding of some of the terms you have chosen. 
Feel free to respond, ponder for why the explanation isn’t getting 
through well to me, or just ignore.

> Summary: I distinguish between

I don’t like using the term “packet” this way for a networking 
audience, since this is not its usual meaning as a layer 3 PDU that is 
intended to be interpreted hop-by-hoop and not just end-to-end.

> - virtual blob (content as defined by a manifest packet)
I guess I don’t understand what is “virtual” about virtual blobs. 
Usually “virtual” means/implies “not bound by the resource 
constraints of the underlying system”. Also, I would have thought what 
is in the manifest are *pointers*  not virtualized representations of 
the object. If the mental model you’re proposing is like the 
translation/indirection of virtual memory systems, I think that’s 
likely to be just as limiting/problematic as the iNode analogy we just 
jettisoned.

> "full stop" excludes object composition.
Not sure what you mean by “object composition”. I thought what you 
were trying to exclude from FLIC’s design was the ability to handle 
sets/directories/groups/enumerations of independent objects.

> have (I had called it inlining).
Great! Let’s take this as a hard requirement then. It would also be 
good to have this work recursively - allowing the embedding/inlining of 
Manifests as well as primitive objects in case we have a multi-level 
architecture where some semantics are in “top level”, while others 
just have pointer arrays.

> manifest construct.

This makes me more than a little nervous. My mental model of annotations 
is that they are not generalized metadata but only “application 
independent hints to drive the fetching process of Interest/Data 
exchanges”. I’m even more nervous about accommodating generalized 
metadata or “sidecar” data along the lines of resource forks used by 
some file systems in manifests. This has the risk of complicating the 
security aspects quite considerably, as it means that either keying used 
for application data may need separate contexts for the data versus the 
annotations, or conversely that you need separate keying for the base 
manifest than the annotations. To quote the NSA, “We never met a data 
we didn’t want to capture and examine”.

Coming back to what started me down this rabbit hole is that in my metal 
model annotations are not arbitrarily big and hence tight limitations on 
their size (e.g. no bigger than a single  SHA256 hash or a few of them) 
would not bother me in the least.

> If the encoding is small enough, the "separate
Yes.

> packets (signed data objects) and produce new packets.

Are you sure you meant to say this? One purpose of Manifests is that you 
don’t want to sign individual packets, especially in hash-named 
schemes like CCNx. Perhaps this confusion is due to my mis-understanding 
of how you use the term “packet”. Don’t we agree that whether an 
input is already signed or not, the end result is that the individual 
pieces pointed to by the Manifest are not signed, but the signature over 
the Manifest covers all the pieces of the object. I *think* just signing 
the Manifest covers the needed security/trust requirements (assuming the 
Manifest is constructed by the same entity who contracted the original 
data) - Marc might weigh in here.

> The first
What about Merkle trees of Manifests?

> the virtual blob; the virtual blob inherits the signature of the
I’m not sure I understand what it means to “inherit” the 
signature. Do you mean “is incorporated into the overall signature 
calculation”? It seems you want to put a signature on each Manifest 
piece of a larger tree of manifest chunks, but I’m clearly missing 
something since that wouldn’t work.

> packets have been forgotten, or that the manifest spec is broken.
I’m not sure you can ever determine whether a given hash value has or 
ever had a corresponding data packet. You only know after the fact that 
you tried to fetch and it failed, but the networking uncertainty 
principle says that you can never tell the difference between 
nonexistent data and failure to fetch without connectivity to ask the 
original creator of the data. Also, depending on how Manifest indexing 
is done, it may be necessary to have bogus hashes in order to represent 
objects with “holes” in them.

> vblob_number_of_leaf_blobs

So, pulling this apart and using my proposed criteria for what kinds of 
annotations we want to permit in a Manifest, I would say:
> vblob_size_in_records -
no, since records are a purely application construct

> vblob_size_in_UTF8_chars
why is this different from size in bytes? And why would I care for the 
purposes of driving a fetch algorithm?

> vblob_tree_depth
yes, possibly useful

> vblob_number_of_leaf_blobs
yes, probably useful. If you have this and you require full packing of 
blobs, the depth can be inferred/calculated from this value.

> creation_time_in_sec
Isn’t this in the base protocol? And why seconds and not NTP time?

> The last example stands for a decoration that cannot be verified.
Define “verified”. You can certainly verify if (a)it’s in the 
future and hence impossible, or (b)it’s earlier than the creation time 
of the base object pieces (which may or may not be carried in the Data 
object portion messages of the fetch protocol).

> limitations in just a moment.
I’m certainly open to this, as long as the extension machinery is 
clear, and the rules for how these are secured are specified up front so 
we don’t have an explosion of independent security contexts as people 
define extensions.

> networking world, for example layered video encodings,
So, I don’t want to argue this, other than pointing out that nobody I 
know treats layered video encodings as resource forks. One could 
generalize all the way to “tracks” in a ISOFF video file (which 
aren’t done as resource forks since they have a thing like a Directory 
style Manifest at the front to tell you what the tracks are and how to 
index them). Also, at least video and audio tracks tend to be 
interleaved to optimize disk I/O.

> labeling

Sure, yes. Again, I think of them as optimizations for transport, 
storage and possibly caching purposes that don’t change the semantics 
of how any of the data in the actual objects are interpreted by the 
receiving application.

> saying that handling decoration data is decoration-specific.

again, if you treat these as “hints” that could be wrong or not 
present, the right expectations are set. Also, I would hope these would 
all be in the category of generalized fetch libraries and not specific 
to an application and whose values would not normally be exposed to 
application code.

> So better

I am of course happy to leave the specification of the individual 
annotations/decorations to separate drafts as long as we agree on the 
exact extension mechanism and that each one of these not have a 
completely different parsing structure. In other words, if we decide 
they are attached to each pointer in the hash pointer array as TLVs, 
then each specification is only the semantics of what each T means, and 
what L’s are allowed. If we decide that they are parallel arrays 
corresponding to the pointer arrays, then we should require that the 
array traversal be the same for all of them, and hopefully in lock-step 
with the traversal and matching of the base pointer arrays. If we decide 
it’s a completely independent object (I hope not, but anyway if we 
do), we should define in the base spec what the allowed structure of 
such objects are.

> and is not synchronized, due to bugs or partial implementations.
Agree. That’s why anything we do ought to support versioning.

> Even
Agree, so I would definitely say up front that none of the 
decorations/annotations shall be mandatory to implement, and certainly 
not mandatory to use.

> provide a "decorated virtual blob" abstraction.
I’m arguing the exact opposite, that these only affect storage, 
transport, and possibly caching in an application agnostic way and that 
anything that is to be consumed by an application be handled as actual 
application data with its own Name and corresponding Manifest.

> decoration with object composition.
Also agree. I see these as separate as well.

> It looks as if one could use the
While I agree in general with the size concern, the example is not a 
good one, since the name has to fit in an Interest packet when fetching 
the file, so it had better fit in the Manifest.

> files and directories are all stored in virtual blobs.
Yes, groups of groups, collections of collections, sets of sets, etc. 
are not the same thing as fragments of a single larger logical data 
object.

> Isn't this
> references from the manifest's fields.
Yes..

> When traversing directories,
> blobs (vblobs) of arbitrary size.

So, I don’t think that’s the only purpose of Manifests. I’ll post 
a separate message on that subject hopefully justifying my views.

> A manifest defined as a list of hash
> hardwiring decoration information into the manifest packet.

That may be ok, as long as certain restrictions are enforced, like
- requiring congruent traversal with the traversal of the pointers of 
the base Manifest structure,
- being application-independent so they are interpretable by things that 
don’t understand the application data
- that they not be of unbounded size, like the base application objects 
so they don’t become a dumping ground for random bits that should have 
been separately named and fetched.
- that they not require additional security contexts beyond those 
applied to secure the base manifest data.

> Inlining
> manifest construct for simply decorating a real or virtual blob.
If this recurses I sort of get off the bus. While it may be 
intellectually elegant to allow decorations to have decorations, we’d 
be stepping way over the bound that at least I had in my mind when 
proposing pointer annotations.

> For
Ditto for this. We don’t want to migrate all of namespace and 
application design into the building and interpreting of Manifests.

Whew, that’s all I can think of for now.

I hope to post a separate message tomorrow on my view about how 
Manifests ought to be useful to elements other than the producer and 
consumer of the underlying data, such as Repos and forwarder caches.

DaveO

DaveO

---

From: Marc Mosko <[email-masked]>
Date: Thu, 6 Aug 2020 20:37:15 +0000

Here are a few comments responding to topics from both Dave and Christian.

FLIC describes a single file.  No.  By definition, FLIC must always describe two files: the manifest and the data.  We choose to make those descriptions use the same data structure. We also choose that when reading a manifest, a reader does not know which pointers belong to which sub-tree until it actually fetches a pointer (unless there are annotations).  We choose that a node in the tree must be either a manifest node or a data node, not a hybrid node.  The manifest sub-tree is really just a FLIC representation of the manifest file and the data sub-tree (it's not so much a tree as the in-order traversal of leaf pointers) is a representation of the data file.  

In some FLIC arrangements, this can be made explicit.  For example, my root manifest has two hash groups.  One group pointing to the manifest tree and the second points to the data leaves.  Or, as is more common, they are just mixed I one hash group.

Inlining: Yes!  One use case we had for this was when someone wants to re-publish a manifest object (e.g. a content distributor wants to point its subscribers to local caches via locators), it would encapsulate the original root manifest in its re-written one so the consumer can verify in one step that they (a) got the thing they were asking for (from the interior manifest) and that someone they trust for such things (e.g. their ISP) tells them a good place to fetch it from.

Simplifying: I would be OK taking out all the namespace stuff, but allowing extensions at the Node level (i.e. top level of each manifest inside the security context) so a publisher could put in a NameCreator element (or any other valid extension) if it so chooses.  Otherwise, the process of forming an Interest is implicit in the application or someone just has to guess.  I would then suggest we immediately publish two extensions, one for hash-based naming (ccnx) and one for segmented naming (ndn).

To go along with the simplification, maybe we could define a range of Ts that define hash groups.  There's nothing technically different about the hashgroups, just having say 16 values that are all HashGroup that allows the app to assign its own app meanings.  This would allow the NameCreator extension to specify different rules for different HashGroup types, or a private producer could have their own secret rules for different HashGroup types.  This allows an app to use different NameCreator approaches for the manifest file and the data file.   It's like being able to use typed name components so an app can have its own rules.

Marc

On 8/5/20, 7:33 PM, "icnrg on behalf of [email-masked]" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Dave and Marc,

    best, c

    ---

    vblob_size_in_records
    vblob_size_in_UTF8_chars
    vblob_tree_depth

    ---

---

From: "David R. Oran" <[email-masked]>
Date: Fri, 07 Aug 2020 09:56:23 -0400

On 6 Aug 2020, at 16:37, Marc Mosko wrote:

> FLIC describes a single file.  No.  By definition, FLIC must always 
> describe two files: the manifest and the data.
Modulo some degree of discomfort using the term “file” as an 
abbreviation of “file-like object”, I agree with Marc here.

> We choose to make those descriptions use the same data structure. We 
> also choose that
> when reading a manifest, a reader does not know which pointers belong 
> to which sub-tree until it actually fetches a pointer (unless there 
> are annotations).
I think this is right, however isn’t it a good idea to enforce a 
canonical ordering so that you don’t need application knowledge to 
re-assemble the pieces of an object in the right order (or are you only 
referring to non-leaf manifests pieces)?

> We choose that a node in the tree must be
> either a manifest node or a data node, not a hybrid node.
That seems sensible simplification.

> The manifest sub-tree is really just a
> FLIC representation of the manifest file and the data sub-tree (it's 
> not so much a tree as the in-order traversal of leaf pointers) is a 
> representation of the data file.
So, looking at my above comment in the context of this sentence we are 
saying the manifest ensures that there is a canonical ordering of leaf 
manifests, and in leaf manifests, the order of the pointers is congruent 
with the order of the pointed-to data bytes, right?

>
> In some FLIC arrangements, this can be made explicit.  For example, my 
> root manifest has two hash groups.  One group pointing to the manifest 
> tree and the second points to the data leaves.  Or, as is more common, 
> they are just mixed I one hash group.
>
Seems fine. Might there be advantages (either in flexibility or 
simplicity) in separating them?

> Inlining: Yes!  One use case we had for this was when someone wants to 
> re-publish a manifest object (e.g. a content distributor wants to 
> point its subscribers to local caches via locators), it would 
> encapsulate the original root manifest in its re-written one so the 
> consumer can verify in one step that they (a) got the thing they were 
> asking for (from the interior manifest) and that someone they trust 
> for such things (e.g. their ISP) tells them a good place to fetch it 
> from.
>
Good.

> Simplifying: I would be OK taking out all the namespace stuff, but 
> allowing extensions at the Node level (i.e. top level of each manifest 
> inside the security context) so a publisher could put in a NameCreator 
> element (or any other valid extension) if it so chooses.
I’m not convinced this is a good tradeoff - I’ll be posting another 
message explaining (indirectly) why I think Name Constructors should be 
part of the base.

> Otherwise, the process of
> forming an Interest is implicit in the application or someone just has 
> to guess.  I would then suggest we immediately publish two extensions, 
> one for hash-based naming (ccnx) and one for segmented naming (ndn).
>
Generally, if you think something will wind up used a significant 
fraction of the time, it is much friendlier to the developers to have it 
in one document rather than sprinkled in multiple, especially if you 
think they ought to progress though review in lock-step. I think that is 
likely the case here, as we are not somehow doing extensions after the 
fact that we didn’t think of when we finalized the first version of 
this.

> To go along with the simplification, maybe we could define a range of 
> Ts that define hash groups.  There's nothing technically different 
> about the hashgroups, just having say 16 values that are all HashGroup 
> that allows the app to assign its own app meanings.
It seems good future-proofing to allow app extensions via “app-private 
hashgroups”, but se my later observation below. I would hope that we 
can cover the common cases directly using ICNRG specified things, just 
as a number of name component types are standardized in RFC8569 since we 
“know” they are useful generally.

> This would allow the NameCreator
> extension to specify different rules for different HashGroup types, or 
> a private producer could have their own secret rules for different 
> HashGroup types.  This allows an app to use different NameCreator 
> approaches for the manifest file and the data file.   It's like being 
> able to use typed name components so an app can have its own rules.

There is a certain danger in this if we expect to have manifests widely 
used by “third parties” other than the producer and consumer of a 
particular app. I also would like to stress the comment I made earlier 
that permitting large random amounts of application “metadata” to 
find their way into Manifests can be deleterious for both security and 
rational namespace design. I am fairly wedded to the notion that the 
only thing that belongs in Manifests are things that allow elements that 
fetch, store, or cache content to correctly fetch large objects (and in 
my view also be able to iterate over collections), and to do so without 
any private knowledge of application semantics.

> Marc
DaveO.

> On 8/5/20, 7:33 PM, "icnrg on behalf of 
> [email-masked]" <[email-masked] on behalf of 
> [email-masked]> wrote:
>
>     Hi Dave and Marc,
>
>              vblob_content( 
> vblob_content(dir_hash_val).access(file_name) )
>

DaveO

---

From: Marc Mosko <[email-masked]>
Date: Sun, 9 Aug 2020 21:08:02 +0000

On 8/7/20, 6:56 AM, "icnrg on behalf of David R. Oran" <[email-masked] on behalf of [email-masked]> wrote:

    > referring to non-leaf manifests pieces)?

Yes, the manifest has an implicit traversal order to read the Data pointers in proper order.

    > with the order of the pointed-to data bytes, right?

Any manifest may have both manifest pointers and data pointers, not just leaf manifests.  As it is now, there is only one class of pointers, and they may point to a Manifest or Data.  There is only one traversal order, and this order ensures that both the manifest tree and data pointers are visited in the proper order such that the data pointers are read in order.

The only way a leaf manifest is different than an interior manifest is that it only has data pointers.  But there is no easy way for a consumer to know it is a leaf manifest (maybe it could guess based on subtree sizes).

    > simplicity) in separating them?

The only reason to separate them is to have different Name Creators for the manifest subtree versus the data pointers.

The options are things like:

Mixed hashgroup: [MDMDDDMMDDMMDDD]
Separate: [MMMMMM] [DDDDDDDDD]
Interleaved: [DDD] [MMMM] [DDDDDD] [MMM]

Note that the M pointers are not the same between these options, as the traversal order is always pre-order DFS.  This means if you want to read data at the current node first, you want to put Data pointers before Manifest pointers.

    > any private knowledge of application semantics.

I agree, I'd prefer to not have secret application data determine how to decode a manifest.  Annotations could give hints to optimized ways of reading things (e.g. a video player wants to skip ahead), but it should never change the canonical "if you traverse it like normal you get the original file" paradigm.  I was throwing out a few options to evaluate, not all of them are good.

Marc