Subject: [icnrg] On FLIC and ISO manifest differences

From: <[email-masked]>
Date: Mon, 30 Nov 2015 23:48:23 +0000

Hi folks: 

During Wednesday’s CCNx-TWG meeting we discussed two proposals for the 
“single object” manifest: FLIC and ISO. There were (at least) two elements 
of the ISO design that separate it from FLIC, which we highlight below. 

- Pointers in ISO have size metadata to allow for fine-grained seeking 
within a tree. 

- ISO's relative naming scheme allows data to be moved and de-duped 
without regenerating manifests. 

Short version: FLIC covers both points. 

Long version: 

Re Pointer Size: The blocksize metadata entry of FLIC enables to 
calculate offsets (and therefore skip downloading some data blocks). For 
example, if the block size of a HashGroup is 2048B then the offset for 
each block in that group is a multiple of 2048B. 

It is also true that in case of a FLIC manifest pointer one first has to 
download the manifest to get the number of bytes covered by such a 
subtree. If this is a concern, one can add a separate HashGroup (just 
for this manifest pointer) and decorate it with a Totalsize metadata 
entry that indicates the number of bytes covered by this subtree. 
Example layout: 

(ManifestMsg 
    (HashGroup 
        (MetaData (Blocksize 2048) (Totalsize 4001)) 
        (DataPtr 0xabcd....) 
        (DataPtr 0x2334....) 
    ) 
    (HashGroup 
        (MetaData (Totalsize 64987)) 
        (MfstPtr 0x8772....) 
    ) 
    (HashGroup 
        (MetaData (Blocksize 2048) (Totalsize 2999)) 
        (DataPtr 0xbcde....) 
        (DataPtr 0x3345....) 
    ) 
) 

Note that this trick of adding a separate HashGroup consumes the same 
number of encoded bytes as in ISO. But we save many more bytes in the 
normal HashGroup case because DataPtr directly leads to the 32 digest 
bytes instead of having internal structure. 

Re Relative Naming and De-Duping: De-Dup is part if FLIC's design goal 
and is best achieved with nameless manifests. In order to make them 
fetchable through names, arbitrarily many "named parent manifests" can 
be published, all pointing to the same nameless manifest. 

For example, consider there exists a piece of data that is FLICed (its 
root manifest doesn't need but even can have a name e.g. /big/file). If 
this data is located at PARC West, then its parent manifest will be 
named /parc/west/big/file, containing a hash pointer to the FLIC root 
manifest. 

The first interest will go for /parc/west/big/file and return the parent 
manifest where we learn this first pointer value. All subsequent 
interests will use the ObjHashRestriction and also go for 
/parc/west/big/file which becomes a mere locator. 

If this data needs to be moved to PARC East, then one simply creates a 
new parent manifest with the name /parc/east/other/big/file (also 
pointing to the FLIC root manifest). The old data does not have to be 
re-parsed or regenerated - all what matters is the hash value of the 
root manifest. 

How the original FLIC root manifest is named is of no importance - this 
name is mere decoration and is never used for locating the FLIC's 
chunks. This is akin to the Unix approach wherein inodes (FLIC trees) 
*do not* contain names. 

To summarize, we believe the ISO approach of mixing named and nameless 
objects *in the manifest tree* should not be done for single object 
manifests. This is not to say this approach won’t be useful for the 
“advanced manifest” to come later. 

Best, 
Christian Tschudin and Christopher Wood