Subject: [icnrg] Some comments on the CCNx Selectors draft (https://datatracker.ietf.org/doc/draft-mosko-icnrg-selectors/)

From: "David R. Oran" <[email-masked]>
Date: Sun, 02 Jun 2019 09:39:56 -0400

All comments with <chair hat off>

Content discovery is important, so I’d like to see something adopted 
by the RG for formal progression to experimental.

I’d like to step back a bit though and ask if evolving the approach in 
CCNx 0.x and NDN is appropriate knowing what we’ve learned in the last 
few years. The current selectors draft 
(https://datatracker.ietf.org/doc/draft-mosko-icnrg-selectors/) 
basically fixes the original design mistake of forcing all forwarders to 
do discovery through prefix match on CS data. It does this by leveraging 
typed name components and using an encapsulation technique so that 
whatever object that matches the selectors can be returned as 
“native” - with its original name and signature.

On the other hand, it still has the following properties:

- returns only one object when multiple might match

- does not have a completely deterministic answer in the presence of 
cache/repo/producer instance desynchronization (not clear this is 
avoidable since we have a distributed system with at best eventual 
consistency semantics)

- requires exclusions, which can make for big interests that in turn 
causes fragmentation to be needed and complicates congestion control as 
a consequence.

- seems to still permit cache exploration (since the selector interests 
are not authenticated), which might be viewed as a privacy problem.

I don’t have a concrete proposal to make though. Here are a few 
thoughts about a possible alternative approach:

- Since we depend a lot more on Manifests now that the original NDN & 
CCNx 0.x designs, why not have the discovery return a Manifest instead 
of a single encapsulated object?

- Since we have fast exact match and nameless objects along with now 
permitting objects to have multiple names and hence multiple ways 
objects can be discovered, perhaps we can exploit that to allow 
different discovery patterns rather than smooshing everything into a 
static/brittle set of selectors

- Some of the properties were driven by trying to make discovery cheap 
and ubiquitous and hence forced into casting discovery as a direct 
single Interest/Data exchange. Since we now know that pawing through a 
large CS or Repo can be computational expensive (and even require a fair 
amount of I/O), maybe this tradeoff isn’t attractive. We now have 
experience with distributed method invocation (e.g. NFN and RICE) so 
maybe we do discovery as an explicit remote method. This may solve a few 
problems:

	- We can do authentication and key exchange, so discovery operations 
and results can be more private and cache exploration is no longer a 
hazard

	- It might make it easier to build extensibility into the supported 
discovery patterns as all of them might not need to be supported by all 
discovery services. (Aside - it seems that a similar direction is being 
taken in the fast-repo work for NDN, where a given application can 
design patterns for the Repo to fetch the right stuff for storage of 
that application’s data).

	- We can more easily manage the resources for expensive discovery 
operations, and following the idea of using Manifests, make those the 
result of the discovery computation.
	- By essentially “binding” to one of the discovery 
services/servers, it may be possible to get a more internally-consistent 
set of results.

Yes, this might make discovery a lot more “expensive” in terms of 
RTTs but in the end it might also make applications easier to write with 
patterns that look more like what the application considers “native” 
and the results as a Manifest easier to paw through and possibly iterate 
on.

Thoughts?

DaveO

---

From: [email-masked]
Date: Mon, 3 Jun 2019 20:17:31 +0200 (CEST)

Thanks David for creating a discussion around this!

Looking at discovery from a function invocation perspective certainly 
resonates well, of course. However, NFN focuses on producing immutable 
results out of immutable arguments, and still makes the most sense in 
such an environment. You target the discovery of potentially new 
content, such that the same query will return different results if asked 
again later, right? That leads me to this comment:

Your writeup did not mention time nor provenance, which I think are 
important in this discussion, another concept to mention would be "query 
expression".

With provenance I mean the state (context) where new content was 
published. This state acts as logical time and permits to "immutify" 
results. The exchange would be:

- fetch a repo's state, or time horizon, S
   (this is the only mutable result call needed and permits to bind to
   that state, instead of a server or service)

followed by one or more

- discover(repo=/ndn/ucla, horizon=S, selector=/unix/ls(/some/repo/path, "*.txt"))

This discovery call will return the same result even of the repo is 
replicated: any repo replica can execute the call in the specific 
context S, no need to pin your discovery to a specific server, and the 
result is fully cachable (if in clear).

A simple way to explain what a repo state S is, would be to point to the 
commit hash of a Git repo. A NDN repo would have to keep track of new 
content through commits, have a log of its actions. I think that the 
word "provenance" is appropriate here, namely how/where/when some 
content was added to the repo, or renamed, aliased etc.

Sure, manifests are good, but I think we can do better, and by better I 
mean repo-side filtering. Once results are immutable, we can have 
deterministic and stateless paging, cursors and other result mangling 
under the requestor's control:

The last 10 entries:
- /nfn/txt/tail(/ndn5/discover(horizon=S, selector=/unix/ls(/ccnx/some/prefix, "*.txt"), -10)
Just counting hits:
- /nfn/txt/wc(/ndn5/discover(horizon=S, selector=/unix/ls(/ccnx/some/prefix, "*.txt"))

I see your objection coming that general NFN is too heavy for minimal 
content discover, and I would agree. But what about a modest query 
_language_ for repo-side execution instead of exposing single RPCs that 
the caller has to chain in multiple RTs? We might pick a small set of 
result filters (selectors really), that are easy to implement in all 
repos, and permit to combine up to three filters in a query expression 
as an example. So no loops, no lambda... Filters could be:

   isDir(), newerThan(d), hasMagic(GIF), publishedBy(key), lexicoAfter(marker)

Then we add Python slicing as in [10:-5], or just classic head() and 
tail(), as well as AND, NOT, possibly OR.

Datalog could serve as a high-end starting point for such a discussion.

Best, c

On Sun, 2 Jun 2019, David R. Oran wrote:

> Content discovery is important, so I’d like to see something adopted by the
> RG for formal progression to experimental.
> 
> I’d like to step back a bit though and ask if evolving the approach in CCNx
> 0.x and NDN is appropriate knowing what we’ve learned in the last few years.
> The current selectors draft
> (https://datatracker.ietf.org/doc/draft-mosko-icnrg-selectors/) basically
> fixes the original design mistake of forcing all forwarders to do discovery
> through prefix match on CS data. It does this by leveraging typed name
> components and using an encapsulation technique so that whatever object that
> matches the selectors can be returned as “native” - with its original name
> and signature.
> 
>  * 
>     returns only one object when multiple might match
>
>  * 
>
>     does not have a completely deterministic answer in the presence of
>  * 
>     requires exclusions, which can make for big interests that in turn
>  * 
>     seems to still permit cache exploration (since the selector interests
> I don’t have a concrete proposal to make though. Here are a few thoughts
> about a possible alternative approach:
>
>  * 
>
>     Since we depend a lot more on Manifests now that the original NDN & CCNx
>     0.x designs, why not have the discovery return a Manifest instead of a
>     single encapsulated object?
>
>  * 
>
>     Since we have fast exact match and nameless objects along with now
>  * 
>     Some of the properties were driven by trying to make discovery cheap and
>     ubiquitous and hence forced into casting discovery as a direct single
>     Interest/Data exchange. Since we now know that pawing through a large CS
>     or Repo can be computational expensive (and even require a fair amount
>     of I/O), maybe this tradeoff isn’t attractive. We now have experience
>     with distributed method invocation (e.g. NFN and RICE) so maybe we do
>     discovery as an explicit remote method. This may solve a few problems:
>
>      +  We can do authentication and key exchange, so discovery operations
>      +  It might make it easier to build extensibility into the supported
>         discovery patterns as all of them might not need to be supported by
>         all discovery services. (Aside - it seems that a similar direction
>         is being taken in the fast-repo work for NDN, where a given
>         application can design patterns for the Repo to fetch the right
>         stuff for storage of that application’s data).
>      +  We can more easily manage the resources for expensive discovery
>         operations, and following the idea of using Manifests, make those
>         the result of the discovery computation.
>      +  By essentially “binding” to one of the discovery services/servers,
>         it may be possible to get a more internally-consistent set of
>         results.
> 
> Yes, this might make discovery a lot more “expensive” in terms of RTTs but
> in the end it might also make applications easier to write with patterns
> that look more like what the application considers “native” and the results
> as a Manifest easier to paw through and possibly iterate on.
> 
> Thoughts?
> 
> DaveO
> 
> 
>

---

From: "Mosko, Marc <[email-masked]>" <[email-masked]>
Date: Mon, 3 Jun 2019 19:40:06 +0000

Dave & Christian,

Thanks for starting to tackle this problem.

In regard to the Selectors draft, I do not want to spend too much time defending it.  As said before, it was a proof-of-concept that one can do opt-in discovery using similar mechanisms to CCNx 0.x and older NDN.  Doing 1 by 1 query response and returning the whole content object each time is not, I think, the right approach.

I'm not going to dive in to a language or NFN vs keywords, etc., at this time.  I think we should first describe what it's supposed to do.

I think it is important that each node be able to control the extent to which it participates in discovery, including the encryption of queries and responses.  Possibly also authenticated discovery with various access control.

Here are some examples of the types of questions I think could be issued:
1) What is the latest version of a prefix?
2) What publishers have published under a name? (i.e., are there multiple keyids for the same name?)
3) regex style matching under a prefix.
4) Where can I get everything in a manifest
5) Which pieces of a manifest do you have (branches, leafs, other ways of answering)?
6) POSIX-like 'ls' functions
7) Discovery sources vs discovery content (e.g. where can I get something versus what can I get).  I don't mean to make this host-based networking, but I might want to get something from my cell provider versus a cache elsewhere or from a privacy-based cache operator.

Like DNS, there we have unauthoritative content stores (CS) and repositories; and authoritative repositories and publishers.  There should, I think, be a way to distinguish between the two and maybe to select whom one queries.

I think it is important to be able to break a response into multiple transactions, such as for scale and performance.  And those should be able to be consistent, such as a cursor.  This could be achieved by talking to only one entity using local context or, as Christian pointed out, using some shared state and talking with an enclave.

One could return a Manifest-like thing in response, but I think one wants to include possibly more information:
- A link (name, keyid, hash) to objects (e.g., "files")
- A sub-namespace (e.g., "directories")
- Information on caching (e.g. the ExpiryTime or remaining RCT)
- Information about inter-linking (manifest relationships)

I think using CCNxKE-like setup for private sessions for a query (and possibly subsequent data transfer) is a good thing.  For a properly configured enclave, one CCNxKE setup could allow for enclave-wide queries and responses.

Marc

________________________________________
From: icnrg <[email-masked]> on behalf of [email-masked] <[email-masked]>
Sent: Monday, June 3, 2019 11:17 AM
To: David R. Oran
Cc: Mosko, Marc <[email-masked]>; ICNRG
Subject: Re: [icnrg] Some comments on the CCNx Selectors draft (https://datatracker.ietf.org/doc/draft-mosko-icnrg-selectors/)

followed by one or more

The last 10 entries:

Best, c