Subject: Re: [p2p-hackers] Re: draft-thiemann-hash-urn-01.txt

From: Benja Fallenstein <[email-masked]>
Date: Fri, 05 Sep 2003 12:38:10 +0300

Hi,

Bryan Ford wrote:
> Secondly, your draft as written seems to imply that a hash-identifier URN can 
> contain _nothing_ but the media type, hash scheme, and encoded hash value.  
> This implication neglects important potential (and likely) applications of 
> this scheme in which the hash ID is used as a _starting_point_ from which to 
> find something else via a more conventional naming strategy.  For example, 
...
> 	urn:hash::sha1:LBPI666ED2QSWVD3VSO5BG5R54TE22QL/foo/bar/blah
...
> Of course, the meaning of whatever comes after the hash-value, if any, can 
> only in general be determined with respect to whatever object the hash-value 
> specifies, so your specification cannot and should not really specify 
> anything about the precise format of this "remainder" part of the URN.

So what *would* specify it? The point of having a central registry for 
URI schemes, URN namespaces and so on is that you can go to that 
registry to find out which specifications apply to a given URI. Who 
specifies what format the remainder has and how it is to be interpreted?

-b

---

From: Bryan Ford <[email-masked]>
Date: Fri, 5 Sep 2003 11:12:24 -0400

Peter Thiemann wrote:
> What you are suggesting makes sense to me. However, formally
> your proposed shorthands are no longer URNs, but general URIs.
> I suppose you'd have to write an RFC to make them official and a
> logical approach would be to first get the URN namespace and then
> submit an RFC that defines these shorthand URIs in terms of the URN
> namespace.

Yes, after reviewing RFC2141 I can see what you mean - I agree with your 
approach.  (Although I think it's a bit unfortunate that RFC2141 formally 
treats the URN NID-space as separate and independent of the URI scheme 
namespace and formally requires all URNs to have a URI scheme of "urn:" - 
because in practice as soon as URNs become commonplace nobody's ever going to 
bother typing the "urn:" part anymore, and we'll just have to treat URN NIDs 
as a subset of the URI schemes anyway, all existing in the same "de facto" 
namespace.)

> People are sceptical about adding application specific parts to URN
> identifiers. In fact, an earlier revision of the proposal (see
> http://www.cbuid.org/) allowed such extensions based on the content
> type of the resource and it defined one such extension for
> message/rfc822. But there were other issues that made the proposal
> baroque.
>
> For your example, this approach would require to define content types
> like application/file-system-root and application/database then define
> a syntax for the suffix.

Sure - is there anything wrong with that?  If we're talking about SFSRO file 
systems, for example, since SFSRO isn't a formally standardized format, I 
might just invent the content-type "application/x-sfsro-root" and then 
informally define the interpretation of the "remainder" part of a URN of type 
"application/x-sfsro-root" to be a path name resolved from the specified 
SFSRO root.  If SFSRO or something like it ever _were_ formally standardized, 
then we might indeed end up with a content-type of 
"application/file-system-root" or something like that, in which case we would 
also have to (formally) standardize the interpretation of the "remainder" 
part of a URN for that content-type.

To be clear, I'm not proposing that your hash-URN specification require ALL 
content-type standards also to specify an interpretation for the remainder 
portion of a hash-URN of that content-type.  That would be impossible since 
there are so many content-types defined already that are oblivious to URNs.  
All you need to do is specify that IF a hash-URN contains a "remainder" 
portion after the hash-value, then it is interpreted in a fashion specific to 
the indicated object's content-type.  If you want to be really formally 
picky, you could even specify explicitly that a hash-URN for a given object 
MUST NOT contain a "remainder" string UNLESS the standard defining the 
object's content-type specifies an interpretation for such a string - but 
personally I don't think it's necessary to go this far.

> It seems rather heavyweight to me

What is "heavyweight" about simply allowing hash-URNs to contain 
application-specific supplementary information?  It seems to me the ultimate 
"lightweight" mechanism - you don't have to specify anything except the fact 
that such supplementary information may be syntactically appended to the end 
of the string.

> and I'm not sure if it's worthwhile
> for the specific examples given. In both variants of the SFS file
> system, any resource is reachable from an SFS enabled computer using a
> URL like
>
> file:///sfs/...
>
> Or am I missing something?

You're missing the fact that SFS and similar systems may have valid and 
legitimate reasons for using URN-format naming of this kind.  The whole point 
of defining a single standardized URI/URL/URN namespace scheme is 
_convergence of namespaces_ - the goal of making resource names as uniform 
and interchangeable as possible.  If you deny a whole class of applications 
entrance to the hash-URN scheme because of this trivial and unnecessary 
limitation, you're reducing the potential convergence of resource namespaces 
the scheme can generate.

To make my argument more concrete, consider your proposal simply to use the 
existing "file://" URI syntax to express an SFS pathname, such as:

	file:///sfs/@sfs.fs.net,uzwadtctbjb3dg596waiyru8cx5kb4an/foo/bar/blah

Now the "file:" URI scheme is explicitly defined by RFC 1630 to be a scheme 
for indicating _local_ file system access, so any agent that tries to 
interpret the above URI must assume that it won't necessarily be valid from 
one system to the next.  Indeed, there is no global standard saying that the 
SFS file system has to be mounted as "/sfs" on any given machine; on _my_ 
machine the appropriate pathname might instead be:

	file:///my-sfs/@sfs.fs.net,uzwadtctbjb3dg596waiyru8cx5kb4an/foo/bar/blah

But SFS by design represents a truly global file system namespace, in which 
location-independent, self-certifying hash-identities play a fundamental role 
-  so why shouldn't SFS be allowed to take part in the convergence of the URN 
hash-namespaces?  It can with your current specification, but only in a 
trivial and limited way.  With your hash-URN specification as it stands, I 
could write a location-independent URN to name the root directory of a 
particular SFS server (named by the hash of its public key), like so:

	urn:application/x-sfs:sha1:uzwadtctbjb3dg596waiyru8cx5kb4an

...but I can't write a URN that names anything else on that server!  Why not, 
by the trivial extension I'm proposing, allow hash-URNs that can name not 
only the hashed object itself, but objects discovered and named _relative to_ 
the hashed object?  Like so:

	urn:application/x-sfs:sha1:uzwadtctbjb3dg596waiyru8cx5kb4an/foo/bar/blah

To take this argument further, consider that SFS file systems can contain 
symbolic links, and those symbolic links can currently refer to other objects 
on the same or a different SFS file system, like so:

	foo -> blah	(relative symbolic link to same SFS file system)
	bar -> /sfs/@metoo.net,uzwadtctbjb3dg596waiyru8cx5kb4an/blah
		(absolute link to object on another SFS file system)

...but SFS symbolic links currently can't refer to objects on anything _but_ 
the local file system or another SFS file system.  Suppose we extended SFS to 
allow symlinks to point to arbitrary URIs rather than just conventional 
pathnames, and added a "plug-in" mechanism of some kind by which other URI 
schemes could be interpreted and resolved by the SFS daemon.  Then I could 
place symlinks such as the following into an SFS file system, and access them 
as if they were ordinary files or directories:

	redhat -> ftp://ftp.redhat.com/pub/redhat
	draft-thiemann-hash-urn-01.txt -> urn::md5:d07a37a1e199acb410b12fb07ffb279b
	bar -> urn:application/x-sfs:sha1:uzwadtctbjb3dg596waiyru8cx5kb4an/foo/bar
	msgpart -> urn:message/rfc822:md5:da1dbd8b93153e8adfaf4bb0220f0293/part1

(...the last example being a reference to a MIME-encapsulated portion of a 
multipart E-mail message, named by the complete message's hash and the MIME 
Content-ID of the desired part.)

I realize that such an extension to solve would create additional practical 
issues to address specifically in the context of SFS and Unix file systems 
(e.g., what you see when you "cd" to the "redhat" link above and then type 
"pwd"), but such issues are orthogonal to this discussion.  The point is that 
the hash-URN scheme you are proposing should maximize opportunities for 
namespace convergence, and if you limit the specification so that only the 
hashed object _itself_ can be named, and not other objects relative to the 
hashed object, then you're severely limiting the usefulness of this URN 
scheme for no good reason.  No application or content-type is forced to take 
advantage of this flexibility, but the flexibility should be there.

Thanks,
Bryan

---

From: Bryan Ford <[email-masked]>
Date: Thu, 4 Sep 2003 21:43:33 -0400

Hi Peter, I have a couple comments.

First, while I have no problems with the "full" syntax your draft specifies 
for hash-based URNs, pragmatically I believe that, whether any standard 
"officially" allows it or not, people are going to use shorthands like:

	md5:5307d294b6ccd9854f2deed8c1628b72

to mean:

	urn:hash::md5:5307d294b6ccd9854f2deed8c1628b72

and:

	sha1:LBPI666ED2QSWVD3VSO5BG5R54TE22QL

to mean:

	urn:hash::sha1:LBPI666ED2QSWVD3VSO5BG5R54TE22QL

Personally I think it would be best if the specification anticipates and 
embraces this inevitable shorthand usage rather than either ignoring or 
attempting to  forbid it.  In effect, I think this specification should 
specify a new URN scheme called "hash:", with the syntax defined already, AND 
define the five "shorthand" URN scheme names "md5:", "sha1:", "sha256:", 
"sha384:", and "sha512:".  Especially if the standard doesn't allow new hash 
methods to be added willy-nilly (a philosophy I agree with), there won't be 
many, so this shorthand usage isn't going to create any problem with 
pollution of the URN scheme namespace.  The shorthand schemes don't support a 
<media-type> specification, of course, but that's OK since the <media-type> 
is optional in the longhand syntax anyway.

Secondly, your draft as written seems to imply that a hash-identifier URN can 
contain _nothing_ but the media type, hash scheme, and encoded hash value.  
This implication neglects important potential (and likely) applications of 
this scheme in which the hash ID is used as a _starting_point_ from which to 
find something else via a more conventional naming strategy.  For example, 
the hash-value in a particular URN might be the hash of the root node of a 
directory metadata tree representing a complete read-only (e.g., SFSRO[1]) 
file system, or the public key of a read-write (e.g., SFS[2]) file system, 
and the user might want to use the "rest" of the URN after the hash-value to 
name a particular file in that file system, like so:

or search a hash-identified database using some kind of "query" syntax:

	urn:hash::sha1:LBPI666ED2QSWVD3VSO5BG5R54TE22QL?find=this

Of course, the meaning of whatever comes after the hash-value, if any, can 
only in general be determined with respect to whatever object the hash-value 
specifies, so your specification cannot and should not really specify 
anything about the precise format of this "remainder" part of the URN.  A 
reasonable restriction, however, would be that the "remainder" part (if 
present) start with a non-alphanumeric character, to avoid any confusion 
about where the hash-value ends.  (All characters in the remainder portion 
must also be legal URN-characters of course.)

Thanks,
Bryan

[1] http://www.pdos.lcs.mit.edu/papers/sfsro.html
[2] http://www.fs.net/sfswww/

---

From: Peter Thiemann <[email-masked]>
Date: 08 Sep 2003 17:05:06 +0200

>>>>> "bf" == Bryan Ford <[email-masked]> writes:

    >> like application/file-system-root and application/database then define

    bf> Sure - is there anything wrong with that?  If we're talking about SFSRO file 
    bf> systems, for example, since SFSRO isn't a formally standardized format, I 
    bf> might just invent the content-type "application/x-sfsro-root" and then 
    bf> informally define the interpretation of the "remainder" part of a URN of type 
    bf> "application/x-sfsro-root" to be a path name resolved from the specified 
    bf> SFSRO root.  If SFSRO or something like it ever _were_ formally standardized, 
    bf> then we might indeed end up with a content-type of 
    bf> "application/file-system-root" or something like that, in which case we would 
    bf> also have to (formally) standardize the interpretation of the "remainder" 
    bf> part of a URN for that content-type.

I got a chance to read "Fast and Secure Distributed Read-Only File
System" more closely. Interestingly, the authors discuss the issue why
they chose to implement that functionality as a file system. On page
2, they say 

  We chose to build a file system because of the ease with which one
  can refer to the file namespace in almost any context---from shell
  scripts to C code to a Web browsers location field.

And later in the text they say that to access /sfs through the web,
all you need is a web server (or a proxy) with sfs software
installed. This is more or less a restatement of my argument for using
the file: URL.

Another implication to consider is the burden that each media-type
specific processing places on the implementor of the URN
resolver. Remember that a conforming implementation would have to
implement all of this special processing for every registered media-type. 

The biggest problem, however, is that the addition of any selection
mechanism to the URN would defeat the purpose of a self-verifying,
content-based addressing scheme because a client that does not receive
the entire resource can no longer verify it against its hash.

My conclusion from this is not to allow further parts in the
identifiers. An application should first request the entire resource,
(be able to) verify it locally against its hash, and then perform
further processing.

    bf> You're missing the fact that SFS and similar systems may have valid and 
    bf> legitimate reasons for using URN-format naming of this kind.  The whole point 
    bf> of defining a single standardized URI/URL/URN namespace scheme is 
    bf> _convergence of namespaces_ - the goal of making resource names as uniform 
    bf> and interchangeable as possible.  If you deny a whole class of applications 
    bf> entrance to the hash-URN scheme because of this trivial and unnecessary 
    bf> limitation, you're reducing the potential convergence of resource namespaces 
    bf> the scheme can generate.

If you have a way of stating these extensions without loosing the
self-verifying property, then I'll be with you. The problem is that
you need someone in the chain that you trust, who sees the entire
resource, and can verify the hash value. The whole point of having
hash-based identifiers is that you only need to trust yourself.

-Peter

---

From: Peter Thiemann <[email-masked]>
Date: 05 Sep 2003 14:02:07 +0200

>>>>> "bf" == Bryan Ford <[email-masked]> writes:

    bf> Hi Peter, I have a couple comments.
    bf> First, while I have no problems with the "full" syntax your draft specifies 
    bf> for hash-based URNs, pragmatically I believe that, whether any standard 
    bf> "officially" allows it or not, people are going to use shorthands like:

    bf> 	md5:5307d294b6ccd9854f2deed8c1628b72

    bf> to mean:

    bf> 	urn:hash::md5:5307d294b6ccd9854f2deed8c1628b72

    bf> and:

    bf> 	sha1:LBPI666ED2QSWVD3VSO5BG5R54TE22QL

    bf> to mean:

    bf> 	urn:hash::sha1:LBPI666ED2QSWVD3VSO5BG5R54TE22QL

    bf> Personally I think it would be best if the specification anticipates and 
    bf> embraces this inevitable shorthand usage rather than either ignoring or 
    bf> attempting to  forbid it.  In effect, I think this specification should 
    bf> specify a new URN scheme called "hash:", with the syntax defined already, AND 
    bf> define the five "shorthand" URN scheme names "md5:", "sha1:", "sha256:", 
    bf> "sha384:", and "sha512:".  Especially if the standard doesn't allow new hash 
    bf> methods to be added willy-nilly (a philosophy I agree with), there won't be 
    bf> many, so this shorthand usage isn't going to create any problem with 
    bf> pollution of the URN scheme namespace.  The shorthand schemes don't support a 
    bf> <media-type> specification, of course, but that's OK since the <media-type> 
    bf> is optional in the longhand syntax anyway.

    bf> Secondly, your draft as written seems to imply that a hash-identifier URN can 
    bf> contain _nothing_ but the media type, hash scheme, and encoded hash value.  
    bf> This implication neglects important potential (and likely) applications of 
    bf> this scheme in which the hash ID is used as a _starting_point_ from which to 
    bf> find something else via a more conventional naming strategy.  For example, 
    bf> the hash-value in a particular URN might be the hash of the root node of a 
    bf> directory metadata tree representing a complete read-only (e.g., SFSRO[1]) 
    bf> file system, or the public key of a read-write (e.g., SFS[2]) file system, 
    bf> and the user might want to use the "rest" of the URN after the hash-value to 
    bf> name a particular file in that file system, like so:

    bf> 	urn:hash::sha1:LBPI666ED2QSWVD3VSO5BG5R54TE22QL/foo/bar/blah

    bf> or search a hash-identified database using some kind of "query" syntax:

    bf> 	urn:hash::sha1:LBPI666ED2QSWVD3VSO5BG5R54TE22QL?find=this

    bf> Of course, the meaning of whatever comes after the hash-value, if any, can 
    bf> only in general be determined with respect to whatever object the hash-value 
    bf> specifies, so your specification cannot and should not really specify 
    bf> anything about the precise format of this "remainder" part of the URN.  A 
    bf> reasonable restriction, however, would be that the "remainder" part (if 
    bf> present) start with a non-alphanumeric character, to avoid any confusion 
    bf> about where the hash-value ends.  (All characters in the remainder portion 
    bf> must also be legal URN-characters of course.)

Is there more support for providing such an extension? 

It seems rather heavyweight to me and I'm not sure if it's worthwhile
for the specific examples given. In both variants of the SFS file
system, any resource is reachable from an SFS enabled computer using a
URL like

file:///sfs/...

-Peter

    bf> Thanks,
    bf> Bryan

    bf> [1] http://www.pdos.lcs.mit.edu/papers/sfsro.html
    bf> [2] http://www.fs.net/sfswww/

---

From: Bryan Ford <[email-masked]>
Date: Tue, 9 Sep 2003 23:17:42 -0400

> they chose to implement that functionality as a file system. On page

I haven't read it in a while, myself. :)

> installed. This is more or less a restatement of my argument for using

Sure, there's an excellent pragmatic argument to making any new 
file-system-like storage/data access system mesh well with existing (local) 
file system namespaces.  But doing so doesn't change the fact that "file://" 
URLs are by definition _local_ namespaces - just because a URL has a 
"file:///sfs/" prefix doesn't even necessarily mean that it's on an SFS file 
system, and it certainly doesn't have the location- and access 
method-transparency properties we want from URNs.  For example, if a public 
web page contains "file:///sfs/..." links to files on an SFS file system, any 
decent web browser will refuse to let you click through to them, because the 
browser assumes such links are to private files on your local file system and 
following such links would be a massive security risk.  There are good 
pragmatic reasons to link SFS and other file systems into local namespaces 
this way, but semantically that's not where they belong, and short-term 
pragmatics shouldn't prevent us from looking for better solutions down the 
road.

> implement all of this special processing for every registered media-type.

No, the URN resolver doesn't have to do anything new at all, because there's 
no reason it needs to be or should be a URN resolver's responsibility to 
interpret the (proposed) relative part of a hash-URN.  This task should 
naturally be the application's responsibility, which is no additional burden 
since the application must understand the media-type anyway.  Here's an 
example of how I envision it might work:

1. The application wants to resolve a URN of the form 
"urn:hash::md5:123...abc:/foo/bar.html".  So the application passes the whole 
URN to a URN resolver that understands the "hash" NID.

2. The "hash" URN resolver understands the "md5:123...abc" part of the URN and 
uses that to locate the flat blob of bytes that the hash refers to, whatever 
it might be.  The URN resolver doesn't have to have any idea what this blob 
of bytes is; it just needs to find it.  The URN resolver also doesn't know 
what the ":/foo/bar.html" on the end of the URN means.  When the URN resolver 
finds a URL for the blob of bytes indicated by the hash, the resolver passes 
that URL (say "http://me.com/root") back to the application along with the 
uninterpreted relative portion of the URN, "/foo/bar.html", which the 
application can use as it sees fit.

3. The application now has one or more URLs for the hashed blob of bytes, and 
the remaining relative portion of the URN.  If there is a relative part and 
the application doesn't know what to do with it, or if the content-type of 
the blob doesn't define the meaning of a relative part, then the application 
can either ignore the relative part or (probably safer) just fail the search.

4. But say the application examines the data blob located by the URN resolver 
and it happens to be an SFSRO-like file system directory node, containing a 
list of files and subdirectories and the hashes (maybe even in URN form) of 
their corresponding data streams or metadata nodes.  The application knows it 
wants the object named "/foo/bar.html" relative to the root node, so it 
locates "foo" in the directory metadata blob and finds its hash.  The 
application might then use some application-specific naming convention to try 
to find "foo", and subsequently "bar.html", on the same server on which it 
found the root node - e.g., by looking up "http://me.com/root/foo" and 
"http://me.com/root/foo/bar.html" respectively and seeing if the objects are 
accessible and the hashes come out right.  But such an application-specific 
method, if any, is just a hint.

5. If the application-specific "tree-walking" method, if any, fails, then the 
application goes back to the URN resolver and asks it to locate the hash-URN 
for "foo" and locate a copy of the metadata for _that_ directory.  Once the 
application has found out, it can finally resolve the hash-URN for "bar.html" 
located in the metadata for "foo".  In any case, the URN resolver has no clue 
what kind of data these hashes actually represent or what they're being used 
for; it's just helping the application to find what it needs, which is 
exactly what URN resolvers are supposed to do.

> the entire resource can no longer verify it against its hash.

Not at all - I think you misunderstood my proposal.  If you have a hash-URN 
like "urn:hash::md5:123...abc:/foo/bar.html", the "123...abc" hash value is 
NOT the hash of the "bar.html" that you're eventually trying to find; it's 
the hash of the object representing the _starting point_ of the search - 
e.g., the root directory metadata node.  It's the application's 
responsibility to work from there in an appropriate application-specific 
fashion, to interpret the "foo/bar.html" part and find the ultimately-named 
(sub-)object in a way that does not violate the URN principle of permanence 
or access method independence.  If the application is an SFSRO-like file 
system and uses something like the traversal procedure I outlined above, then 
the self-verifying nature of the hash-URN is not violated at all, because:

1. the original hash-URN contains the hash of the root (starting point) 
directory metadata, so the root directory can't change without the hash-URN 
changing.

2. The root directory metadata contains the hash of the "foo" subdirectory 
metadata, so the "foo" subdirectory metadata can't change without its hash in 
the root directory changing.

3. Finally, the "foo" subdirectory metadata contains the hash of the file 
"bar.html", which can't change without the hashes of "/foo" and "/" in the 
higher-level metadata nodes changing.

Thus, the complete URN "urn:hash::md5:123...abc:/foo/bar.html" represents a 
secure, self-verifying, and permanent link to the ultimately-named file 
"bar.html" even though "bar.html" isn't directly the blob of bits that the 
hash in the URN represents.

Similarly, if the URN contains a content-type specification, it's the 
content-type of the _starting point_ (the blob of bits from which the hash 
was derived), not the content-type of the object eventually named.  The 
latter can presumably be determined from whatever metadata the application 
picks up during its traversal.

> (be able to) verify it locally against its hash, and then perform

I agree 100% that "An application should first request the entire resource, 
..., and then perform further processing."  At least if by the "entire 
resource" you specifically mean "the entire byte-stream that the hash was 
computed from."  But I also think it is indispensable for the hash-URN 
standard at least to _allow_ URNs to include additional information that may 
be useful to the application in that "further processing".  I only outlined 
one possible use for this additional information; there could be many others.  
I just don't want the hash-URN specification to preclude them.

> hash-based identifiers is that you only need to trust yourself.

I think I've stated the extensions without losing the self-verifying 
property...  Are you with me? :)

Cheers,
Bryan