Subject: Re: [urn] several different proposals for URN syntax to include fragments and queries

From: John C Klensin <[email-masked]>
Date: Sun, 07 Jul 2013 09:58:59 -0400

Hi.

Trying to pick back up on this after being distracted by some
other issues for the last couple of weeks.  I'm going to reply
to the correspondence between Pete and Barry from two weeks ago
because it is a convenient anchor, but have read all the traffic
until now.  I sent a separate response to Juha's main comments
Friday.

--On Friday, June 21, 2013 11:09 -0400 Keith Moore
<[email-masked]> wrote:

> On 06/21/2013 10:51 AM, Barry Leiba wrote:
>> I'll also note that I see a reasonable case for fragments
>> actually changing what resource the URN winds up resolving
>> to.  Consider this: "urn:isbn:0-679-40758-8" would name the
>> edition of "Don Quixote" that I have on my bookshelf.  It's a
>> big book; it has, in one cover, "Volume 1" (which subdivides
>> into "Book I" through "Book IV") and "Volume 2" (which has
>> "Book V" and "Book VI").  If fragments were allowed, it might
>> be reasonable for, say,
>> "urn:isbn:0-679-40758-8#volume.1" to get me to
>> "http://example.com/don-quixote/vol1", and for
>> "urn:isbn:0-679-40758-8:volume.2" to get me to

Barry, I see where you are headed here, but I think it leads to
madness, at least as I understand URNs.  From that perspective,
there are two rational choices:  One is to treat the entire
two-volume "book" (with an ISBN assigned independent of the
"volume" boundary).  For that case, Juha's notion of "the URN
identifies the 'book' and any fussing with fragment IDs occurs
after retrieval  time" might work well if some performance
issues are ignored.   There are some subtle issues with that
which I'll let Juha, or, if she has joined the list by now,
Stella  Griffiths, comment on, but one of them is whether it is
appropriate to assign one ISBN to a two-volume set and, if it
is, if it is appropriate to then treat that set as anything
other than atomic (i.e., an artifact of binding as far as the
ISBN is concerned, perhaps such that an otherwise-identical
printing on really thin paper and bound as one volume would get
the same ISBN.  But, if the ISBN really refers to an atomic
"book" unit, then "urn:isbn:0-679-40758-8#book.iii" would be
perfectly reasonable, but "urn:isbn:0-679-40758-8#volume.1" is
nonsense -- not with regard to your mental picture or what you
see on your shelf, but with regard to the intrinsic properties
if ISBN assignments and hence with the ISBN namespace.

It is that sort of issue which caused me to say "namespace
dependent".   If one has any delusions about fragments being
part of the resolution process on the server, then it seems to
me that standards for what is valid and what isn't have to be
set as part of the definition of the namespace and that, if the
namespace is itself dependent on what I think Juha would call a
formally managed process (language supported by RFC 3406), by
the agency that defines and interprets that process.  The other
possibility is to treat fragments as not part of the URN
dereferencing (see below) process but as part of a discussion
between the user who specifies a URN to a user interface and
that interface, not the URN-resolving server.

What makes this more interesting is that, if I correctly
understand the ISBN rules, it would be possible for a publisher
of an electronic version to assign separate ISBNs to the various
books of Cervantes's original organization.  That would make it
quite plausible to refer to "Don-Quixote-book3" by 
   "URN:ISBN:0-679-NNNNN-N"
and to "Don-Quixote-book4" by
   "URN:ISBN"0-679-MMMMM-M"
but that obviously has nothing to do with fragments.  If one
wants to treat all printings or editions as the same, then URNs
and fragments aren't the problem.  The problem is that ISBNs are
just not the right tool.
 
>> The critical part here is that what we define has to be
>> useful to the folks who will actually *use* the URNs in the
>> real world, and I think that's where John's proposal takes
>> us.  If they don't need fragments as part of the URNs (or
>> queries, which I find much more problematic as anything other
>> than something that's passed to the resource), then we're
>> fine in saying that they're not part of the URNs.  But if they
>> *do* (because they need to do the sorts of things above, or
>> perhaps because they need to do something entirely different
>> and I got it all wrong), then we have to allow them to do
>> what they actually have to do -- otherwise, we'll have simply
>> defined some architecturally pure thing that's not actually
>> useful.

Indeed, that is consistent with the point I was trying to make.

> I've had a couple of ideas about this:
> 
> 1. One is that URN resolution of a resource could return a
> list of fragment mappings.  So if you resolved
> urn:isbn:0-679-40758-8 you might get back something that
> included:
> ...
> "fragments" = { ( "volume.1" =
> "http://example.com/don-quixote/vol1" ),
>                             ( "volume.2" =
> "http://example.com/don-quixote/vol2" ),
>                             ( "book.i" =
> "http://example.com/don-quixote/vol1#bk1" ),
>                             ( "book.ii" =
> "http://example.com/don-quixote/vol1#bk2" ),
>                             ( "book.iii" =
> "http://example.com/don-quixote/vol1#bk3" ),
>                             ( "book.iv" =
> "http://example.com/don-quixote/vol1#bk4" ),
>                             ( "book.v" =
> "http://example.com/don-quixote/vol2#bk5" ),
>                             ( "book.vi" =
> "http://example.com/don-quixote/vol2#bk6" ) }
> ...

Arggh.  I think this ignores what ISBNs actually are.  It is the
collision between our engineering speculations and theories and
the reality of well-established identifier systems that
represent consensus and extensive practices with their own rules
that gets us into trouble here.  I suppose one could establish a
reference-to-book-that-happens-to-be-identified-by-an-ISBN URI
(I have no opinion as to whether that would be a URN or not)
that would have the above properties.   However, whatever sort
of critter that was, it wouldn't be an ISBN URN because the ISBN
is perfectly clear about what is being identified and is isn't a
collection or URLs that locate fragments.
 
> (Note that there's no real reason that, for an electronic
> copy, it would actually be needed to split this work into the
> equivalent of two volumes.   Though maybe you could have such
> a case for a work that required several terabytes of storage.)

My supposition from having read the ISBN standard (but certainly
not being an expert on it) is that, if one ISBN has been
assigned, the "two volume" property is an artifact.  And see
above about ISBN-per-Book.

> 2. Another idea is that you might need separate mechanisms to
> permit identifiers to refer to HTML-style fragments, than
> those used to refer to persistent fragments.   For consistency
> with other URIs that refer to HTML-style fragments I'd
> recommend continuing to use the '#' notation for these.   But
> using the syntax I proposed, you could define a /region=xxx
> option (call it something different than fragment to minimize
> the potential for confusion).   And I'm thinking that all such
> options should be input to any resolution service (so the
> client wouldn't have to know which options applied at which
> layer), so the resolution service would see that the client
> wanted region "book.iv" and could return the right URL (if not
> the whole list of regions).

This gives me a nasty headache.  But, at least the way I read
3896, there is nothing that requires the semantics of fragments
to be constant across URI types (or even across resources).

> In summary, I do see the potential that some fragment-like
> construct that were resolved as part of the URN could be
> useful.   I just don't think it's a good idea to prevent users
> from addressing individual HTML-style fragments of a document
> in conjunction with URNs.   And I think it would be confusing
> if we repurposed '#' to mean something subtly different for
> URNs than for other URIs.

See above.  At present, fragments are subtly different for
different resources even within the HTTP URL environment.
Requiring them to somehow be consistent between URLs and URNs
when they aren't consistent within URLs strikes me as hard.

Picking up text from another note:

--On Friday, June 21, 2013 19:13 -0400 Keith Moore
<[email-masked]> wrote:

>...
> Also, URNs were never intended to be just another URI scheme.
> They were always intended to be stable, location-independent
> names that resolved to URLs.

I just don't think that is true.  We've talked about URNs that
can't be resolved at all, much less to URLs.  That is quite
explicitly mentioned, with "resolution" even in quotes, in RFC
3406. 

That document, incidentally, contains the statement "not all
strings that conform to URN syntax are necessarily valid URNs".
Exactly that same language appears in RFC 2611, so, to the
extent that is a problem, it is hard to blame it entirely on RFC
3986.

Suggestion:  My guess is that this conversation would be
considerably clarified if we stopped talking about "resolving"
in favor of "dereferencing" and avoided trying to distinguish
"what is part of a URN" or not in favor of distinctions about
what what happens on the relevant server and what happens
post-retrieval or post-dereferencing (not the same thing either,
obviously).

best,
   john

---

From: Juha Hakala <[email-masked]>
Date: Mon, 08 Jul 2013 11:23:02 +0300

Hello,

On 7.7.2013 16:58, John C Klensin wrote:
>>> to.
This would extend the scope of the URI/URN fragment well beyond what is 
desirable, and also make it a lot harder to manage the identifier 
assignment process.
>>> Consider this: "urn:isbn:0-679-40758-8" would name the
> madness, at least as I understand URNs.
In the real world, printed versions of Don Quijote will get different 
ISBN than the digital versions. And the (separately published) volumes 
of the digital Don Quijote will each get their own ISBNs. Iff there is a 
one volume digital version (with a single ISBN) which is encoded so that 
the beginning of each part can be indicated with a fragment, then 
fragment usage will be OK - but they will not identify the parts, they 
just indicate the starting position of each part within the 
all-encompassing volume.

Examples such as the one Barry has provided above illustrate why 
identifier assignment has to be a managed process. There is a reason why 
the ISBN community has not only the standard but also an extensive guide 
book which tells how ISBNs should be used. And perhaps all this 
complexity also explains why some publishers and ISBN centers 
occasionally get it wrong ;-).

One reason why we need rules and principles for usage of fragment is to 
prevent people from doing things which are technically possible, but 
would undermine the appropriate principles for naming documents within 
that namespace. Of course there will be namespaces where anything goes. 
But if we were to say that fragment (or query) is part of the NSS, it 
would be necessary to revise the ISBN standard and all the other 
existing standard identifiers so that the usage of URI fragments is 
taken into account. I doubt anybody wants to go into that, for various 
reasons (most likely many communities would just say no). The easiest 
solution is to keep both fragments and queries separate from the NSS, 
which a priori means that a URN and the same URN with fragment always 
identify the same resource.

> It is that sort of issue which caused me to say "namespace dependent". 
> If one has any delusions about fragments being part of the resolution 
> process on the server, then it seems to me that standards for what is 
> valid and what isn't have to be set as part of the definition of the 
> namespace and that, if the namespace is itself dependent on what I 
> think Juha would call a formally managed process (language supported 
> by RFC 3406), by the agency that defines and interprets that process. 

Many documents have component parts which must be identified separately. 
For instance, there are periodicals (identified with ISSN) plus 
periodical issues, articles in those issues, and images within those 
articles, all of which can be identified by e.g. national bibliography 
numbers. In order to facilitate long term preservation, typical approach 
is to give identifiers to everything and not use fragments within e.g. 
the serial issue to indicate its component parts. This means that URN 
resolution is used to access "logical fragments" such as journal 
articles, but URN fragments have nothing to do with the process.

> The other possibility is to treat fragments as not part of the URN 
> dereferencing (see below) process but as part of a discussion between 
> the user who specifies a URN to a user interface and that interface, 
> not the URN-resolving server. What makes this more interesting is 
> that, if I correctly understand the ISBN rules, it would be possible 
> for a publisher of an electronic version to assign separate ISBNs to 
> the various books of Cervantes's original organization. 

Correct. If the book is published in four volumes, there should be one 
ISBN for the whole book, and each volume should also get its own ISBN. 
In practice, ISBN assignment is largely based on the needs of the trade: 
one can buy either the entire book (all volumes) or just one volume; 
therefore there is a need to separate them.

> That would make it quite plausible to refer to "Don-Quixote-book3" by 
> "URN:ISBN:0-679-NNNNN-N" and to "Don-Quixote-book4" by 
> "URN:ISBN"0-679-MMMMM-M" but that obviously has nothing to do with 
> fragments. If one wants to treat all printings or editions as the 
> same, then URNs and fragments aren't the problem. The problem is that 
> ISBNs are just not the right tool. 
Yes - if somebody wants to find for instance all the editions of Don 
Quixote in English, then ISTC is the identifier to use.
> that gets us into trouble here.
+ 1

(This is just an example.) According to the ISBN assignment rules, there 
should be an ISBN for e.g. an English translation of Don Quixote 
published by Victor Gollancz in EPUB 3 (all six volumes), and in 
addition each volume should have its own ISBN. If the service the user 
wants is URN to URLs, then the URN:ISBN of the whole thing should 
resolve to a list of six URLs. On the other hand, URN:ISBN of a single 
volume will resolve to single URL (provided that there is only one 
location from which the book can be retrieved; often there are many).

Within the technical limitations of the EPUB version, anyone should be 
able to cite / refer to the chapters etc. within the volumes using URI 
fragments. But these fragments do not identify chapters.

In some circumstances it might be useful to have a URN resolution 
service (available via a query parameter) which harvests the structure 
of the document (chapters, parts, names of articles within a book which 
consists of articles). But since the encoding of the documents can be 
non-informative, it is a better idea to check descriptive metadata to 
find out about the resource. Table of contents is probably much more 
readable than any fragment listing.
> to be constant across URI types (or even across resources).
My preference is to get started with HTML-style fragments using the 
#-notation, and to provide basic guidelines for development of namespace 
specific solutions. This is what Alfred Hoenes and myself were already 
doing in the previous versions of URNbis I-Ds.

There are a lot of projects etc. out there which are hoping to identify 
component parts of resources. These component parts may be for instance 
concepts in a corpus database, or variables within research data sets. 
The 1000 USD question is whether it is possible / advisable to use 
generic URN syntax tools for this. I do not think so.

Given the diversity of the URN namespaces, it is not likely that we can 
find a generic syntax that would work for everyone. But this is not even 
necessary. What we need to say is that each namespace may have its own 
practices for identification of component parts of resources, as long as 
they do not conflict with URI syntax and there are resolution services 
which can deal with the resulting URNs.

For identification of component parts of resources ("logical fragments") 
one should never use  URI fragments, but identifiers created on the 
basis of namespace specific practices which must be supplied in the 
namespace registration request. For the ISBN namespace the principle is 
carved in stone in the standard: one can assign ISBN for e.g. a chapter 
of a book (like chapter 1 of Don Quixote, vol. 1) if the chapter is 
available as a separate item. For NBN, each organization assigning 
identifiers has the freedom to decide what to do within its own 
sub-namespace.
>> URNs than for other URIs.
+ 1 to all of the above, except that I think that the best we can have 
is namespace specific fragment-like constructs.

Juha

-- 

  Juha Hakala
  Senior advisor

  The National Library of Finland
  P.O.Box 15 (Unioninkatu 36, room 503)
  FIN-00014 Helsinki University
  tel +358 9 191 44293

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 08 Jul 2013 17:37:08 -0600

Hi Juha, thank you for continuing this discussion. I think we are
getting closer to a shared understanding.

Comments inline, with the intent of finding text we can agree on.

> desirable, and also make it a lot harder to manage the identifier

I tend to agree (and to agree with John that the word "resolution" is
misleading in the context of URNs).

> identify the same resource.

(And also "the same URN with query", I assume?)

Keeping fragments and queries separate from the NSS makes it easier for
us to determine the exact text in 2141bis and 3406bis.

Juha, do I understand correctly that you might suggest that we continue
to define assigned URNs (or, perhaps more precisely, URIs with the urn:
scheme) as "urn" ":" NID ":" NSS and then specify that fragments and
queries might be strings that can be added to an assigned URN? (Perhaps
the term "assigned URN" is not quite right -- suggestions are welcome.)

Thus I could see something like this in 2141bis:

namestring    = assigned-name [ "?" query ] [ "#" fragment ]
assigned-name = "urn" ":" NID ":" NSS

> readable than any fragment listing.

Juha, I admit that I still don't understand how such a dereferencing
(or, better, metadata-retrieval) service would work. How would a client
of this service discover the address and API of the service? In today's
web, I could see us defining a well-known URI (RFC 5785) at which one
could send a request (passing the assigned URN as a parameter) and then
receiving in response a metadata file in a format like XML or JSON. But
IMHO that doesn't necessitate adding query strings to assigned URNs. In
any case, if people feel that they need to add query strings onto the
end of assigned URNs (or are already doing that in practice) then I now
think it is acceptable to define a way to do that, as long as we say
that it is the responsibility of each namespace that allows query
strings (and fragment identifiers) to carefully define what those mean
in the context of that namespace.

> identifiers has the freedom to decide what to do within its own

Juha, I see a tension between your statements and I am hoping you can
clarify something.

As I read your email, I think you are saying that you would like to use
fragment identifiers with '#' notation. However, then you say that
identification of component parts of resources should never use URI
fragments (i.e., never use '#' notation). It would help me if you could
explain when you think URI fragment syntax is appropriate and when it is
not. What exactly are "logical fragments"? Are there other kinds of
fragments (e.g., "physical fragments")? Your statement "if the chapter
is available as a separate item" seems to indicate that that the item
might be physically separate and therefore that URN-assigning entity
might decide to allow '#' notation to identify such items (it's true
that ISBN does not allow this, but that's a matter for the ISBN
namespace -- the NBN namespace might allow that or might allow
sub-namespaces to come up with their own syntax for fragments, using '.'
instead of '#' or whatever).

Do I understand you correctly?

> is namespace specific fragment-like constructs.

By "namespace specific fragment-like constructs" do you mean "each
namespace is allowed to define its own constructs for identifying
fragment-like things, as long as the syntax does not involve the use of
'#' as the denotation of the start of the fragment-like thing in the
name"? Or do you think that '#' is allowed as well?

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Juha Hakala <[email-masked]>
Date: Tue, 09 Jul 2013 11:14:40 +0300

Hello Peter,

> getting closer to a shared understanding.
I hope so; if we manage to agree on things then I may avoid writing 
national extensions to the IETF RFCs ;-).
>   The easiest
Yes. But while clients should not even send the fragment to servers when 
they retrieve documents, queries will be evaluated by (in this case) URN 
resolvers. In a complex environment resolvers must have quite a lot of 
intelligence ot cope with the situation; for instance, what the user 
wants may best be met by descriptive metadata from the national 
bibliography, the document itself from the institutional repository, or 
the oldest possible manifestation of the work from the digital archive.
> us to determine the exact text in 2141bis and 3406bis.
Good.

> Juha, do I understand correctly that you might suggest that we 
> continue to define assigned URNs (or, perhaps more precisely, URIs 
> with the urn: scheme) as "urn" ":" NID ":" NSS and then specify that 
> fragments and queries might be strings that can be added to an 
> assigned URN? (Perhaps the term "assigned URN" is not quite right -- 
> suggestions are welcome.) Thus I could see something like this in 
> 2141bis: namestring = assigned-name [ "?" query ] [ "#" fragment ] 

Exactly. And it might not do any harm if we make it clear that while 
identifier assignment can be a managed process in some namespaces, once 
the URN has been assigned, anyone - human or a machine - can add a 
fragment or a query to it.

> of this service discover the address and API of the service?
Clients would not need to know anything about it.

Currently URN resolution services have relatively simple mapping tables 
which may simply say that document with this URN is available from this 
/ these location / locations. These resolution services must become more 
versatile with the digital library applications. In the future the 
resolver has to know (for instance) where to obtain descriptive metadata 
about resources, the resources themselves, or past versions of these 
resources. These services may be available via 1-n applications and 1-n 
APIs. Specifying them is beyond the scope of URNbis.

What we need is a set of URN standards which does not become a 
bottleneck when there is a need to specify new resolution services. It 
looks like URNbis is heading to the right direction.
> In today's
I'll take a look at RFC 5785.
> fragment identifiers with '#' notation.
Yes, we should allow the use of URI fragment for indicating a location 
within a resource which has already been identified by a URN.
> However, then you say that
> fragments (i.e., never use '#' notation).
The point being that URI fragments must not identify anything and 
therefore they must not be a part of the NSS.

> It would help me if you could
> fragments (e.g., "physical fragments")?
Yes.  As I see it, URI fragment can be applied to physical fragments in 
a document that has a URN. For instance, if a resource is a digitized 
journal issue with a single URN:NBN, it is possible to cite an article 
within the issue by giving the URN and fragment which takes the users to 
the beginning of the article. Depending on the physical encoding of the 
document, URI fragment can be used to pinpoint any location within the 
article. If URI fragments do identify subordinate resources (as stated 
for instance in http://en.wikipedia.org/wiki/Fragment_identifier) we 
would need some guidelines on what these resources actually are.  Having 
a lot of experience on the use of bibliographic identifiers, I find for 
instance this statement from the Wikipedia page rather interesting:

> The following example identifies lines 11 through 20 of a text document:
>
>   * |http://example.com/document.txt#line=10,20|
>
What happens when the content on lines 11-20 (why not 10-20) changes? 
What is identified after the author cuts the document so that it 
contains only 9 lines? What happens if somebody adds a few lines to the 
beginning of the text? What if document.txt becomes document.xxx and the 
relevant content is on different lines or the fragment does not work any 
more? These, and many other problems disappear if we decide that the URL 
above does not identify anything, and the fragment just indicates a 
location within a resource which at least used to be available in the 
network address given.

Logical fragments such as journal articles or book chapters may or may 
not be represented in the physical encoding of the resource. Be that as 
it may, URI fragments should not be used to identify them. For instance, 
in order to facilitate long term preservation and access the National 
Library of Finland assigns URN:NBNs to e.g. both journal articles and 
images within those articles. This will make it easier to migrate these 
resources to new file formats. If we were using just fragments for 
identifying articles within a journal issue, we might be in trouble if 
the new file format (resulting from migration) would not support the use 
of URI fragments.
> Your statement "if the chapter
Not exactly. It is possible to give an ISBN to chapter 1 of volume 1 of 
Don Quijote, if the chapter is for sale as an e-book. Once the book has 
URN:ISBN, anybody can use URI fragment to provide access to any location 
within the chapter. RFC 3986 would say that these users have identified 
subordinate parts within the chapter. A lot of hairy problems regarding 
e.g. the definition of subordinate parts can be avoided if we just say 
that in theory it is possible to specify any location within the 
document with a fragment, provided that has been identified by the URN.

As I see it, the mess we end up with when we assume that URI fragment 
actually identifies something is just another example why it is a bad 
idea to mix location (in this case not in the Web, but within a single 
document) and identification.
> name"? Or do you think that '#' is allowed as well?
No, we should not allow "#" for namespace specific constructs.

Juha

-- 

  Juha Hakala
  Senior advisor

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 09 Jul 2013 18:40:09 -0600

Hi Juha,

> fragment or a query to it. 

Good point. I'll see what helpful text I can formulate.

> looks like URNbis is heading to the right direction.

I hope so!

> I'll take a look at RFC 5785.

It's not really relevant to 2141bis and 3406bs, but you might find it
helpful or interesting for 2483bis.

> within a resource which has already been identified by a URN.

Understood.

> therefore they must not be a part of the NSS.

OK.

I will work to write some text reflecting that understanding.

> would need some guidelines on what these resources actually are. 

It seems to me that it's the responsibility of the manager / spec for a
particular URN namespace to define that (since it will depend very much
on the kinds of URNs being minted / resources being named).

> Having

Don't believe everything you read on Wikipedia. :-)

> location within a resource which at least used to be available in the

Right.

> the new file format (resulting from migration) would not support the use

OK, that makes sense.

> document) and identification.  

Yes, naming is hard. If I may paraphrase you, an identifier (in this
case a URN) provides a name for a resource, whereas a fragment merely
distinguishes a particular location within or segment of a resource.

> No, we should not allow "#" for namespace specific constructs.

Great, thanks for clariying your statement.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/