Subject: [art] I-D next steps (draft-taylor-uuid-ncname)

From: "Dorian Taylor (IETF Lists)" <[email-masked]>
Date: Mon, 4 Jan 2021 20:56:03 -0800

Greetings folks,

I penned an I-D back in July and got distracted away from it. That would be this:

https://tools.ietf.org/html/draft-taylor-uuid-ncname-00

It’s a spec for alternate representations of UUIDs, so that they can be tucked places the syntax wouldn’t normally go. From the document:

1.  Introduction

  There are a number of places in formal languages where it would be
  useful to put UUIDs, but the grammar forbids it.  Many grammars
  forbid identifiers to begin with numbers, or contain hyphens, or
  contain colons (as with the URN representation in RFC 4122 [RFC4122]).
  The NCName production [XML-NAMES], which is pervasive in
  XML and RDF applications, is one such example.  Up until a recent
  change, the HTML ID production had similar constraints.  Virtually
  every programming language likewise requires identifiers such as
  variables and function names to start with a letter or underscore,
  and very few admit hyphens.  This constraint causes developers to
  turn to ad-hoc solutions when they want to use UUIDs in these places.

  This document specifies a representation - or rather, two
  representations - as well as the related transformations to and from
  the familiar UUID format.  A provisional name for these
  representations is _UUID-NCName_, with the two variants styled as
  _UUID-NCName-32_ and _UUID-NCName-64_, referring to the base of their
  respective encodings.  The goal of this specification is in part to
  eliminate an extra decision on the part of developers who find
  themselves in this position, and in part to provide alternative
  representations for UUIDs which remain valid but are shorter than the
  original.

1.1.  Motivation & Applications

  The purpose of an identifier in general is to pick out some
  information resource or other, such that it can be referred to,
  ideally unambiguously.  The purpose of a large, generated identifier
  like the UUID, is to satisfy the uniqueness criterion while also
  specifying a datatype and normal form for said identifiers, and
  ultimately alleviate the need to sit down and think these identifiers
  up.  Why one would want to go inserting UUIDs in places they wouldn't
  otherwise fit, is so these UUIDs can be cross-referenced in some
  other database where they _do_ fit.  Consider:

  *  A component content management system that uses UUIDs to identify
     elementary content components, uses the UUID-NCName-64
     representations of the same UUIDs as fragment identifiers for when
     those components are transcluded.

  *  A literate programming system uses the UUID-NCName-32
     representation as stable identifiers for all symbols (variables,
     constants, class names, etc.), enabling said identifiers to be
     defined and described elsewhere, while still yielding
     syntactically-correct code.

(The symbols look like 03087b4b-07c9-42b0-aca5-c1b52dd52937 <-> EAwh7SwfJKwylwbUt1Sk3K <-> eamehwsyhzevqzjobwuw5kkjxk .)

I’m not sure what the next steps are, but I recall “emailing ART” was high on the list. I would appreciate any help in moving this forward.

Happy new year,

--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 5 Jan 2021 16:35:19 +0900

Hello Dorian,

I wonder why you didn't choose to just add a prefix and convert the 
hyphens to something else. That would make conversion much easier, and 
would make it possible to compare the different representations. It 
would also possibly make do with a single new reprenestation.

You may have had good reasons for your choice, but at the least that 
choice should be explained in the draft.

Regards,   Martin.

On 05/01/2021 13:56, Dorian Taylor (IETF Lists) wrote:
> _______________________________________________
> art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/art
>

---

From: "Dorian Taylor (IETF Lists)" <[email-masked]>
Date: Tue, 5 Jan 2021 01:02:00 -0800

Hi Martin,

Thanks for reading the draft. Indeed there are reasons, and one of the reasons is the lexical length of the symbol (which I thought I did address but I suppose I could emphasize it further, though I suppose that is what this internet draft process is for).

You indeed see a lot of examples in the wild of using UUIDs to identify e.g. document fragments, with ad-hoc prefixes to get around the grammar constraint that an ID must start with ^[A-Za-z_] (and it is worth remarking that the HTML5 people have since relaxed the definition of ID to permit something like an arbitrary UUID). However, in the specific case of ID-typed markup attributes, XML IDs are still bound by the ^[A-Za-z_] grammar, so if you, for instance, had a content management system in DITA that projected results into HTML5, and you wanted to mark the ID attributes in the source with UUIDs, you’d still have to come up with some kind of mechanism for getting around the ^[A-Za-z_] problem, and then the question there is inevitably do we ad-hoc this once again or do we try to come up with a standard?

So, yes, I suppose one possibility is to create an RFC that just says “prepend `uuid_`* to every UUID and maybe remove the hyphens between the segments” and that will solve the problem of fitting into those particular grammar constraints.

(* Note we use an underscore in this hypothetical because we are also considering grammars that don’t admit hyphens, which is most of them.)

The problem with this is we now have a symbol that is 37 (36 - 4 + 5) characters long. Now, for markup purposes this isn’t an especially big deal, as there is compression over the wire and whatnot, but markup isn’t the only use case I’m thinking of for this identifier format.

In particular, I believe these identifiers have value as stand-ins in literate programming systems for arbitrary programming languages, such that programs will compile and run whether or not the identifiers have been substituted. Here the length of the identifier is actually significant, most notably in C (limited to 31 ASCII characters, presumably because of the string-terminating null byte). This Base64 UUID representation is always 22 bytes and the Base32 representation is always 26. Furthermore, certain programming languages (notably Ruby) impute certain semantics to the letter case of the identifier (CONSTANT Class variable &c), so it is helpful that the first character of the identifier be case-insensitive.

So the goal behind this specification (and its currently two concomitant implementations) is to specify a representation (actually two, base64url and base32) for UUIDs that can be inserted into places that UUIDs, when represented ordinarily, wouldn’t be able to fit, and furthermore to demarcate that they are (relatively) unambiguously UUIDs and not something else. Given the applications, we have both the grammar constraints and the available symbol length to think about.

I *will* potentially concede one possibility however and that is to supplement the Base64 representation with Manu Sporny's Base58 <https://tools.ietf.org/html/draft-msporny-base58-02>, which was unknown to me when I designed the initial spec. This will eliminate the hyphen character altogether which is the biggest technical sticking point in the versatility of these symbols.

Regards,

> On Jan 4, 2021, at 11:35 PM, Martin J. Dürst <[email-masked]> wrote:
> 
> Hello Dorian,
> 
> I wonder why you didn't choose to just add a prefix and convert the hyphens to something else. That would make conversion much easier, and would make it possible to compare the different representations. It would also possibly make do with a single new reprenestation.
> 
> You may have had good reasons for your choice, but at the least that choice should be explained in the draft.
> 
> Regards,   Martin.

--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 6 Jan 2021 10:59:15 +0900

Hello Taylor,

Thanks for your explanations.

> Thanks for reading the draft. Indeed there are reasons, and one of the reasons is the lexical length of the symbol (which I thought I did address but I suppose I could emphasize it further, though…

I re-checked, and found 'length' once and 'long' twice, all in context 
of describing the length of your identifiers. There's no discussion of 
the advantage of shortening.

> So, yes, I suppose one possibility is to create an RFC that just says “prepend `uuid_`* to every UUID and maybe remove the hyphens between the segments” and that will solve the problem of fitting i…

It wouldn't even have to be 'uuid_'; your identifiers also don't make it 
obvious that they are UUIDs. But it would be good to have a standard way 
to do it while still maintaining the ability to compare (by hand) with 
actual UUIDs.

> In particular, I believe these identifiers have value as stand-ins in literate programming systems for arbitrary programming languages, such that programs will compile and run whether or not the identifiers have been substituted.

I have my doubts that using UUIDs (actually, identifiers derived from 
UUIDs) have much of a purpose as identifiers (not data) in programming 
systems. Even if there are such uses, I would strongly object to call 
systems using such identifiers "literate programming". Literate 
programming is about being able to read programs, and using explanatory 
as opposed to random-looking is the first (and in most actual programs, 
almost the only) and very important step in that direction.

> Here the length of the identifier is actually significant, most notably in C (limited to 31 ASCII characters, presumably because of the string-terminating null byte). This Base64 UUID representation is always 22 bytes and the Base32 representation is always 26. Furthermore, certain programming languages (notably Ruby) impute certain semantics to the letter case of the identifier (CONSTANT Class variable &c), so it is helpful that the first character of the identifier be case-insensitive.

The later point would work with any prefix to actual UUIDs, too.

> So the goal behind this specification (and its currently two concomitant implementations) is to specify a representation (actually two, base64url and base32) for UUIDs that can be inserted into pla…

I'm not sure you can say that e.g. the examples in your Table 2 are 
"(relatively) unambiguously UUIDs".

> I *will* potentially concede one possibility however and that is to supplement the Base64 representation with Manu Sporny's Base58 <https://tools.ietf.org/html/draft-msporny-base58-02>, which was u…

What about defining just one base-x variant, but also a variant that can 
be textually mapped by hand?

Regards,   Martin.

>> I wonder why you didn't choose to just add a prefix and convert the hyphens to something else. That would make conversion much easier, and would make it possible to compare the different representations. It would also possibly make do with a single new representation.
>>

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 05 Jan 2021 22:18:20 -0500

A couple of points:

   There are a number of places in formal languages where it would be
   useful to put UUIDs, but the grammar forbids it.

A UUID is a 128-bit identifier, as far as I've seen; the hex format
"4abc6330-f548-4e67-b9f9-12d4323769cd" is just one way of representing
it.

You really want to say something like "There are a number of places in
formal languages where it would be useful to put UUIDs, but the grammar
forbids all of the currently-standardized representations."

It's not clear to me why the UUID is put through a complicated
reformatting.  That has the disadvantage of risking incompatibility with
future forms of UUID, and increasing the number of incorrect
implementations.

Why not say "an entry in the Obfuscated C Code contest"?

The syntax section should start with a positive statement of the
restrictions that the syntax is intended to achieve, expressed as
concisely as possible.  E.g. "the UUID is encoded into N alphanumerics
(of both cases)".

Dale

---

From: "Dorian Taylor (IETF Lists)" <[email-masked]>
Date: Tue, 5 Jan 2021 20:35:23 -0800

> On Jan 5, 2021, at 7:18 PM, Dale R. Worley <[email-masked]> wrote:
> 
> A couple of points:
> 

I like this. Thanks.

> future forms of UUID, and increasing the number of incorrect

To my knowledge there is nothing you could do with a future UUID version (that you could still call a UUID) that would break compatibility with this representation. UUID versions can only go up to 15 (or `f` in the hexadecimal representation, because that is all there is room for) and the size must always be 128 bits. Whatever goes on inside is opaque. Indeed the “complicating” part of pulling the version and variant bits out and sticking them on the ends — in addition to making the remaining bits flush in the encoding, and providing a hint as to what the symbol is — means that literally anything can go in the middle. (And if the contents aren’t immediately intelligible I’m not especially concerned. Be honest, when’s the last time you’ve eyeballed a timestamp or MAC address in a V1 UUID anyway? ;)

As for implementation compliance, there are already two reference implementations if the algorithm description is too ambiguous. (And by all means, if it is too ambiguous please tell me where so I can correct it.)

*  Perl, https://metacpan.org/pod/Data::UUID::NCName
*  Ruby, https://rubygems.org/gems/uuid-ncname

> Why not say "an entry in the Obfuscated C Code contest"?

I am explaining this in great detail in another message to this list but you’re the second person to complain about it so clearly it’s too confusing and I should just nuke it from the spec.

> concisely as possible.  E.g. "the UUID is encoded into N alphanumerics

Sure. an XML ID is basically ^[A-Za-z_][0-9A-Za-z_.-]*$ (where A-Za-z includes Unicode alphas and 0-9 includes Unicode digits). Same goes for Turtle and NTriples (and TriG and NQuads; they all pretty much use NCName). Identifiers in programming languages are for the most part constrained to ^[A-Za-z_][0-9A-Za-z_]*, and many have additional constraints on the length any identifier is allowed to be, (e.g. 31 characters in the C standard). The pattern of constraining symbols to “must start with a letter or underscore, followed by one or more letters, digits, or underscores” is pretty ubiquitous in computing. If it helps to say that explicitly, I very well can say that.

Thanks for your feedback,

--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 6 Jan 2021 08:06:13 +0100

> (The symbols look like 03087b4b-07c9-42b0-aca5-c1b52dd52937 <-> EAwh7SwfJKwylwbUt1Sk3K <-> eamehwsyhzevqzjobwuw5kkjxk .)

I like having a compact, less arcane text string/identifier representation of UUIDs.

Is saving four characters (from 26 down to 22) really worth having two of them (one of which has the hassle of being both kebab- and snake_, although you did think about bookending them)?

I’d stick with the base32 one only, which will work out of the box in any programming language (that is not limiting identifier length) as well.

Grüße, Carsten

---

From: "Dorian Taylor (IETF Lists)" <[email-masked]>
Date: Wed, 6 Jan 2021 00:27:24 -0800

> On Jan 5, 2021, at 5:59 PM, Martin J. Dürst <[email-masked]> wrote:
> 
> I re-checked, and found 'length' once and 'long' twice, all in context of describing the length of your identifiers. There's no discussion of the advantage of shortening.

The title of the document is “Compact, Grammar-Friendly Representations for UUIDs”. If you think it needs a paragraph explaining why shorter is better I can consider putting one in the next draft.

> It wouldn't even have to be 'uuid_'; your identifiers also don't make it obvious that they are UUIDs. But it would be good to have a standard way to do it while still maintaining the ability to compare (by hand) with actual UUIDs.

What makes a conventional UUID representation obvious that that’s what it is? It’s a certain length and has peculiarities in its representation that give it away—namely four specifically-placed hyphens (which themselves are legacy), and for the astute, the first characters in the third and fourth segments. With the hyphens removed the string *might* be a UUID or it just might be 32 random hexadecimal characters. Moreover one could argue the only reason why the conventional representation is distinctive is because we are used to it.

The symbols I designed are 22 and 26 characters respectively, and always start and end with a letter (which correspond 1:1 to the symbols in the third and fourth segments, respectively). Is that ambiguous? Perhaps in the grand scheme of things, when completely devoid of context, but what about something like:

  <section id="EG0gCP2vb65ED-5k0utZxL">

Will that still be ambiguous in 2040?

(That, by the way, is 1b48023f-6bdb-4eb9-b103-fb9934bad671.)

As for comparing by hand with the existing conventional representation, I did not—and still do not—consider that to be an important requirement. I have already written two implementations to do the conversion:

* https://metacpan.org/pod/Data::UUID::NCName
* https://rubygems.org/gems/uuid-ncname

I should also note that there are a ton of implementations out there that already decode and re-encode UUIDs to binary or naïve base64 representations. This is already something that is happening in the wild; my goal with this I-D is to give developers a desirable set of characteristics for compact isomorphic representations of UUIDs, wrapped up in a bow, with a lot of the thinking already done so they don’t have to go and reinvent the wheel. This is analogous to specifying the `urn:uuid:` prefix in RFC4122—do you know how much I use that? A lot.

> I have my doubts that using UUIDs (actually, identifiers derived from UUIDs) have much of a purpose as identifiers (not data) in programming systems. Even if there are such uses, I would strongly object to call systems using such identifiers "literate programming". Literate programming is about being able to read programs, and using explanatory as opposed to random-looking is the first (and in most actual programs, almost the only) and very important step in that direction.

This really was just meant to be an example of an application and not a major digression but perhaps I should have taken more time to explain this part:

My understanding of literate programming is that the human-readable prose dominates both the process and the artifacts. An author writes a master document and organizes it in a way most amenable to writing and reading. Then the author performs an operation that transforms the master, splitting it into 1) a polished document and 2) an executable program, and rearranging the latter to be more favourable to the compiler/interpreter if need be.

(This is of course not the same as API documentation generator packages; rather it is something close to the inverse.)

One thing you could do in Knuth’s WEB but can’t in say, Literate Haskell (unless something has recently changed), is define your symbols in the prose. Instead you have to define them in the code, and by all means you can *mention* them in the prose, but the code and prose do not interact. This means symbols can accumulate in the code that are not elucidated in the prose, effectively reverting the endeavour to regular, everyday, *non*-literate programming, with fancy comments. So any modern LP system is going to want a mechanism for controlling (and therefore propagating) all symbol declarations from the prose.

Now, symbols aren’t just symbols, they represent variables, constants, classes, function/method names, etc. They generally have a notion of a type and possibly some additional semantic constraints. Indeed, there are  variables (in particular) that transcend individual programs (like the use of `i` in a `for` loop) and are useful to stake out as addressable entities in their own right. (Indeed, the notion of an iterator variable is distinct whether you call it `i` or not.)

This brings me to my main point: the compiler/interpreter doesn’t care about what identifiers are called, only that they are declared, that they conform to the grammar, and they are distinct. If you’re controlling symbol declarations from the prose, why restrict yourself to the code’s grammar? Why not simply associate whatever word or phrase you want with a durable identifier and have the system handle the rest?

(If that’s too weird for you then you can happily assign a conforming identifier *in addition to* a descriptive declaration in the prose — indeed you will probably want to do this anyway for debug output — but the point is the code will run either way.)

I could go on about hypermedia in LP systems, content-addressable code, etc, but really my objective with this is to be able to remove the task of naming things from the domain of formal language and put it into the domain of natural language, and furthermore not hold up the development process because you’re stuck trying to decide on a name for something. I use UUIDs for that purpose in other areas and I want to use them in this one, but I can’t because UUIDs are too long and don’t fit the grammar constraints. So I invented an isomorphic representation with the desired characteristics.

Note that this is something I am already working on and I will be using this identifier format whether it is accepted as an RFC or not. I am not looking for a critique on this approach. If all mentioning literate programming in the I-D is going to do is detract from the technical content of the specification itself, I will happily remove it.

> The later point would work with any prefix to actual UUIDs, too.

I suppose, but you would also have to remove the hyphens to accommodate the stricter grammars, and even then you would have a symbol that was no fewer than 33 characters long.

> I'm not sure you can say that e.g. the examples in your Table 2 are "(relatively) unambiguously UUIDs".

They will be in 20 years; likely much sooner.

> What about defining just one base-x variant, but also a variant that can be textually mapped by hand?

What is your concern here? It seems like you are preoccupied with the requirement that a UUID representation ought to be reckoned by eyeball with the conventional representation. You’re just not going to get that *and* get under 33 characters, but moreover you haven’t explained why you think this is so important. Why shouldn't there be a well-thought-out alternate representation of UUIDs? People are already cramming the raw bytes into base64 and sticking those all over the place.

--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com

---

From: "Dorian Taylor (IETF Lists)" <[email-masked]>
Date: Wed, 6 Jan 2021 00:34:05 -0800

> On Jan 5, 2021, at 11:06 PM, Carsten Bormann <[email-masked]> wrote:
> 

The base64 representation is slightly shorter than the base32, and it is suitable for XML NCName grammar productions which were the original target. I included both encodings because it cost me only slightly more to specify (and implement) both, and invariably somebody would complain if one was present but not the other.

> I’d stick with the base32 one only, which will work out of the box in any programming language (that is not limiting identifier length) as well.

Indeed, I anticipate using the base32 representation in anything other than NCNames (which all things considered is most places).

Regards,

--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com