Subject: Re: [Cbor] 7049bis: The concept of "optional tagging" is not really used in practice #126

From: Laurence Lundblade <[email-masked]>
Date: Sun, 3 Nov 2019 08:45:33 -0800

I’m not really a data structure scientist or such, but I think I can see Christophe’s point. 

Maybe CBOR-based (and JSON-based) protocols don’t have a formal schema language, but these protocols rely on ordering and such. For example in a COSE_Sign1 it is expected that the first data item is the protected headers, the second the unprotected headers, the third the payload and the fourth the signature. I don’t think you can call them self-describing.

It seems like CBOR and JSON say “no schema’” to distance from the horror of XML schemas, but in reality CDDL and prose protocol specs are schemas in spirit.

Maybe a key question here is whether you can say in CDDL “this next item must always be interpreted as a date even though it will never have a date tag”. If CDDL doesn’t have than, then you can’t describe some CBOR-protocols with it. CWT would be one of those protocols as it forbids adding the tag to dates.

To summarize what I understand about tagging:

The designer of a new CBOR data item type like a date format will generally register a tag for it. These new data types can be really simple, like epoch dates or really complex like COSE_Sign1.

The designer of a protocol using a new data type will indicate in their protocol for each occurrence of it whether the tag must be present or not (never saying the tag may or may not be present). The designer will typically require the tag only when necessary to disambiguate the type of the data item.

The implementor of a general purpose library to generate one of these new data item types must give the caller the option to include or not include the tag. Maybe this is just by never automatically outputting the tag and having a distinct output tag function.

The implementor of a general purpose library to decode one of these new data types must allow the caller to say that the next data item should be decoded as this new data type whether or not it is tagged. Maybe it even errors out if it is tagged for the cases where the protocol document says no tag should be used.

What I don’t know is whether CDDL can describe all this desired behavior.

LL

> On Oct 24, 2019, at 1:50 AM, Christophe Lohr <[email-masked]> wrote:
> 
> On 23/10/2019 13:38, Carsten Bormann wrote:
>> Section 3.4 talks about "optional tagging" as a secondary purpose of tags. But in today's CBOR protocols, tags are rarely "optional" in the sense that they can simply be left out without a change in semantics, as 3.4 para 3 implies.
>> 
>> This concept comes up again in 4.2.2, where "optional tagging" is outlawed in deterministic encoding (but then the text goes on to explain that protocols might choose to retain tags, but doesn't say why).
> 
> To be honest, I don't really understand how much optional are tags.
> 
> A CDD rule with tags matchs cbor items with tags and reject cbor items
> without tags. Tags are not optional from the data-model point of view.
> 
> 
> Moreover, please consider this CDDL objective:
> (https://tools.ietf.org/html/rfc7049#section-1.1)
> 
>    3.  Data must be able to be decoded without a schema description.
>        *  Similar to JSON, encoded data should be self-describing so
>           that a generic decoder can be written.
> 
> 
> Well, how to do this without putting tags everywhere for everything?
> (Or I need more explanation about what is "self-describing" and what is
> a "schema description")
> 
> Let say I receive data. How may I know that this number is a temperature
> and not a distance, and that this byte-string is an uuid and not a small
> picture?
> 
> The first way is to have a schema (written or not): That is to say a
> certain preliminary knowledge of expected data which tell me that this
> number at this place or associated to this map-key is a temperature.
> The second way is to decorate data with tags, all data.
> A third way is a compromise between the two first ones: I have a certain
> level of preliminary knoledge of what data are (a kind of schema
> description), with possibly some missing parts that are filled by tags.
> 
> But the only way to decode data _without_ a schema description is to
> have tags everywhere for everything.
> Surprisingly, json has no tags and is claimed to be self-describing. Is
> it really? I'm lost.
> 
> My feeling is that this objective CBOR should be not so demanding.
> 
> Best regards,
> Christophe
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 3 Nov 2019 17:59:41 +0100

Hi Laurence,

CDDL does not describe behavior, but the shape of data.

> Maybe a key question here is whether you can say in CDDL “this next item must always be interpreted as a date even though it will never have a date tag”.

CDDL can say “this item is a number”.  It does not tell you how to “interpret” things, that would be the job of a language that transforms the data just received into data that is used by an application.

> If CDDL doesn’t have than, then you can’t describe some CBOR-protocols with it.

You sure can “describe” the shape of data in CBOR protocols, but you will also need some information about how you plan to interpret the data.

> CWT would be one of those protocols as it forbids adding the tag to dates.

(For the exp, nbf, and iat claims, or claim numbers 4 to 6:) Yes, the CWT RFC (RFC 8392) tells you that the value here is a number, not a tagged date.

> The designer of a protocol using a new data type will indicate in their protocol for each occurrence of it whether the tag must be present or not (never saying the tag may or may not be present). T…

Right.  If you need to register for CWT a new claim that could either take a number (such as the longitude of the satellite that this claim is about) or a date/time (such as the time the satellite was launched), then a tag could be useful to make that distinction.  

That example is a bit contrived, because it’s just not as usual to have a choice between a number and a date.  More likely might be a choice between a date/time represented as a Tag 1 and a Tag 1001.  The encoding could choose to leave off the tag from the number that is the enclosed item of Tag 1, so you would have a choice between a number and a Tag 1001.

> The implementor of a general purpose library to decode one of these new data types must allow the caller to say that the next data item should be decoded as this new data type whether or not it is…

Right.

> What I don’t know is whether CDDL can describe all this desired behavior.

CDDL can describe the shape of the data interchanged, but it can’t describe the mapping to application semantics.  It can provide hints, and that’s one of the things that the unwrap operator is good for: When you apply it to a tag, this is a hint that you do not just want the data shape of that tag’d enclosed item, but also its semantics.

Grüße, Carsten

---

From: Henk Birkholz <[email-masked]>
Date: Sun, 3 Nov 2019 18:04:47 +0100

Hi Laurence,

please see below.

On 03.11.19 17:45, Laurence Lundblade wrote:
> I’m not really a data structure scientist or such, but I think I can see 
> Christophe’s point
> 
> Maybe CBOR-based (and JSON-based) protocols don’t have a formal schema 
> language, but these protocols rely on ordering and such. For example in 
> a COSE_Sign1 it is expected that the first data item is the protected 
> headers, the second the unprotected headers, the third the payload and 
> the fourth the signature. I don’t think you can call them self-describing.
> 
> It seems like CBOR and JSON say “no schema’” to distance from the horror 
> of XML schemas, but in reality CDDL and prose protocol specs are schemas 
> in spirit.
> 
> Maybe a key question here is whether you can say in CDDL “this next item 
> must always be interpreted as a date even though it will never have a 
> date tag”.

I would not use the term "interpretation" here. CDDL is a language, it 
in unambiguous in that regard. But I think it can do exactly what you 
mean :-) It describes the structure of CBOR/JSON messages/documents - if 
it precise enough, CBOR tags are not needed, but to ensure that 
precision you need a dedicated authority, I think.

If CDDL doesn’t have than, then you can’t describe some
> CBOR-protocols with it. CWT would be one of those protocols as it 
> forbids adding the tag to dates.
> 
>     The designer of a new CBOR data item type like a date format will
>     generally register a tag for it. These new data types can be really
>     simple, like epoch dates or really complex like COSE_Sign1.
> 
>     The designer of a protocol using a new data type will indicate in
>     their protocol for each occurrence of it whether the tag must be
>     present or not (never saying the tag may or may not be present). The
>     designer will typically require the tag only when necessary to
>     disambiguate the type of the data item.
> 
>     The implementor of a general purpose library to generate one of
>     these new data item types must give the caller the option to include
>     or not include the tag. Maybe this is just by never automatically
>     outputting the tag and having a distinct output tag function.
> 
>     The implementor of a general purpose library to decode one of these
>     new data types must allow the caller to say that the next data item
>     should be decoded as this new data type whether or not it is tagged.
>     Maybe it even errors out if it is tagged for the cases where the
>     protocol document says no tag should be used.
> 
>> On Oct 24, 2019, at 1:50 AM, Christophe Lohr 
>> <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>> Section 3.4 talks about "optional tagging" as a secondary purpose of 
>>> tags. But in today's CBOR protocols, tags are rarely "optional" in 
>>> the sense that they can simply be left out without a change in 
>>> semantics, as 3.4 para 3 implies.
>>>
>>> This concept comes up again in 4.2.2, where "optional tagging" is 
>>> outlawed in deterministic encoding (but then the text goes on to 
>>> explain that protocols might choose to retain tags, but doesn't say why).
>>
>> [email-masked] <mailto:[email-masked]>

---

From: Christophe Lohr <[email-masked]>
Date: Sun, 3 Nov 2019 21:40:39 +0100

>>           that a generic decoder can be written.

My confusion comes probably from vocabulary issues.

The sentence here probably talks about "a schema description" from the
/coding-decoding/ point of view, and nothing else.

Such as, let's say, the IP header: without the RFC in hands (its
description), there is no way to decode a set of 20 bytes and to guess
that the first 4 bits are an unsigned int, followed by another 4 bits
unsigned int, followed by an array of 8 booleans, etc. 
A CBOR (or a JSON) coding format can tell this, without the need of
another writing document to specify this.

However, I read "schema description" from the /semantic/ point of view:
a description which explains the meaning of data items.  The IP RFC not
only tells that the first 4bits are an unsigned int, it also tells that
this number is the protocol version. 
CBOR (neither JSON) can't tells this by itself, except if one defines a
TAG for this.

So, the next question is: "is there some guidelines for using TAGs?"

Well, it's probably too early. One may have to wait that CBOR usages
grow in maturity.
But what's your opinion?

Let's say I'm designing a system.  CBOR is used (for messages, recording
data files, or whatever).
What should I decide for my system design regarding CBOR TAGs?
Shall I:
- prohibit TAGs since this is redundant with other parts of my design
specifications (which already explicit the meaning of each field); or
- put TAGs everywhere for everything because TAGs bring semantic to data; or
- add TAGs to some fields and not to others (which ones and why?)

The right answer is probably: "this depends". ;-)
but of what?

Best regards
Christophe

---

From: Laurence Lundblade <[email-masked]>
Date: Sun, 3 Nov 2019 13:02:16 -0800

OK. I understand.  To use it one of these new types in CDDL, particularly those with a internal structure, you make a pair of CDDL headers, one for tagged and one for untagged that describe it like COSE did (from 8152):

   COSE_Messages = COSE_Untagged_Message / COSE_Tagged_Message

   COSE_Untagged_Message = COSE_Sign / COSE_Sign1 /
       COSE_Encrypt / COSE_Encrypt0 /
       COSE_Mac / COSE_Mac0

   COSE_Tagged_Message = COSE_Sign_Tagged / COSE_Sign1_Tagged /
       COSE_Encrypt_Tagged / COSE_Encrypt0_Tagged /
       COSE_Mac_Tagged / COSE_Mac0_Tagged

   ...

   COSE_Sign1_Tagged = #6.18(COSE_Sign1)

   COSE_Sign1 = [
       Headers,
       payload : bstr / nil,
       signature : bstr
   ]

Would be cool if there were standard CDDL definitions ready for normative reference for all the “tagged” data types defined in 7049 / 7049bis.

LL

> On Nov 3, 2019, at 8:59 AM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Laurence,
>

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 3 Nov 2019 22:03:59 +0100

> a description which explains the meaning of data items.  

Right.  “Schema” probably is one of the most misused terms in this space.
(That’s why CDDL is called a “data definition language”.)

> The IP RFC not

Tags (we prefer to write simple English words in lower case) tell you how to interpret an enclosed item with different (additional) data semantics.  So a tag with number 1 tells you the enclosed number really is to be interpreted as a POSIX epoch-based date.

> So, the next question is: “is there some guidelines for using TAGs?"
> 

CBOR has been around for half a decade now, so I think we have a pretty good comprehension now of when to use tags.

> What should I decide for my system design regarding CBOR TAGs?

➔ Use tags when they are useful.

There are no general guidelines like the ones you propose below, because the usefulness depends on the specific context.

> specifications (which already explicit the meaning of each field); or

If you have a relatively rigid data shape (“schema” in the usual structural sense), you may indeed not need tags, because you can infer an alternative interpretation from structure (e.g., field names in a map used as a struct, position in a record, etc.).  They may still be useful when you want to express a choice, e.g., if you want to support both epoch-based and text-based dates, use Tag 0 or Tag 1.  Another example is integers: If you expect to interchange integers that might not fit into 64 bits, use a choice between a built-in integer (major types 0 and 1) and a tag 2/3:

                  uint = #0
                  nint = #1
                  int = uint / nint
                  biguint = #6.2(bstr)
                  bignint = #6.3(bstr)
                  bigint = biguint / bignint
                  integer = int / bigint

> - put TAGs everywhere for everything because TAGs bring semantic to data; or

"Everywhere” I don’t know.  But if you expect your implementations to rely on generic decoders/encoders doing the work, using tags may be a labor-saving device.  This is particularly useful when CBOR is used for general serialization in a programming environment (where you may not have a hard and fast data definition with your data).

> - add TAGs to some fields and not to others (which ones and why?)

Yes.  Only add them when they are useful.  To express a choice, and/or to have the generic codec do the work.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 3 Nov 2019 22:05:16 +0100

> Would be cool if there were standard CDDL definitions ready for normative reference for all the “tagged” data types defined in 7049 / 7049bis.

Do you mean anything beyond Appendix D of RFC 8610?

https://tools.ietf.org/html/rfc8610#appendix-D

Grüße, Carsten

---

From: Henk Birkholz <[email-masked]>
Date: Sun, 3 Nov 2019 22:06:51 +0100

Well there is this for starters

On 03.11.19 22:02, Laurence Lundblade wrote:
> OK. I understand.  To use it one of these new types in CDDL, 
> particularly those with a internal structure, you make a pair of CDDL 
> headers, one for tagged and one for untagged that describe it like COSE 
> did (from 8152):
> 
>     ....
> Would be cool if there were standard CDDL definitions ready for 
> normative reference for all the “tagged” data types defined in 7049 / 
> 7049bis.
> 
> LL
> 
> 
>> On Nov 3, 2019, at 8:59 AM, Carsten Bormann <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Hi Laurence,
>>
>>> Maybe a key question here is whether you can say in CDDL “this next 
>>> item must always be interpreted as a date even though it will never 
>>> have a date tag”.
>>
>> CDDL can say “this item is a number”.  It does not tell you how to 
>> “interpret” things, that would be the job of a language that 
>> transforms the data just received into data that is used by an 
>> application.
>>
>>> CBOR-protocols with it.
>> You sure can “describe” the shape of data in CBOR protocols, but you 
>> will also need some information about how you plan to interpret the data.
>>
>> (For the exp, nbf, and iat claims, or claim numbers 4 to 6:) Yes, the 
>> CWT RFC (RFC 8392) tells you that the value here is a number, not a 
>> tagged date.
>>
>> Right.  If you need to register for CWT a new claim that could either 
>> take a number (such as the longitude of the satellite that this claim 
>> is about) or a date/time (such as the time the satellite was 
>> launched), then a tag could be useful to make that distinction.
>>
>> That example is a bit contrived, because it’s just not as usual to 
>> have a choice between a number and a date.  More likely might be a 
>> choice between a date/time represented as a Tag 1 and a Tag 1001.  The 
>> encoding could choose to leave off the tag from the number that is the 
>> enclosed item of Tag 1, so you would have a choice between a number 
>> and a Tag 1001.
>>
>>> The implementor of a general purpose library to generate one of these 
>>> new data item types must give the caller the option to include or not 
>>> include the tag. Maybe this is just by never automatically outputting 
>>> the tag and having a distinct output tag function.
>>>
>> CDDL can describe the shape of the data interchanged, but it can’t 
>> describe the mapping to application semantics.  It can provide hints, 
>> and that’s one of the things that the unwrap operator is good for: 
>> When you apply it to a tag, this is a hint that you do not just want 
>> the data shape of that tag’d enclosed item, but also its semantics.
>>
>> Grüße, Carsten
>>
>>
> 
>

---

From: Henk Birkholz <[email-masked]>
Date: Sun, 3 Nov 2019 22:07:57 +0100

Okay, as I am only seem to add redundancy, I'll just add an offset of 30 
min before I reply here :o)

---

From: Jim Schaad <[email-masked]>
Date: Sun, 3 Nov 2019 13:48:01 -0800

From: CBOR <[email-masked]> On Behalf Of Laurence Lundblade
Sent: Sunday, November 3, 2019 8:46 AM
To: Christophe Lohr <[email-masked]>
Cc: [email-masked]
Subject: Re: [Cbor] 7049bis: The concept of "optional tagging" is not really used in practice #126

[JLS] I consider the term self-describing to be a completely different beast.  The data can be parsed without the knowledge of the schema.  This is not a true statement with ASN.1 where a data description label, such as this is a SEQUENCE can be replace with a tag and the parser without the schema has no idea what the data type and structure of the data type is supposed to be.  This is not an issue for XML either as far as I know.

Jim

LL

On Oct 24, 2019, at 1:50 AM, Christophe Lohr <[email-masked] <mailto:[email-masked]> > wrote:

Best regards,
Christophe

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 4 Nov 2019 01:03:10 +0100

> It seems like CBOR and JSON say “no schema’” to distance from the horror of XML schemas, but in reality CDDL and prose protocol specs are schemas in spirit.

One of the innovations of XML with respect to SGML was that it allowed schemaless parsing (parsing without a DTD).  In practice, this was marred by having default values in the DTD, so you couldn’t quite use schemaless parsing with a lot of real-world data.  But it still was a major item of progress, and people who got the point started to shun adding defaults to DTDs.

Schemaless parsing means that you can develop a parser independent of the application.
This is a major boon when you consider the fact that it is quite easy to introduce bugs in parsers, and bugs in input parsers are a significant attack vector.
Application considerations don’t enter a schemaless parser, so you can use a hardened, debugged implementation for parsing.

Default value processing is an example of data transformation, from the parsed input to something that an application can use (and back!).  Adding information about more general forms of such a transformation to a data definition language is a major item of interest.  First of all, it needs a data model for what the application can use.  Doing only defaults processing as in a DTD-based XML parser is easy, because the application data model is a simplified version of the interchange data model.  Now if you add something like tag processing (or, more generally, conversions between platform data types and the conventions used for representing them in interchange), the abstractions can get more leaky.

Fortunately, none of this is needed for completing 7049bis.

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Mon, 4 Nov 2019 08:28:01 -0800

> On Nov 3, 2019, at 1:05 PM, Carsten Bormann <[email-masked]> wrote:
> 

Was thinking more like COSE:

decfrac = [
              e10: int,
              m: integer
           ]

decfrac_tagged = #6.4(decfrac)

That way you can say exactly which you want in CDDL without constructing the CDDL for the non-tagged form. Will also help make it clear that  protocol designers need to be clear and pick one.

Maybe even ask that every new registered data type have CDDL like this published suitable for normative reference as a requirement for getting into the IANA registry?

LL

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 4 Nov 2019 17:34:17 +0100

> decfrac_tagged = #6.4(decfrac)

That kind of style was written when we didn’t have the unwrap operator.

RFC 8610 says:

                  decfrac = #6.4([e10: int, m: integer])

Which is exactly what is called “decfrac_tagged” in the above.
If you don’t want the tag, just write ~decfrac, and you have what you called decfrac in your example.

(And the fact that the unwrap convention works universally obviates the need to come up with noisy conventions such as _tagged.)

> Maybe even ask that every new registered data type have CDDL like this published suitable for normative reference as a requirement for getting into the IANA registry?

Requiring this would create too much coupling.
Suggesting this is a good job for the DE :-)

(“suitable for normative reference” is a different issue, though.)

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Wed, 6 Nov 2019 12:18:00 -0800

Thanks for all the explanations and comments and pointing out things I missed when I read CDDL.

One more observation.

Seems like this is all about defining new registered data types, very similar to a typedef in C.

For example, this is about a data type for temperature, but not indication whether the temperature is of the turkey in the oven or of the Chernobyl reactor core.  To distinguish reactor core from turkey it is by the ordering in an array or the key/label in a map, never the tag. In C it is the variable name.

LL

> On Nov 4, 2019, at 8:34 AM, Carsten Bormann <[email-masked]> wrote:
>