Subject: [Cbor] Chunks with tags inside indefinite-length string (major type 2 and 3)

From: Faye Amacker <[email-masked]>
Date: Thu, 5 Dec 2019 09:27:17 -0600

While implementing support for tags, I ran into a scenario that might be
errata or could use some clarification in 7049bis.

Section 3.2.3 states:

> Indefinite-length strings are represented by a byte containing the major
type and additional information value of 31, followed by a series of zero
or more byte or text strings (“chunks”) that have definite lengths,
followed by the “break” stop code (Section 3.2.1). The data item
represented by the indefinite-length string is the concatenation of the
chunks (i.e., the empty byte or text string, respectively, if no chunk is
present).

And the pseudocode in Appendix C allows tags for chunks inside
indefinite-length strings.

Chunks are simply fragments to be concatenated, so tags applied to chunks
doesn't seem intuitive.  Chunks are not intended for independent access
like array elements.  Some tags (like #2 and #3) transform byte string into
bignum which makes sense for arrays, not chunks.

If tags must be applied to each chunk, there should be some text mentioning
that because library authors might think it should be rejected or applied
to the concatenated string rather than chunk.

However, if the tag for a chunk applies to the entire concatenated string,
then what happens when there are multiple chunks with different tags? Which
tag wins?

A simpler way forward is to treat tags on chunks as malformed.  If this is
the way to go, then the pseudocode in Appendix C needs to be updated and an
example could be added to Appendix G.

GitHub issue for RFC 7049bis: https://github.com/cbor-wg/CBORbis/issues/148
GitHub issue for my CBOR library:
https://github.com/fxamacker/cbor/issues/44

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 5 Dec 2019 11:18:17 -0800

I *think*
https://cbor-wg.github.io/CBORbis/draft-ietf-cbor-7049bis.html#indefinite-length-byte-strings-and-text-strings
is
pretty clear that the contained things have to have the same major type as
the indefinite-length string, and tags have a different major type. e.g. "a
series of zero or more byte or text strings" doesn't include the
possibility of tags, and "If any item between the indefinite-length string
indicator (0b010_11111 or 0b011_11111) and the “break” stop code is not a
definite-length string item of the same major type, the string is not
well-formed." says any other content is not well-formed.

The pseudocode includes

if (it != mt)           // need finite-length chunk
          fail();               //    of same type

which I think also rejects tags inside indefinite strings. What am I
missing?

Jeffrey

On Thu, Dec 5, 2019 at 7:27 AM Faye Amacker <[email-masked]> wrote:

> If tags must be applied to each chunk, there should be some text
> mentioning that because library authors might think it should be rejected
> or applied to the concatenated string rather than chunk.
>
> GitHub issue for RFC 7049bis:
> https://github.com/cbor-wg/CBORbis/issues/148
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor
>

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 5 Dec 2019 13:56:47 -0800

I believe that's correct, with the caveat that protocols should be able to
specify that they accept a "number", which would accept primitive integers,
primitive floats, and tagged bigints, bigfloats, bigdecimals, and any new
tags defined to represent numbers.
https://cbor-wg.github.io/CBORbis/draft-ietf-cbor-7049bis.html#epochdatetimesect
has
some wording to restrict to just primitive integers.

Jeffrey

On Thu, Dec 5, 2019 at 1:47 PM Faye Amacker <[email-masked]> wrote:

> Jeffrey, I think you are correct.  It seems I misread a line in the
> pseudocode which made me doubt my interpretation of the text.
>
> Can you confirm that decoders (both generic and non-generic) should:
> * reject a tagged positive number if a protocol specifies a positive
> number.
> * reject a positive number if a protocol specifies a tagged positive
> number.
> * accept both positive number and tagged positive number only if the
> protocol specifies the specific tag is optional.
>
> Thanks again for your help.  I'm relieved tags on chunks are malformed.
>
> On Thu, Dec 5, 2019 at 1:18 PM Jeffrey Yasskin <[email-masked]>
> wrote:
>
>> I *think*
>> https://cbor-wg.github.io/CBORbis/draft-ietf-cbor-7049bis.html#indefinite-length-byte-strings-and-text-strings is
>>> > Indefinite-length strings are represented by a byte containing the
>>> major type and additional information value of 31, followed by a series of
>>> zero or more byte or text strings (“chunks”) that have definite lengths,
>>> Chunks are simply fragments to be concatenated, so tags applied to
>>> chunks doesn't seem intuitive.  Chunks are not intended for independent
>>> access like array elements.  Some tags (like #2 and #3) transform byte
>>> string into bignum which makes sense for arrays, not chunks.
>>>
>>> However, if the tag for a chunk applies to the entire concatenated
>>> string, then what happens when there are multiple chunks with different
>>> tags? Which tag wins?
>>>
>>> A simpler way forward is to treat tags on chunks as malformed.  If this
>>> is the way to go, then the pseudocode in Appendix C needs to be updated and
>>> an example could be added to Appendix G.
>>>

---

From: Laurence Lundblade <[email-masked]>
Date: Fri, 6 Dec 2019 10:06:22 -0800

I agree that chunks in indefinite length strings should not be tagged. I think a decoder can aggregate the parts of an indefinite length string into what looks like a definite length string to the next layer up before processing tags.

It would be good to have some of these in the not-well-formed CBOR appendix. Here’s three:
    // Indefinite length byte string with tagged 2nd chunk
    {(uint8_t[]){0x5f, 0x41, 0x00, 0xc3, 0x42, 0x00, 0x00, 0xff}, 8}
    // Indefinite length byte string with tagged chunk
    {(uint8_t[]){0x5f, 0xc2, 0x41, 0x00, 0xff}, 5}
    // Indefinite length text string with tagged chunk
    {(uint8_t[]){0x7f, 0xc1, 0x61, 0x00, 0xff}, 5}

I think your three rules about tagging are about right (applied to any tagged item, not just to positive (big) numbers). CDDL allows you to specify any of the three. It even gives an example of the third one. See my_uri = #6.32(tstr) / tstr in section 3.6, though I hear we want to discourage that third rule.

One reason for restricting epoch date to a few types is to make it more interoperable. Smaller decoders will not support big nums, decimal fractions and such. Also, these larger types are not particularly needed for dates (unless you wish to date dinosaur bones to the microsecond).

LL

> On Dec 5, 2019, at 1:56 PM, Jeffrey Yasskin <[email-masked]> wrote:
> 
> I believe that's correct, with the caveat that protocols should be able to specify that they accept a "number", which would accept primitive integers, primitive floats, and tagged bigints, bigfloats, bigdecimals, and any new tags defined to represent numbers. https://cbor-wg.github.io/CBORbis/draft-ietf-cbor-7049bis.html#epochdatetimesect <https://cbor-wg.github.io/CBORbis/draft-ietf-cbor-7049bis.html#epochdatetimesect> has some wording to restrict to just primitive integers.
> 
> Jeffrey
> 
> On Thu, Dec 5, 2019 at 1:47 PM Faye Amacker <[email-masked] <mailto:[email-masked]>> wrote:
> Jeffrey, I think you are correct.  It seems I misread a line in the pseudocode which made me doubt my interpretation of the text.
> 
> * reject a tagged positive number if a protocol specifies a positive number.
> * reject a positive number if a protocol specifies a tagged positive number.
> * accept both positive number and tagged positive number only if the protocol specifies the specific tag is optional.
> 
> On Thu, Dec 5, 2019 at 1:18 PM Jeffrey Yasskin <[email-masked] <mailto:[email-masked]>> wrote:
> I *think* https://cbor-wg.github.io/CBORbis/draft-ietf-cbor-7049bis.html#indefinite-length-byte-strings-and-text-strings <https://cbor-wg.github.io/CBORbis/draft-ietf-cbor-7049bis.html#indefinite-length-byte-strings-and-text-strings> is pretty clear that the contained things have to have the same major type as the indefinite-length string, and tags have a different major type. e.g. "a series of zero or more byte or text strings" doesn't include the possibility of tags, and "If any item between the indefinite-length string indicator (0b010_11111 or 0b011_11111) and the “break” stop code is not a definite-length string item of the same major type, the string is not well-formed." says any other content is not well-formed.
> 
> The pseudocode includes
> 
> which I think also rejects tags inside indefinite strings. What am I missing?
> 
> Jeffrey
> 
> On Thu, Dec 5, 2019 at 7:27 AM Faye Amacker <[email-masked] <mailto:[email-masked]>> wrote:
> While implementing support for tags, I ran into a scenario that might be errata or could use some clarification in 7049bis.
> 
> Section 3.2.3 states:
> 
> > Indefinite-length strings are represented by a byte containing the major type and additional information value of 31, followed by a series of zero or more byte or text strings (“chunks”) that have definite lengths, followed by the “break” stop code (Section 3.2.1). The data item represented by the indefinite-length string is the concatenation of the chunks (i.e., the empty byte or text string, respectively, if no chunk is present).
> 
> And the pseudocode in Appendix C allows tags for chunks inside indefinite-length strings.
> 
> Chunks are simply fragments to be concatenated, so tags applied to chunks doesn't seem intuitive.  Chunks are not intended for independent access like array elements.  Some tags (like #2 and #3) transform byte string into bignum which makes sense for arrays, not chunks.
> 
> If tags must be applied to each chunk, there should be some text mentioning that because library authors might think it should be rejected or applied to the concatenated string rather than chunk.
> 
> However, if the tag for a chunk applies to the entire concatenated string, then what happens when there are multiple chunks with different tags? Which tag wins?
> 
> A simpler way forward is to treat tags on chunks as malformed.  If this is the way to go, then the pseudocode in Appendix C needs to be updated and an example could be added to Appendix G.
> 
> GitHub issue for RFC 7049bis: https://github.com/cbor-wg/CBORbis/issues/148 <https://github.com/cbor-wg/CBORbis/issues/148>
> GitHub issue for my CBOR library: https://github.com/fxamacker/cbor/issues/44 <https://github..com/fxamacker/cbor/issues/44>_______________________________________________
> CBOR mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/cbor <https://www.ietf.org/mailman/listinfo/cbor>

---

From: [email-masked] (Dale R. Worley)
Date: Mon, 09 Dec 2019 20:46:42 -0500

> indefinite-length strings.

It seems to me that the safe option, the one that will cause the least
trouble in the future, is to define tags on items inside
indefinite-length structure as syntactically well-structured but
semantically having no defined meaning.  Actually, that's the same as we
treat use of tags whose values haven't been defined.

Functionally, that means that no current application should generate
them (and thus an application for the same universe of values should
expect to never see them), but "generic" decoders should be prepared to
see such tags.

Dale

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 10 Dec 2019 06:03:44 +0100

>> indefinite-length strings.

As Jeffrey noted, the pseudocode actually catches chunks that are of a different major type than the indefinite string they are contained in.

> semantically having no defined meaning.  

Why would we allow tags here?  Indefinite strings are a very limited syntactical structure; there is no need to allow flexibility here (in particular if there is no semantic gain from that).

> Actually, that's the same as we
> treat use of tags whose values haven’t been defined.

I don’t follow.

> expect to never see them), but "generic" decoders should be prepared to

I don’t think so.  Both the text and the pseudocode disallow tags on chunks.

Grüße, Carsten

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 17 Dec 2019 22:33:53 -0500

Carsten Bormann <[email-masked]> writes:
> Why would we allow tags here?  Indefinite strings are a very limited
> syntactical structure; there is no need to allow flexibility here (in
> particular if there is no semantic gain from that).

The reason I argue is that having worked with SIP, I've become very
respectful of extensibility.  SIP has taken over the internals of every
major telecom company, it seems.  And that doesn't seem to be because
SIP is wonderfully designed; it's sort of a running joke to list the
things one would change in the next version of SIP if it didn't have to
be upward-compatible.  But SIP has a number of very general
extensibility mechanisms that make it possible to implement almost any
telecomm feature in an upward-compatible way.  And there are dozens if
not hundreds of SIP extensions.

And while I cannot guess what use might be made of tagging chunks of
indefinite-length strings, it seems to me that as long as the
fundamental structure of CBOR allows such a thing to be constructed
unambiguously, the wisest course of action is to make it clear that such
constructs can sensibly be made, but the current standard assigns no
meaning to them.

Dale

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 18 Dec 2019 06:44:52 +0100

Hi Dale,

I completely agree that extensibility (and, more generally, evolvability) is what can make or break a protocol.  One of the big differences between JSON and CBOR is that JSON was designed to be “stable” (fixed, non-extensible), while CBOR has well-defined extension points.

Indefinite length strings are a syntactic feature of CBOR that was not designed to be extensible.  I have a hard time imagining why one would want to extend it.  I don’t think this is proof by lack of imagination, either; this mechanism already has more complexity than one would reasonably like; it just happens to be needed.  There are no data model extensions that would need to extend this mechanism.  If there ever is a need to extend it, this would be much like using additional information values 28..30: A revision of the standard, a new version of the format.  For the present format, indefinite length strings of major type N (2 or 3) are composed of definite length strings of the same major type, only.

Grüße, Carsten

> On Dec 18, 2019, at 04:33, Dale R. Worley <[email-masked]> wrote:
>