Subject: [Cbor] correctness of implied top level array?

From: Laurence Lundblade <[email-masked]>
Date: Sat, 23 Feb 2019 12:32:27 -0800

The current CBOR playground works like this:

    Input 0x00, (one byte) and it decodes to a single integer of value 0
    Input 0x82 0x00 0x00 (three bytes) and it decodes to an array of two zeros
    Input 0x00 0x00 (two bytes) and it decodes to a single integer with one extra byte, an error
    Similarly substitute 0x40, a zero length byte string, for 0x00 and it works the same as the above three
    Similarly substitute 0xf4, “false", for 0x00 and it works the same above three

That is, the top level of a chunk of encoded CBOR can either be a map, an array or a single non-structured data item like an integer, string or simple type. 

I think in theory you could have an implied array at the top level and allow 0x00 0x00 to decode correctly, but it is probably better not to allow this implied array. It might save someone a byte or two or three or five or nine, but I think allowing it creates an unnecessary special case. 

So 0x00, 0x00 and such should be documented as not-well-formed in the RFC and the Appendix C pseudo code should be fixed by tracking the whether the top-level is being decoded or not.

LL

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 24 Feb 2019 13:36:08 +0100

Hi Laurence,

the construction you describe is different from a CBOR data item:
It is a CBOR Sequence.  You piqued me to finally write it up, see below.

More generally speaking, an encoded CBOR data item followed by residual data is a very valid construction, for instance for a piece of metadata tacked on as a header to a binary blob.  CBOR’s self-delimiting is all you need here.
The metadata part doesn’t stop to be well-formed by being followed by that binary blob.
But of course the whole thing *is* not, but still *has* a well-formed encoded CBOR data item.

Unfortunately, not very many CBOR libraries have an API that allows isolating that metadata header from its residual data.  Maybe it’s worth writing a draft about suggested API functionality as well.  (One other thing that recently came up and is supported by too few CBOR library APIs is using an already encoded CBOR data item during encoding an array, map, or Tag.)

Grüße, Carsten

Name:		draft-bormann-cbor-sequence
Revision:	00
Title:		Concise Binary Object Representation (CBOR) Sequences
Document date:	2019-02-24
Group:		Individual Submission
Pages:		8
URL:            https://www.ietf.org/internet-drafts/draft-bormann-cbor-sequence-00.txt
Status:         https://datatracker.ietf.org/doc/draft-bormann-cbor-sequence/
Htmlized:       https://tools.ietf.org/html/draft-bormann-cbor-sequence-00
Htmlized:       https://datatracker.ietf.org/doc/html/draft-bormann-cbor-sequence

Abstract:
  This document describes the Concise Binary Object Representation
  (CBOR) Sequence format and associated media type "application/cbor-
  seq".  A CBOR Sequence consists of any number of encoded CBOR data
  items, simply concatenated in sequence.

  Structured syntax suffixes for media types allow other media types to
  build on them and make it explicit that they are built on an existing
  media type as their foundation.  This specification defines and
  registers "+cbor-seq" as a structured syntax suffix for CBOR
  Sequences.

---

From: Laurence Lundblade <[email-masked]>
Date: Tue, 26 Feb 2019 20:11:06 -0800

Hi Carsten,

So wouldn’t that mean application/cbor is required to consist of one and only one data item with no extra data? And maybe we need a section that says when you design a CBOR protocol you should be clear whether it carries a CBOR sequence or a CBOR data item. Maybe say the preferred encoding the CBOR data item because to be similar to JSON where you can’t have loose name/value pairs?

LL

> On Feb 24, 2019, at 4:36 AM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Laurence,
>

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 27 Feb 2019 09:30:38 -0700

I've always assumed that I would eventually write an XMPP-like protocol that used a stream of CBOR items without further delineation, and node-cbor reflects that, perhaps to the detriment of other use cases.  I *think* you could trick it into reading one CBOR item from a stream and stopping with the rest of the bytes still intact, but I'd have to write a few tests to be sure.

However, in my opinion both of the above cases are instances of protocols built on top of CBOR, not inherent in the CBOR design itself.

— 
Joe Hildebrand

> On Feb 26, 2019, at 9:11 PM, Laurence Lundblade <[email-masked]> wrote:
> 
> Hi Carsten,
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 27 Feb 2019 17:54:28 +0100

> However, in my opinion both of the above cases are instances of protocols built on top of CBOR, not inherent in the CBOR design itself.

Indeed, that’s why cbor-seq is a separate document (and a separate media type).

When updating CBOR, we just need to be careful not to rule out decoders that are fine to find additional data when done with the data item they were tasked to decode.  (Both for cbor-seq and for piggy-backing CBOR data in front of a binary blob in some other form.)
It took me a while to add that API to my CBOR implementation…

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Wed, 27 Feb 2019 19:53:24 -0500

Carsten Bormann <[email-masked]> wrote:
    >> However, in my opinion both of the above cases are instances of
    >> protocols built on top of CBOR, not inherent in the CBOR design itself.

    > When updating CBOR, we just need to be careful not to rule out decoders
    > that are fine to find additional data when done with the data item they
    > were tasked to decode.  (Both for cbor-seq and for piggy-backing CBOR
    > data in front of a binary blob in some other form.)

An alternative way to do the desired action (and bring this inside of cbor)
would be an (indefinite?) array type that was specified to concatenate.

I'm not really arguing for this, but I think it's worth knowing why this
would be less good a thing.

BTW: we certainly would up with "ASN1-SEQ" being an unevenly defined/implemented
     defacto-non-standard.  Certificates can sometimes be concatenated, and
     sometimes they can't, depending upon the location and the software being used.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Joe Hildebrand <[email-masked]>
Date: Thu, 28 Feb 2019 12:23:24 -0700

> On Feb 27, 2019, at 5:53 PM, Michael Richardson <[email-masked]> wrote:
> 

It's an interesting idea.  One of the reasons this didn't feel right to me was that my initial take on indefinite-length arrays was to read the whole array, growing memory as needed.  I've moved on from that, but would expect others to find themselves in a similar spot.

More interestingly though, there are times when I might want to use an optional external length-framing approach, like embedding a single CBOR data item in a WebSocket message.

— 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Feb 2019 21:46:50 +0100

> On Feb 28, 2019, at 20:23, Joe Hildebrand <[email-masked]> wrote:
> 

It is hard for a format like CBOR to take in external length information — data items need to be self-delimiting *within* the tree, so taking in that information at the top level would mean that a different encoding would be needed there.

CBOR sequences are actually exactly that, for the specific case of a top-level array.
So this is your “array type that was specified to concatenate” — except that this cannot be used within a CBOR data item ([[1, 2], [3, 4]] ≠ [[1, 2, 3, 4]]), only right on top.

Whether a single data item or a CBOR sequence is expected is indicated by meta information, such as the Content-Type.  We could define more things like CBOR maps, CBOR strings (hey, we already have ct 0 for text and 42 for bytes).  This is becoming complicated quickly.

CBOR sequences stand out because they are often exactly what is needed for streaming, and for flexible storage of partial streams in containers that can then simply be concatenated.

Next stop: Getting a CDDL spec to define top-level sequences…  (Right now, we simply describe a top-level array and add English-language information that this is to be encoded as a sequence.  Quite similar to how .cborseq works inside the spec.)

Grüße, Carsten