Subject: [Cbor] Consistent translations between JSON and CBOR

From: Burt Harris <[email-masked]>
Date: Sat, 26 May 2018 02:08:36 +0000

Let’s discuss clarifying the JSON interop goals in CBOR-BIS.

Carson raised some good points earlier, saying certain decisions had been made for consistency with JSON, and I see the value in that.   We should,  enhance section 4 of RFC 7049 dealing with this (it is currently non-normative), and the latest draft-ietf-cbor-7049bis section 5 hasn’t changed that in a meaningful way.  Both sections tackle the relationship to JSON in an ad-hock way, making testing implementations more difficult.   I think we can improve on this in the goals section of draft-ietf-cbor-7049bis as follows:

  1.  CBOR-BIS SHOULD include a normative specification for translating any valid JSON text into CBOR, and back.

  1.  The JSON -> CBOR translator SHOULD be non-ambiguous.   Any given semantically meaningful JSON text should have exactly one straightforward CBOR translation.  (CBOR compression schemes can be excluded.)

  1.  A CBOR -> JSON translator for supported JSON types MAY involve some ambiguity.  However a JSON -> CBOR -> JSON translation SHOULD introduce minimal text changes, allowing for limited following semantically-equivalent changes:

     *   Non-significant whitespace outside of JSON strings MAY change.  E. g. [ 0, 1, 2 ] could have whitespace modified to become [0,1,2].  (pretty printing lost or gained)

     *   Backslash-encoded character in quoted JSON strings MAY  change to/from UTF-8 equivalents.  The euro currency symbol in [ "₠" ] could be changed to an escape sequence [ "\u20a0" ], or vice versa.

     *   Unquoted JSON number MAY change in value-preserving ways. E. g. input of [1.0] might could become [1] in the round-tripped JSON.  Quoted numeric strings SHOULD NOT be changed, so ["1.0"] should stay unaltered.

     *   Invalid Unicode characters in JSON, weather quoted and escaped or not, may be replaced with something else, or even generate an error.   (See unpaired surrogates below if you want more information.)

     *   Unquoted non-negative numeric keys in an map MAY be sorted in ascending order and moved the front of the list of key/value pairs.   E.g.  {1: "one", 3: "three",  2: "two"} might become {1: "one",  2: "two", 3: "three" }.

  1.  The CBOR <-> JSON round-trip options (a, b, c and d above) MAY be selected by translator options, or (in the future) represented by some CBOR tag scheme to further improve the preservation of JSON text representation.

CBOR compression schemes are not “straightforward” and thus are excluded from #2 above, but they should still always round-trip from and to JSON as required by the other goals.

Preserving order of keys in round-trip translation

Limiting the JSON text changes in a round-trip scenario as specified above may be particularly useful us humans.   Consider the case where a file exists both in CBOR format and JSON, where the JSON exists so that it can be dealt with by humans and version control systems like git:

A human author may have a good reason to put entries in a JSON object in a particular order, for example putting the most “important” fields up front.   Most tools, including tools that (edit JSON in some automated way) won’t re-order the keys without good reason, and that helps the changes recorded in the version control system remain minimal.

But I’ve worked with some tools that do (unintentionally) reorder keys in a JSON object, its generally considered a bug by the users even though RFC 8259 formally says:

An object is an unordered collection of zero or more name/value
pairs, where a name is a string and a value is a string, number,
boolean, null, object, or array.

Such bugs have largely become a non-issue for tools written in JavaScript, as all modern implementations preserve the order of fields in an object, with one notable exception:   keys which are non-negative numbers always sort to the top.    I suspect the reason for the exception in JavaScript runtimes is exactly the same as that mentioned in draft-ietf-cbor-7049bis section 4.7: numeric keys are often used in cases where order is important, perhaps to normalization or performance.

Perhaps this helps explain my earlier thoughts that the default (untagged) definition of CBOR major type 5 should be treated as order-preserving at a syntax level, or perhaps even better adopt the latest ECMAscript’s spec’s description of the key-ordering behavior of objects as an alternative to the RFC 7049 concept of normalization.

Resource Constrained Systems

On a highly resource constrained system, encoding CBOR object with some specific (or consistent) ordering of fields is an advantage.   These systems will typically not use hash tables to store data.   And the most resource constrained systems may opt for numeric keys as an optimization, and could treat any keys that aren’t encoded with major type 0 as low-priority or irrelevant details, and might be able to quickly skip to the end of the map without recording the keys or values.

Unpaired UTF-16 surrogates (advanced topic)

JSON has a gray area.   RFC 8259 section 8.2 points out an ambiguity in JSON semantics to the JSON string "\uDEAD", The JSON RFC says “The behavior of software that receives JSON texts containing such values is unpredictable.”  But under the formal syntax (ABNF) of the JSON it is not explicitly forbidden.    Skip here to end if your brain is overloaded by that concept.

This backslash-prefixed sequence is technically the JSON representation of a UTF-16 surrogate.  It is a gray area because it DOES NOT represent a valid UNICODE character, there is no UTF-8 encoding it, per RFC 3629.   Thus I limit the preservation of JSON rule in (3)(e) above.   But there are a few thing we could do:

  1.  Default option:  Treat JSON strings representing unpaired surrogates as an error, since JSON doesn’t define a meaning, even though they are syntactically allowed. This is what the CBOR Playground (http://cbor.me) does, and IMHO it is a perfectly reasonable option.  The error message given by the playground for "\uDEAD" is “Expected one of [89abAB], [0-7] at line 1, column 5 (byte 5) after "\uD”, which seems technically correct.

  1.  Replace any unknown, unrecognizable or unrepresentable character, with U+FFFD � REPLACEMENT CHARACTER.  (This loose some information but it is non-fatal, but it  is exactly what the REPLACMENT CHARACTER code point is meant for.)

  1.  Leverage certain parts of this JSON grey-zone to define ways to encode CBOR datatypes not normally found in JSON.  For example, we COULD define a special interpretation of a JSON string starting with "\uDB64" to mean the following text is actually Base 64 encoded binary data, leveraging to RFC 3548!  A "\uDB16" prefix COULD mean Base 16 (hexadecimal) encoded binary.   There are literally thousands of unpaired surrogates that are possible to give meaning to in a “CBOR in JSON” context.  While not as mnemonic as CBOR’s diagnostic notation, this approach might give a way to translate any valid CBOR into syntactically correct JSON, and back!

  1.  Preserve unpaired surrogates, perhaps by defining a CBOR tag for labeling Wobbly Transformation Format  (WTF-8.)  [Not a good idea, WTF-8 is not meant for information interchange, but I include it for completeness.]

---

From: Jim Schaad <[email-masked]>
Date: Fri, 25 May 2018 21:58:52 -0700

From: CBOR <[email-masked]> On Behalf Of Burt Harris
Sent: Friday, May 25, 2018 7:09 PM
To: Carsten Bormann <[email-masked]>; [email-masked]
Subject: [Cbor] Consistent translations between JSON and CBOR

2.	The JSON -> CBOR translator SHOULD be non-ambiguous.   Any given semantically meaningful JSON text should have exactly one straightforward CBOR translation.  (CBOR compression schemes can be excluded.)

[JLS]  I assume that this means that you do not care about trying to map binary items from base64 to binary fields which is one of the major reasons that I can see for encoding JSON as CBOR.  Of course doing so without a schema is problematic.

3.	A CBOR -> JSON translator for supported JSON types MAY involve some ambiguity.  However a JSON -> CBOR -> JSON translation SHOULD introduce minimal text changes, allowing for limited following semantically-equivalent changes:

a.       Non-significant whitespace outside of JSON strings MAY change.  E. g. [ 0, 1, 2 ] could have whitespace modified to become [0,1,2].  (pretty printing lost or gained)

b.	Backslash-encoded character in quoted JSON strings MAY  change to/from UTF-8 equivalents.  The euro currency symbol in [ "₠" ] could be changed to an escape sequence [ "\u20a0" ], or vice versa.

[JLS] Do you care about the translation being syntactically the same or semantically the same. Items b-e all come out in the was if all you care about is semantics.

c.	Unquoted JSON number MAY change in value-preserving ways. E. g. input of [1.0] might could become [1] in the round-tripped JSON.  Quoted numeric strings SHOULD NOT be changed, so ["1.0"] should stay unaltered.

d.	Invalid Unicode characters in JSON, weather quoted and escaped or not, may be replaced with something else, or even generate an error.   (See unpaired surrogates below if you want more information.)

e.	Unquoted non-negative numeric keys in an map MAY be sorted in ascending order and moved the front of the list of key/value pairs.   E.g.  {1: "one", 3: "three",  2: "two"} might become {1: "one",  2: "two", 3: "three" }.

4.	The CBOR <-> JSON round-trip options (a, b, c and d above) MAY be selected by translator options, or (in the future) represented by some CBOR tag scheme to further improve the preservation of JSON text representation.

[JLS] Would you consider using a hash table for keys for efficiency to be a good reason?   Anybody relying on this outside of an absolute latest browser is probably hoping w/o reason.  It is not something that I would ever think about relying on.

[JLS] You need to look at what you believe the constraint is before you make this statement.  If the constraint is transmission time/power then the highest priority is the smallest possible encoding so that using positive and negative numbers as well as one byte text strings or binary strings because what you care about rather than sorting all of the integer values first.

2.	Replace any unknown, unrecognizable or unrepresentable character, with U+FFFD � REPLACEMENT CHARACTER.  (This loose some information but it is non-fatal, but it  is exactly what the REPLACMENT CHARACTER code point is meant for.)

3.	Leverage certain parts of this JSON grey-zone to define ways to encode CBOR datatypes not normally found in JSON.  For example, we COULD define a special interpretation of a JSON string starting with "\uDB64" to mean the following text is actually Base 64 encoded binary data, leveraging to RFC 3548!  A "\uDB16" prefix COULD mean Base 16 (hexadecimal) encoded binary.   There are literally thousands of unpaired surrogates that are possible to give meaning to in a “CBOR in JSON” context.  While not as mnemonic as CBOR’s diagnostic notation, this approach might give a way to translate any valid CBOR into syntactically correct JSON, and back!

4.	Preserve unpaired surrogates, perhaps by defining a CBOR tag for labeling Wobbly Transformation Format  (WTF-8.)  [Not a good idea, WTF-8 is not meant for information interchange, but I include it for completeness.]

[JLS] Or look at section 4.2 and not really worry about it as both formats currently support it.

Finally, the data may be invalid, such as
   invalid UTF-8 strings or date strings that do not conform to
   [RFC3339 <https://tools.ietf.org/html/rfc3339> ].

---

From: Burt Harris <[email-masked]>
Date: Sat, 26 May 2018 09:06:02 +0000

Jim,

Thank you for your feedback.   Detailed replies below your quotes follow.

Please don’t assume that, were in agreement there is a problem around binary data in JSON in need of a solution.   The problems I think implied in your response is that that there is no basis today for assuming that JSON strings are in fact binary data encoded in base64.   I guess you may talking in terms of an assumption that a JSON string may be base64 by matching some pattern, but I assert that a pattern match of normal Unicode characters alone is not a good reason to encode that string using CBOR major type 2 instead of 3.   An escape mechanism of some sort should be required.

This problem comes up because JSON (like early JavaScript) simply punted on binary data, but modern JavaScript implementations have tackled the problem with JavaScript typed arrays, like UInt8Array.    Unfortunately, JSON wasn’t written with the flexibility to deal with an expandable set of types, it was designed to be a strict subset of early JavaScript, so native JSON.parse()implementations I know of do not return a UInt8Array.

I do care about representing binary data compactly while preserving round tripping without schema, and in that spirit make the proposal later in my message to address exactly that scenario, using the admittedly obscure case of an \u escaped unpaired surrogate codepoints in JSON, in place of requiring schema knowledge or pattern matching assumptions.   JSON syntax rules support this corner case, but it is semantically undefined in JSON due to restrictions in UTF-8.   This makes this form of unpaired surrogate fertile ground for out-of-the-box thinking, and allows a number of escape mechanisms for base16, base32, base 64, and potentially other encodings if needed, all without interfering with any valid UNICODE characters.

Both, to the extent reasonably possible.   I think preserving JSON semantics seems like a MUST, and preserving JSON syntax  is a SHOULD or at least a MAY, given an well thought out definition of what is reasonably possible.

The numbered paragraphs 3 a--e are my best attempt to date to define a formal set of JSON syntax preservations exceptions that cover what is reasonably possible in a round-trip from JSON to a more compact CBOR format and back, without introducing additional CBOR tags for the purpose.

Tools like version control, compare utilities, editors, etc. simply don’t know about JSON semantics, they simply treat JSON as text, or just a raw sequence of bytes.   Even tools that understand JSON semantics tend to preserve syntax where possible, though there are obviously counter-examples.   A good example might be the NPM tool in the NodeJS ecosystem.   Recent versions may change the package.json file in the working directory, but in general it avoids reordering the keys, despite the fact key order is semantically irrelevant.   However, recent use of that tool has introduced syntactic changes like removing a line brake between the braces of a empty object, or pretty printing an array with consistent indentation.    I’ve used such behavior as a datapoint in considering what is reasonably possible in terms of preserving syntax.

I understand the question, but not sure I understand the statements that follow.   Yes, I have thought about hash tables, and I conclude think their efficacies are good reason in a round-trip conversion scenario.   I suggest there is there is no intrinsic advantage of a hash table in that scenario, because it does not call for random access.   There can be performance benefits of hash tables, but they  only occur under specific circumstances.

For many if not most scenarios on systems with cache memories, order dictionaries outperform hash tables for small numbers of entries.   Leveraging this observation is associated with JIT compiled JavaScript and the “hidden-class” optimization, and that change is not a new thing.  In Chrome and the V8 engine for example I believe it dates back to the first release in 2008.    In Node.js the key assumptions I made date all the way back to v0.9.10.   You  can use the following JavaScript expression at a console to test:

     JSON.stringify({ beta: 1, alpha: 2, 3:3, 1:1 })

My expected result:

'{"1":1,"3":3,"beta":1,"alpha":2}'
Please let me know if you find other platforms that produce a different JSON result.   Obviously really old browsers may generate an error, because native support for JSON isn’t on the really old ones, but the latest browser market share figures suggest there aren’t many people at all running ancient browsers, Chrome is clearly the market leader.   (https://yourbrowser.is/browser-market-share/2018/05/)

Good point, but if transmission bandwidth (not compute power or memory) is what you are optimizing to minimize, some form of CBOR compression will probably win out, and I specifically excluded compression in this thread (but plan to address it soon.)

IoT devices are already quite inexpensive, but costs still matter.   In the most cost-sensitive applications, the expense of a wireless radio may be significant if wired a connection is available.   Wired connections will remain competitive in some markets.   Today for example it is possible to buy a SoC with wired (I2C) processor and I2C bus repeaters cheaper than BLE radio, and reliably get longer distances.   RS-422/485,  CAN will probably remain a good option for even longer haul wires in various scenarios, but at a higher cost than rock-bottom I2C.

As an example of what I think of as a highly constrained system today, consider a PSOC 4 (CY8C4013SXI-400<http://www.cypress.com/part/cy8c4013sxi-400>), with a 16MHz ARM Coretex-M0 and I2C at up to 1 megabit/sec; and with a budgetary price in 2018 under $1/unit.    The constraints on that model are that it has only 5 GPIOs (including 2 for the I2C), 8KB of flash, and 2KB of RAM.    That’s way more bandwidth and MIPS than the first minicomputers I worked on almost 40 years ago, but no more memory.

Perhaps you can clarify this last point.   I’ve looked at section 4.2 in a number of document, I’m not sure document you mean.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 26 May 2018 13:44:06 +0200

> Please don’t assume that, were in agreement there is a problem around binary data in JSON in need of a solution.   The problems I think implied in your response is that that there is no basis today…

CBOR was designed to enable “shrinking” JSON text strings into CBOR byte strings in those cases where the text strings happen to match base64 or hex data.  Tags 21 to 23 (Section 2.4.4.2) are designed to be used on the CBOR side to indicate the binary data needs to be transformed back to text strings, and how.  These are single-byte (1+0) tags because we expected them to be used frequently in JSON-to-CBOR converters.

Without schema information, simply trying to recognize base64 (or base16) poses a danger that any other text string will look like it is base64/16-encoded.  E.g., “beef” and “unshrinkable” might turn into 23(h’beef’) and 21(h’ba7b21ae29e469b95e’) (well, the latter example only saves a couple of bytes, but larger ones of course save more)…  It is therefore important to define a unique transformation back into a text string so encoders can check that there will be no change.  Also, software that processes the data on the CBOR side needs to be able to handle Tag-21 encoded strings where text strings were expected.

Matching strings against some pattern and encoding them in binary could also be used for other data such as UUIDs (tag 37).  Here again, for a schema-less application, we lack instructions on how exactly to convert tag 37 back into text strings (which is also needed by the JSON-to-CBOR converter to check whether the conversion to binary is safe).  This is a bit more questionable, as it overloads the semantic tag 37 “this is a UUID” by an alternate meaning “this could be a UUID syntactically”.  We also have been thinking about defining additional such tags for common information in YANG-CBOR, such as IP addresses, which would then also be available for JSON conversion.

With schema information, all this can be done more cleanly (e.g., see how SenML handles “data values” in CBOR vs. in XML and JSON).

In any case, all the above is illustration for why I don’t think there is a single “right” conversion from JSON to CBOR.  Still, to enable generic implementations of that conversion, it would be good to provide some guidelines.

Grüße, Carsten

---

From: Burt Harris <[email-masked]>
Date: Mon, 28 May 2018 23:01:49 +0000

As I understand it, Carsten is saying that CBOR today reserves tag 21-23 for CBOR -> JSON translators.   That might be good for debugging, but there is no safe and standard approach to go from JSON -> CBOR on the round trip if the original CBOR contained binary because JSON doesn't have syntax or binary or schema.   JSON string Heuristics like matching regular expressions for base16, base32, or base64 could make simple examples like `"beef"` ambiguous as to if they are binary or text, and if binary, which format!   The tags can't help that, right Carsten?

That make sense to me.   I know, YAML allows (but does not require) similar pattern matching, but it only applies to unquoted strings, and JSON doesn't have unquoted strings.   YAMLs pattern matching solution may not be perfect, but there is an alternative, `!!binary` means base64.   Did you consider using a YAML serializer instead of making up CBOR diagnostic language?

Microsoft Interview Tactics...

Below is the sort of thing we used to dream up for interviews at Microsoft when I worked there.  You might find it fun.   A true consultant's answer might be "it depends", which is correct, but not as much as someone might imagine.   Can anyone out there make note of surprising answers from running it in a particular environment? I would be quite interesting to hear about, and I think it may be relevant to this discussion:

I was most surprised on Windows 10 Node.js (any version) at the output rendering of the colon character in the value associated with key 3.   Does anyone else see something funny?  Care to explain?

But re. CBOR:  should sending this same data structure through CBOR serialization & deserialization work?  Without changing the output?

```
// JavaScript/JSON consultant's test: What's the output?  Extra points: try without running it.

var isThisValid = {
  "Q": "Is this valid JavaScript?",
  1: "dead beef",
  2: "\udead \ubeef",
  3: "\uDB16: dead beef",
  0: "No offense intended.",
  // "CBOR diagnostic notation won't work": h'dead beef', 
  "A": "Yes, but what does it all mean?"
};

function log(x) { console.log(x||""); }

log();
log( JSON.stringify(isThisValid) );
log();
log( isThisValid );
log();
log( isThisValid[3].charCodeAt(0).toString(16)  )
log( isThisValid[3].codePointAt(0).toString(16)  )
log();
```

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 29 May 2018 08:06:39 +0200

Hi Burt,

lots of interesting questions.  Let me try addressing them in pieces.

On May 29, 2018, at 01:01, Burt Harris <[email-masked]> wrote:
> 
> Did you consider using a YAML serializer instead of making up CBOR diagnostic language?

Absolutely.  I personally use YAML as the diagnostic notation for my CBOR-based work(*): e.g., when I open a binary CBOR file in Emacs, it is automatically converted to YAML (and back when writing out).  Section 6 of RFC 7049 even recommends looking at YAML.  However, YAML would have been one more thing to learn for people who want to move on from JSON to CBOR, so we thought relying on it too much would create more of a threshold than we liked.  (YAML also is not a formal standard that has ongoing maintenance and a formal reference that would be immediately acceptable as a normative reference in an RFC.)  So the official diagnostic notation of CBOR, as defined in Section 6 of RFC 7049, is based on JSON and extends that, e.g., with binary data and tags.

Grüße, Carsten

(*) Actually, I usually also look at JSON files after converting them to the more readable YAML first :-)