Subject: [Json-canon] Support for a WG

From: "Struse, Richard J." <[email-masked]>
Date: Fri, 18 Jan 2019 02:50:00 +0000

A standardized deterministic canonicalization of JSON data streams is essential and as such I support the formation of a WG to tackle this problem.

Thank you.

Richard J. Struse

Co-chair, OASIS Cyber Threat Intelligence Technical Committee
Chief Strategist for Cyber Threat Intelligence
The MITRE Corporation
+1-703-983-7049 (office)
+1-703-342-8368 (mobile)

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Jan 2019 07:46:28 +0100

On Jan 18, 2019, at 03:50, Struse, Richard J. <[email-masked]> wrote:
> 
> A standardized deterministic canonicalization of JSON data streams is essential

For what?

I think we would all benefit if you could explain your use case.

Grüße, Carsten

---

From: Bret Jordan <[email-masked]>
Date: Fri, 18 Jan 2019 18:40:15 +0100

I fully support the creation of a WG for JSON canonicalization as well.  

Bret 

Sent from my Commodore 128D

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

> On Jan 18, 2019, at 3:50 AM, Struse, Richard J. <[email-masked]> wrote:
> 
> -- 
> json-canon mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json-canon

---

From: Samuel Erdtman <[email-masked]>
Date: Mon, 21 Jan 2019 22:17:57 +0100

I too support the creation of a WG for JSON canonicalization.

To me the obvious use case is signing of cleartext JSON I know that is
controversial for some reason. To me it is just another tool to use. I know
the argument for not wanting to base64url encode a json-payload has not hit
home everywhere most often since it is "want" (and not need). To me there
is never (or seldom) an absolute need for a new schema e.g. one could still
do ASN1 signatures or XMLDigSig but many preferable to do a JOSE signature
and I think that to some it will be preferable to do a cleartext JSON
signature and for those JSON canonicalization is needed.

Cheers
//Samuel

On Fri, Jan 18, 2019 at 6:40 PM Bret Jordan <[email-masked]> wrote:

> A standardized deterministic canonicalization of JSON data streams is
> essential and as such I support the formation of a WG to tackle this
> problem.
>
>
>
> Thank you.
>
>
>
> *Richard J. Struse*
>
>
>

---

From: "Struse, Richard J." <[email-masked]>
Date: Tue, 22 Jan 2019 12:31:10 +0000

The use case is to enable other standards that use a JSON serialization to be able to count on two objects, each with the same contents, having the same hash value based on their representation in JSON.

Does that help?

On 1/18/19, 1:47 AM, "Carsten Bormann" <[email-masked]> wrote:

    For what?

    Grüße, Carsten

---

From: Allan Thomson <[email-masked]>
Date: Tue, 22 Jan 2019 15:19:05 +0000

Particularly important in what Rich says is also is to ensure the same hash is created where those JSON objects have optional or empty/missing properties that are defined in the schema of the object.

Regards

Allan

On 1/22/19, 4:31 AM, "json-canon on behalf of Struse, Richard J." <[email-masked] on behalf of [email-masked]> wrote:

    Does that help?

        
        For what?

        Grüße, Carsten

---

From: Samuel Erdtman <[email-masked]>
Date: Tue, 22 Jan 2019 17:37:37 +0100

I do not think we should go into schemas, that is one of the things that
makes XMLDigSig inconvenient. But instead keep to simple JSON stringifying
with predictable result.
On Tue, 22 Jan 2019 at 16:19, Allan Thomson <[email-masked]>
wrote:

> Particularly important in what Rich says is also is to ensure the same
> hash is created where those JSON objects have optional or empty/missing
> properties that are defined in the schema of the object.
>
> Regards
>
> Allan
>
> On 1/22/19, 4:31 AM, "json-canon on behalf of Struse, Richard J." <
> [email-masked] on behalf of [email-masked]> wrote:
>
>     The use case is to enable other standards that use a JSON
> serialization to be able to count on two objects, each with the same
> contents, having the same hash value based on their representation in JSON.
>
>     Does that help?
>

---

From: Allan Thomson <[email-masked]>
Date: Tue, 22 Jan 2019 16:47:25 +0000

I’m not sure how you can create a consistent hash value without considering how empty/optional fields in a JSON structure are treated consistently across implementations.

So if the same optional/empty fields exist from 2 different vendors/instances then do they end up with the same hash?

Allan

From: Samuel Erdtman <[email-masked]>
Date: Tuesday, January 22, 2019 at 8:37 AM
To: Allan Thomson <[email-masked]>
Cc: "Struse, Richard J." <[email-masked]>, Carsten Bormann <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Json-canon] [EXT] Re: Support for a WG

I do not think we should go into schemas, that is one of the things that makes XMLDigSig inconvenient. But instead keep to simple JSON stringifying with predictable result.
On Tue, 22 Jan 2019 at 16:19, Allan Thomson <[email-masked]<mailto:[email-masked]>> wrote:
Particularly important in what Rich says is also is to ensure the same hash is created where those JSON objects have optional or empty/missing properties that are defined in the schema of the object.

Regards

Allan

On 1/22/19, 4:31 AM, "json-canon on behalf of Struse, Richard J." <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

    Does that help?

    On 1/18/19, 1:47 AM, "Carsten Bormann" <[email-masked]<mailto:[email-masked]>> wrote:

        For what?

        Grüße, Carsten

    --
    json-canon mailing list
    [email-masked]<mailto:[email-masked]>
    https://www.ietf.org/mailman/listinfo/json-canon

--
json-canon mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/json-canon

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 22 Jan 2019 19:05:39 +0100

Hi Richard,

> On Jan 22, 2019, at 13:31, Struse, Richard J. <[email-masked]> wrote:
> 

Well, that is the technical requirement, but what do you use this property for?

JSON was discovered/presented around 2002 and so far has worked quite well without this property.  What use cases have sprung up that now need it?

I’m asking because for many use cases I have heard, there actually are better technical approaches than canonicalization.  Having said that, CBOR, a data format defined in 2013, does have recommendations for deterministic encoding (“canonicalization”), which turned out to be useful for instance for test scripts.  One thing we found out quickly is that deterministic encoding is dependent on the application concept of equality, so it is hard to define a single variant of it for a serialization used in multiple applications.  One more reason to find out about use cases.

Grüße, Carsten

---

From: Bret Jordan <[email-masked]>
Date: Wed, 23 Jan 2019 10:27:37 +0100

Carsten,

I do not see what the hang up is for doing this. There is a document that proposes a very nice solution.  There are libraries for various programming languages that can do this.  What we need is a standard that people can reference when they want to use it.  At the end of the day people, organizations, vendors, and groups need a way of dealing with white space, ordering, and the JSON Number type in a consistent way.  They do not want to use CBOR, ASN.1, XML, or anything else.  The market wants to use plain and simple JSON. Right now we have various groups and organizations going off and doing their own form of this. It will save a lot of pain if we could just, finally, standardize this. 

Lets not make this overly complicated.  Organizations, vendors, other standards, and other working groups want the ability to use JSON “as-is” and perform security operations on that string of data.  To do this, we need a canonical string of data that can be consistently generated.   

Thanks,
Bret
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

> On Jan 22, 2019, at 7:05 PM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Richard,
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 23 Jan 2019 11:32:28 +0100

Hi Bret,

I’ve been in your shoes.
“It’s not that complicated.”  Yeah.

Allan already hinted at one possible complication.
You can of course declare that (very real!) issue out of scope.
But that is easier done if we know why we are doing this.

What’s so difficult about actually saying what the use cases are?
I’m not trying to suggest there needs to be an RFC that documents every single use case before there can be any work on the solution (I’m saying this because unfortunately we have had process hangups like this before and there simply is no need for this).
Instead, this mailing list is a great place to have some discussion on the use cases so the requirements are better understood.
“We want this because some people are telling us it’s easy” is not the level of discussion that can help with this.

On the solution side, having a solution but not having a well-understood description of what it does and what it does not is dangerous.  Even more so when some of the use cases have security considerations.

After all, the IETF would not be doing this solution as an ultimate goal — the point is that other protocols can build on top of this solution, and as we want to achieve interoperability (and maintain security), we need to know what the solution does and what requirements it places on its usage to actually achieve its desired properties.

Grüße, Carsten

> On Jan 23, 2019, at 10:27, Bret Jordan <[email-masked]> wrote:
> 
> Carsten,
>

---

From: Bret Jordan <[email-masked]>
Date: Wed, 23 Jan 2019 11:47:56 +0100

Carsten,

Here are two examples…

1) Creating deterministic IDs for STIX Cyber Observables.  STIX Cyber Observables are simple JSON blobs (think network traffic, file data, email data, etc) of data and we define for each object what properties should be used for the ID.  And when you build out your graph, you do not want a million vertices for the same thing.  You want to be able to just relate things together in a deterministic way. 

So say you have something super super simple like this file object:

{
  "type": "file",
  "hashes": {
    "SHA-256": "fe90a7e910cb3a4739bed9180e807e93fa70c90f25a8915476f5e4bfbac681db"
  },
  "size": 25536,
  "name": "foo.dll"
}

You would want to be able to treat this as a string that you could pipe in to some hashing algorithm.  So you would want something like this:

Id := Sha256hash(<JSON DATA>)

2) Creating digital signatures for JSON objects using the JSON data “as-is”.  

Thanks,
Bret

> On Jan 23, 2019, at 11:32 AM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Bret,
> 
> I’ve been in your shoes.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 23 Jan 2019 13:22:48 +0100

Hi Bret,

> 1) Creating deterministic IDs for STIX Cyber Observables.  

Now that is very concrete!  Thank you.
What would be the best reference I could use to learn how STIX uses JSON?

> 2) Creating digital signatures for JSON objects using the JSON data “as-is”.  

Right, that is one very general use case, but as Allan and I hinted, it may be worth looking at the JSON data in a bit more detail.  Using Richard’s problem statement:

“two objects, each with the same contents, having the same hash value based on their representation in JSON.”

This has this gaping hole what “same contents” actually mean.
So far, I-JSON (RFC 7493) is the closest we have to a specification of the generic data model with which we are using JSON as an interchange format.
Maybe that is all we need, but that remains to be verified — best in the context of the applications that actually have a concept of what “same contents” would be.

So I still think analyzing a few applications would be a good thing before we can claim the generic use case (2) is covered.

Grüße, Carsten

---

From: Bret Jordan <[email-masked]>
Date: Wed, 23 Jan 2019 14:09:08 +0100

I would probably point you to this site that we put together. However, if there is something specific you would like to know, I can answer that.  Or we can schedule a WebEx and I can walk you through it. 

https://oasis-open.github.io/cti-documentation/ <https://oasis-open.github.io/cti-documentation/>

Thanks,
Bret

> On Jan 23, 2019, at 1:22 PM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Bret,
>

---

From: Bret Jordan <[email-masked]>
Date: Wed, 23 Jan 2019 14:10:35 +0100

Another use-case I have is for the digital signatures that we need for CACAO, and WG that we are trying to get setup in the IETF.  I have some slides that talk about the layers of digital signatures that we need to be able to do for that work. I can share them with you, or we can have a call and I can walk you through it. 

Thanks,
Bret

---

From: Samuel Erdtman <[email-masked]>
Date: Thu, 24 Jan 2019 07:17:21 +0100

Thanks for the followup Allan.

In my opinion the application layer schemas for defining what the JSON can
look like (optional values different values meaning the same thing etc),
should be verified by application layer. I would like to keep the
canonicalization (and potential signatures) at the level where they say the
JSON has not changed.

When it comes to empty empty values I think those are values too, they
might convey some meaning to the application layer.and with empty I mean
{"empty": ""}. And since I do not want the canonicalization to be involved
with application level details optional attributes are not a concept.

Does that make sense?

Cheers
//Samuel

On Tue, Jan 22, 2019 at 5:47 PM Allan Thomson <
[email-masked]> wrote:

> I’m not sure how you can create a consistent hash value without
> considering how empty/optional fields in a JSON structure are treated
> consistently across implementations.
>
>
>
> So if the same optional/empty fields exist from 2 different
> vendors/instances then do they end up with the same hash?
>
>
>
> Allan
>
>
>
> *From: *Samuel Erdtman <[email-masked]>
> *Date: *Tuesday, January 22, 2019 at 8:37 AM
> *To: *Allan Thomson <[email-masked]>
> *Cc: *"Struse, Richard J." <[email-masked]>, Carsten Bormann <[email-masked]>,
> "[email-masked]" <[email-masked]>
> *Subject: *Re: [Json-canon] [EXT] Re: Support for a WG
>
>
>
> On Tue, 22 Jan 2019 at 16:19, Allan Thomson <
> [email-masked]> wrote:
>

---

From: Samuel Erdtman <[email-masked]>
Date: Thu, 24 Jan 2019 08:13:29 +0100

Carsten, Here comes a few another concrete example where canonical JSON
would be beneficial (not an absolute need).

Open Banking (
https://openbanking.atlassian.net/wiki/spaces/DZ/pages/5785171/Account+and+Transaction+API+Specification+-+v1.1.0#AccountandTransactionAPISpecification-v1.1.0-Processforsigningapayload
)
In this case RFC 7797 is used to keep the payload separated form the
signature and headers.
If you look at the processing steps the HTTP body (josn) is Base64 encoded
and used as payload for the JWS calculation, it is however not the base64
encoded payload that is sent in the HTTP body but the cleartext json (as
can be seen in the verification steps). This is a workable but brittle
solution. With a canonical JOSN format this solution could be made stable.
(I have no personal contact with this group so I do not know if they
formally would be interested, but it is a case here cleartext JSON is
signed)

Then there are the TEEP usecase (
https://tools.ietf.org/id/draft-ietf-teep-opentrustprotocol)
In this case JOSE is used and there is several layers of signing (and
encryption) being done, since the payload is always base64 encode and then
wrapped the messages becomes very opaque. So to me they have a workable
solution but would benefit from canonical JOSN in combination with RFC
7797. (Once again I have no personal contact with this group so I do not
know if they formally would be interested)

Then there is the Signing HTTP Messages case (
https://tools.ietf.org/html/draft-cavage-http-signatures)
This one is a bit more far fetched, but instead of defining how to
canonicalize the HTTP headers they could do a simple mapping form https
headers to JOSN and have it canonicalized and then signed using RFC 7797.
(Once again I have no personal contact with this group so I do not know if
they formally would be interested)

Best regards
//Samuel