Subject: [jose] Possible change to protected field

From: "Jim Schaad" <[email-masked]>
Date: Sun, 9 Jun 2013 22:46:22 -0700

<no hat>

I am trying to figure out if I am missing something.  This is not yet a
formal proposal to actual change the document.

I was thinking about proposing that we make a change to the content of the
protected field in the JWS JSON serialization format.  If we encoded this as
a UTF8 string rather than the base64url encoded UTF8 string, then the
content would be smaller.  The computation of the signature would be
unchanged in that it would still be computed over the base64url encoded
string.  I believe that the conversion from the UTF8 string to the base64url
encoded UTF8 string is a deterministic encoding and thus would not generate
any problems from that point.

At this point I and trying to figure out if I missed anything that would
preclude this from working.  I am not worried about how hard or easy it
would be to do, just if it is even possible.

Jim

---

From: Richard Barnes <[email-masked]>
Date: Mon, 10 Jun 2013 12:28:31 -0400

This sounds like a fine idea to me.  It saves space and makes the JSON
format more human-readable.  It actually makes kind of a nice analogy to
ASN.1, namely use of OCTET STRING to encapsulate more DER content.

The compact serialization can continue to base64url-encode that field, so
it would not be a breaking change for that serialization.

--Richard

On Mon, Jun 10, 2013 at 1:46 AM, Jim Schaad <[email-masked]> wrote:

> <no hat>****
> ** **
> formal proposal to actual change the document.****
>
> ** **
>
> protected field in the JWS JSON serialization format.  If we encoded this
> as a UTF8 string rather than the base64url encoded UTF8 string, then the
> string.  I believe that the conversion from the UTF8 string to the
> base64url encoded UTF8 string is a deterministic encoding and thus would
> not generate any problems from that point.****
>
> ** **
>
> would be to do, just if it is even possible.****
>
> ** **
>
> Jim****
>
> ** **
>
> _______________________________________________
> jose mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jose
>
>

---

From: Richard Barnes <[email-masked]>
Date: Mon, 10 Jun 2013 12:31:42 -0400

On second thought: It seems like the signature should actually be over the
UTF-8 octets, rather than the base64 octets.  To reason by analogy: For
every other field in a JWE, you decode the field before using it in a
cryptographic computation.  So it should be the same with the protected
header.

--Richard

On Mon, Jun 10, 2013 at 12:28 PM, Richard Barnes <[email-masked]> wrote:

> On Mon, Jun 10, 2013 at 1:46 AM, Jim Schaad <[email-masked]>wrote:
>
>> <no hat>****
>>
>> ** **
>>
>> I was thinking about proposing that we make a change to the content of
>> the protected field in the JWS JSON serialization format.  If we encoded
>> this as a UTF8 string rather than the base64url encoded UTF8 string, then
>> the content would be smaller.  The computation of the signature would be

---

From: Mike Jones <[email-masked]>
Date: Mon, 10 Jun 2013 16:38:42 +0000

The problem with this suggested change is that it requires unencoded JSON objects to be transmitted with no transformations whatsoever, whereas the problems with this assumption are well known.  For starters, on UNIX-based systems, newlines are typically represented by a bare LF character, whereas on DOS-based systems, newlines are typically represented by a CRLF pair.  In transmission, many systems, including mail agents tend to convert newlines to the system's normal format.  This would break these JOSE objects.

Some agents wrap lines after a certain length.  Particularly when being transmitted in HTML or XML, many systems replace two or more consecutive spaces with a single space.  Others replace two or more spaces with a single space and N-1 non-breaking space characters.  I could go on...

I appreciate the attempt to make things appear to be more uniform and readable, but the CRLF/LF problems alone are enough to make this a non-starter.

                                                                Best wishes,
                                                                -- Mike

From: [email-masked] [mailto:[email-masked]] On Behalf Of Richard Barnes
Sent: Monday, June 10, 2013 9:29 AM
To: Jim Schaad
Cc: [email-masked]
Subject: Re: [jose] Possible change to protected field

This sounds like a fine idea to me.  It saves space and makes the JSON format more human-readable.  It actually makes kind of a nice analogy to ASN.1, namely use of OCTET STRING to encapsulate more DER content.

The compact serialization can continue to base64url-encode that field, so it would not be a breaking change for that serialization.

--Richard

On Mon, Jun 10, 2013 at 1:46 AM, Jim Schaad <[email-masked]<mailto:[email-masked]>> wrote:
<no hat>

I am trying to figure out if I am missing something.  This is not yet a formal proposal to actual change the document.

I was thinking about proposing that we make a change to the content of the protected field in the JWS JSON serialization format.  If we encoded this as a UTF8 string rather than the base64url encoded UTF8 string, then the content would be smaller.  The computation of the signature would be unchanged in that it would still be computed over the base64url encoded string.  I believe that the conversion from the UTF8 string to the base64url encoded UTF8 string is a deterministic encoding and thus would not generate any problems from that point.

At this point I and trying to figure out if I missed anything that would preclude this from working.  I am not worried about how hard or easy it would be to do, just if it is even possible.

Jim

_______________________________________________
jose mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/jose

---

From: Richard Barnes <[email-masked]>
Date: Mon, 10 Jun 2013 13:00:53 -0400

I think you've misunderstood the proposal.  It's not that the field will be
unencoded, it's that it will be encoded as a UTF-8 string instead of a
base64 string.

OLD: (json) --> UTF-8 --> base64
NEW: (json) --> UTF-8

OLD:  { protected: "eyJhbGciOiJSUzI1NiJ9Cg==" }
NEW: { protected: "{\"alg\":\"RS256\"}" }

Now, there might still be a problem with that, because JSON strings aren't
canonical.  In the example above, the quote characters could also have been
presented as "\u0022".  So if some JSON re-processor were to change the
encoding ot the string, it would break the signature.  However, that's also
a problem for the base64-encoded version ("a" vs. "\u0061").

On Mon, Jun 10, 2013 at 12:38 PM, Mike Jones <[email-masked]>wrote:

>  The problem with this suggested change is that it requires unencoded
> JSON objects to be transmitted with no transformations whatsoever, whereas
> the problems with this assumption are well known.  For starters, on
> UNIX-based systems, newlines are typically represented by a bare LF
> character, whereas on DOS-based systems, newlines are typically represented
> by a CRLF pair.  In transmission, many systems, including mail agents tend
> to convert newlines to the system’s normal format.  This would break these
> JOSE objects.****
>
> ** **
>
> Some agents wrap lines after a certain length.  Particularly when being
> transmitted in HTML or XML, many systems replace two or more consecutive
> spaces with a single space.  Others replace two or more spaces with a
> single space and N-1 non-breaking space characters.  I could go on…****
>
> ** **
>
> I appreciate the attempt to make things appear to be more uniform and
> readable, but the CRLF/LF problems alone are enough to make this a
> non-starter.****
>
> ** **
>
>                                                                 Best
> wishes,****
>
>                                                                 -- Mike***
> *
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Richard Barnes
> *Sent:* Monday, June 10, 2013 9:29 AM
> *To:* Jim Schaad
> *Cc:* [email-masked]
> *Subject:* Re: [jose] Possible change to protected field****
>
> ** **
>
> ASN.1, namely use of OCTET STRING to encapsulate more DER content.****
>
> ** **
>
> it would not be a breaking change for that serialization.****
>
> ** **
>
> --Richard****
>
> ** **
>
> On Mon, Jun 10, 2013 at 1:46 AM, Jim Schaad <[email-masked]>
> wrote:****
>
> <no hat>****
>
>  ****
>
>  ****
>  ****
>  ****
>  ****
> https://www.ietf.org/mailman/listinfo/jose****
>
> ** **
>

---

From: Mike Jones <[email-masked]>
Date: Mon, 10 Jun 2013 17:08:35 +0000

This would take us down the road to canonicalization.  How would you represent a CRLF in the value?  How would you represent a bare LF?  How would you represent a tab?  Canonical representations for all of these, and more, would have to be specified.

Using a special encoding for this field that is different than the encoding used for all other fields (base64url encoding) is developer-hostile.  It forces them to have (and test) special code to produce a different for this field alone.  This would undoubtedly result in interop problems, because the corner cases almost never get tested (and often never get coded correctly either).

We already have a simple means of ensuring the correct transmission of fields.  Occam's Razor would suggest just using it.

                                                                -- Mike

From: Richard Barnes [mailto:[email-masked]]
Sent: Monday, June 10, 2013 10:01 AM
To: Mike Jones
Cc: Jim Schaad; [email-masked]
Subject: Re: [jose] Possible change to protected field

I think you've misunderstood the proposal.  It's not that the field will be unencoded, it's that it will be encoded as a UTF-8 string instead of a base64 string.

Now, there might still be a problem with that, because JSON strings aren't canonical.  In the example above, the quote characters could also have been presented as "\u0022".  So if some JSON re-processor were to change the encoding ot the string, it would break the signature.  However, that's also a problem for the base64-encoded version ("a" vs. "\u0061").

On Mon, Jun 10, 2013 at 12:38 PM, Mike Jones <[email-masked]<mailto:[email-masked]>> wrote:
The problem with this suggested change is that it requires unencoded JSON objects to be transmitted with no transformations whatsoever, whereas the problems with this assumption are well known.  For starters, on UNIX-based systems, newlines are typically represented by a bare LF character, whereas on DOS-based systems, newlines are typically represented by a CRLF pair.  In transmission, many systems, including mail agents tend to convert newlines to the system's normal format.  This would break these JOSE objects.

                                                                Best wishes,
                                                                -- Mike

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Richard Barnes
Sent: Monday, June 10, 2013 9:29 AM
To: Jim Schaad
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [jose] Possible change to protected field

--Richard

Jim

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 10 Jun 2013 10:24:50 -0700

Mike,

I can dispose of the CRLF question quite simply.  I have not removed the
UTF8 string from the protected field so all CRLF characters (along with a
number of others) will be escaped so they will not be transformed.

It would still be

{"protected":"---UTF8 encoded string ---"}

Rather than

{"protected":"--- Base64 URL encoded ( -- UTF8 encoded string  -- ) ---" }

There are a number of other questions that do come into play that you did
raise however.

The first would be space folding -  It has been my experience that this is
normally the XML parser that does it rather than things in transit.  There
will always be a problem with things in the middle that could possibly
change things with this approach.  It is something that I need to be looked
at and think about.

If we have a line wrapping issue, this is going to be a problem with JSON
parsers today.  It is not legal to have any of the CRLF characters inside of
a string in JSON according to the current ABNF in the document.  I think
this may already cause parsers to go boom in some cases.  If this is going
to be a real issue then we probably need to start dealing with it sooner
rather than later.

Jim

From: Mike Jones [mailto:[email-masked]] 
Sent: Monday, June 10, 2013 9:39 AM
To: Richard Barnes; Jim Schaad
Cc: [email-masked]
Subject: RE: [jose] Possible change to protected field

The problem with this suggested change is that it requires unencoded JSON
objects to be transmitted with no transformations whatsoever, whereas the
problems with this assumption are well known.  For starters, on UNIX-based
systems, newlines are typically represented by a bare LF character, whereas
on DOS-based systems, newlines are typically represented by a CRLF pair.  In
transmission, many systems, including mail agents tend to convert newlines
to the system's normal format.  This would break these JOSE objects.

Some agents wrap lines after a certain length.  Particularly when being
transmitted in HTML or XML, many systems replace two or more consecutive
spaces with a single space.  Others replace two or more spaces with a single
space and N-1 non-breaking space characters.  I could go on.

non-starter.

                                                                Best wishes,

                                                                -- Mike

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Richard Barnes
Sent: Monday, June 10, 2013 9:29 AM
To: Jim Schaad
Cc: [email-masked]
Subject: Re: [jose] Possible change to protected field

The compact serialization can continue to base64url-encode that field, so it
would not be a breaking change for that serialization.

--Richard

<no hat>

Jim

---

From: Richard Barnes <[email-masked]>
Date: Mon, 10 Jun 2013 13:42:44 -0400

It's not canonicalization, it's just UTF-8 serialization.  As to your
questions about CR / LF / tab, let us recall RFC 4627:

         char = unescaped /
                escape (
                    %x22 /          ; "    quotation mark  U+0022
                    %x5C /          ; \    reverse solidus U+005C
                    %x2F /          ; /    solidus         U+002F
                    %x62 /          ; b    backspace       U+0008
                    %x66 /          ; f    form feed       U+000C
                    %x6E /          ; n    line feed       U+000A
                    %x72 /          ; r    carriage return U+000D
                    %x74 /          ; t    tab             U+0009
                    %x75 4HEXDIG )  ; uXXXX                U+XXXX

Empirically, it appears that this serialization is stable with regard to
re-encoding, at least for browser JSON engines:
<http://www.ipv.sx/jose/json_test.html>

The only thing that would lead me to prefer this serialization against the
Occam's Razor argument is that it's more readable for someone that receives
a JOSE object.  Instead of staring at base64-encoded junk, you can actually
sort of see what's in there.  But I don't feel very strongly about this.

--Richard

On Mon, Jun 10, 2013 at 1:08 PM, Mike Jones <[email-masked]>wrote:

>  This would take us down the road to canonicalization.  How would you
> represent a CRLF in the value?  How would you represent a bare LF?  How
> would you represent a tab?  Canonical representations for all of these, and
> more, would have to be specified.****
>
> ** **
>
> Using a special encoding for this field that is different than the
> encoding used for all other fields (base64url encoding) is
> developer-hostile.  It forces them to have (and test) special code to
> produce a different for this field alone.  This would undoubtedly result in
> interop problems, because the corner cases almost never get tested (and
> often never get coded correctly either).****
>
> ** **
>
> We already have a simple means of ensuring the correct transmission of
> fields.  Occam’s Razor would suggest just using it.****
>
> ** **
>
>                                                                 -- Mike***
> *
>
> ** **
>
> *From:* Richard Barnes [mailto:[email-masked]]
> *Sent:* Monday, June 10, 2013 10:01 AM
> *To:* Mike Jones
> *Cc:* Jim Schaad; [email-masked]
>
> I think you've misunderstood the proposal.  It's not that the field will
> be unencoded, it's that it will be encoded as a UTF-8 string instead of a
> base64 string.****
>
> ** **
>
> OLD: (json) --> UTF-8 --> base64****
>
> NEW: (json) --> UTF-8****
>
> ** **
>
> OLD:  { protected: "eyJhbGciOiJSUzI1NiJ9Cg==" }****
>
> NEW: { protected: "{\"alg\":\"RS256\"}" }****
>
> ** **
>
> a problem for the base64-encoded version ("a" vs. "\u0061").****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> On Mon, Jun 10, 2013 at 12:38 PM, Mike Jones <[email-masked]>
> wrote:****
>
> on DOS-based systems, newlines are typically represented by a CRLF pair.
> In transmission, many systems, including mail agents tend to convert
> newlines to the system’s normal format.  This would break these JOSE
> objects.****
>
>  ****
>

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 10 Jun 2013 10:52:09 -0700

I don't understand why you think it takes us down the road to
canonicalization.  If there are canonization problems in this proposal I
would say that they may already exist.

I don't know if you are reading the JSON list or not, however you seem to be
confusing the encoded and the unencoded versions of the string.  The string
does need to be encoded into a format that is legal to place into a JSON
string field.  In that case there is a number of ways that it can be done
using quoting.  However when the JSON parser gives that string to the
application - it will be the original UTF8 encoded string.  This means that
a bare LF character would be the "code point" 10.  That is the UTF8 encoding
for a line feed character.  It does not get canonicalized in any way.  A
UTF8 string does have possible code points that are not legal in a JSON
string, but that is JSONs problem not ours.  The JSON string encoding format
is responsible for the string to text and text to string transformations and
making sure that those transformations are loss-less.

I agree that it may be harder to do, I have not formally presented this as a
change to the document and will not do so until I have done a full analysis
on how hard it is.  I am currently just trying to figure out if it is even
technically feasible.

Jim

From: Mike Jones [mailto:[email-masked]] 
Sent: Monday, June 10, 2013 10:09 AM
To: Richard Barnes
Cc: Jim Schaad; [email-masked]
Subject: RE: [jose] Possible change to protected field

This would take us down the road to canonicalization.  How would you
represent a CRLF in the value?  How would you represent a bare LF?  How
would you represent a tab?  Canonical representations for all of these, and
more, would have to be specified.

Using a special encoding for this field that is different than the encoding
used for all other fields (base64url encoding) is developer-hostile.  It
forces them to have (and test) special code to produce a different for this
field alone.  This would undoubtedly result in interop problems, because the
corner cases almost never get tested (and often never get coded correctly
either).

We already have a simple means of ensuring the correct transmission of
fields.  Occam's Razor would suggest just using it.

                                                                -- Mike

NEW: (json) --> UTF-8

 

wrote:

                                                                Best wishes,

                                                                -- Mike

--Richard

<no hat>

Jim

---

From: Mike Jones <[email-masked]>
Date: Mon, 10 Jun 2013 20:25:06 +0000

It's a form of canonicalization when you take one representation of a string and apply a set of rules to transform it to a more stable/transmissible string.  In this case, we'd be proposing that developers have to translate CRLF to \u000DU\u00A, and other similar canonicalization transformations.  That's a kind of canonicalization.  Worse, it's one that I don't believe there's standard library support for, so developers would have to write it for every implementation and platform.  They'd hate us, and with good reason.

And this isn't academic.  It's commonplace to use newlines in formatted JSON, such as:

{"alg": "RSA-OAEP",
"enc": "A256GCM",
"kid": "7"
}

Occam's Razor suggests always using the same encoding when a stable, transmissible string representation is needed - in this case base64url encoding.

The somewhat improved readability of the canonicalization being discussed isn't worth the interop problems or developer pain it would cost.  Therefore, I sincerely hope that no one actually proposes this.

                                                            -- Mike

From: Jim Schaad [mailto:[email-masked]]
Sent: Monday, June 10, 2013 10:52 AM
To: Mike Jones; 'Richard Barnes'
Cc: [email-masked]
Subject: RE: [jose] Possible change to protected field

I don't understand why you think it takes us down the road to canonicalization.  If there are canonization problems in this proposal I would say that they may already exist.

I don't know if you are reading the JSON list or not, however you seem to be confusing the encoded and the unencoded versions of the string.  The string does need to be encoded into a format that is legal to place into a JSON string field.  In that case there is a number of ways that it can be done using quoting.  However when the JSON parser gives that string to the application - it will be the original UTF8 encoded string.  This means that a bare LF character would be the "code point" 10.  That is the UTF8 encoding for a line feed character.  It does not get canonicalized in any way.  A UTF8 string does have possible code points that are not legal in a JSON string, but that is JSONs problem not ours.  The JSON string encoding format is responsible for the string to text and text to string transformations and making sure that those transformations are loss-less.

I agree that it may be harder to do, I have not formally presented this as a change to the document and will not do so until I have done a full analysis on how hard it is.  I am currently just trying to figure out if it is even technically feasible.

Jim

From: Mike Jones [mailto:[email-masked]]
Sent: Monday, June 10, 2013 10:09 AM
To: Richard Barnes
Cc: Jim Schaad; [email-masked]<mailto:[email-masked]>
Subject: RE: [jose] Possible change to protected field

                                                                -- Mike

From: Richard Barnes [mailto:[email-masked]]
Sent: Monday, June 10, 2013 10:01 AM
To: Mike Jones
Cc: Jim Schaad; [email-masked]<mailto:[email-masked]>
Subject: Re: [jose] Possible change to protected field

                                                                Best wishes,
                                                                -- Mike

--Richard

Jim

---

From: Richard Barnes <[email-masked]>
Date: Mon, 10 Jun 2013 17:29:31 -0400

Mike: They already have to do that stuff before they base64-encode it.  All
Jim is talking about is *not* doing the base64.

On Mon, Jun 10, 2013 at 4:25 PM, Mike Jones <[email-masked]>wrote:

>  It’s a form of canonicalization when you take one representation of a
> string and apply a set of rules to transform it to a more
> stable/transmissible string.  In this case, we’d be proposing that
> developers have to translate CRLF to \u000DU\u00A, and other similar
> canonicalization transformations.  That’s a kind of canonicalization.
> Worse, it’s one that I don’t believe there’s standard library support for,
> so developers would have to write it for every implementation and
> platform.  They’d hate us, and with good reason.****
>
> ** **
>
> And this isn’t academic.  It’s commonplace to use newlines in formatted
> JSON, such as:****
>
> ** **
>
> {"alg": "RSA-OAEP",****
>
> "enc": "A256GCM",****
>
> "kid": "7"****
>
> }****
>
> ** **
>
> Occam’s Razor suggests always using the same encoding when a stable,
> transmissible string representation is needed – in this case base64url
> encoding.****
>
> ** **
>
> The somewhat improved readability of the canonicalization being discussed
> isn’t worth the interop problems or developer pain it would cost.
> Therefore, I sincerely hope that no one actually proposes this.****
>
> ** **
>
>                                                             -- Mike****
>
> ** **
>
> *From:* Jim Schaad [mailto:[email-masked]]
> *Sent:* Monday, June 10, 2013 10:52 AM
> *To:* Mike Jones; 'Richard Barnes'
> *Cc:* [email-masked]
>
> *Subject:* RE: [jose] Possible change to protected field****
>
>  ** **
>
> I don’t understand why you think it takes us down the road to
> would say that they may already exist.****
>
> ** **
>
> I don’t know if you are reading the JSON list or not, however you seem to
> be confusing the encoded and the unencoded versions of the string.  The
> string does need to be encoded into a format that is legal to place into a
> JSON string field.  In that case there is a number of ways that it can be
> done using quoting.  However when the JSON parser gives that string to the
> application – it will be the original UTF8 encoded string.  This means that
> a bare LF character would be the “code point” 10.  That is the UTF8
> encoding for a line feed character.  It does not get canonicalized in any
> way.  A UTF8 string does have possible code points that are not legal in a
> JSON string, but that is JSONs problem not ours.  The JSON string encoding
> format is responsible for the string to text and text to string
> transformations and making sure that those transformations are loss-less.*
> ***
>
> ** **
>
> I agree that it may be harder to do, I have not formally presented this as
> a change to the document and will not do so until I have done a full
> analysis on how hard it is.  I am currently just trying to figure out if it
> is even technically feasible.****
>
> ** **
>
> Jim****
>
> ** **
>
> ** **
>
> *From:* Mike Jones [mailto:[email-masked]<[email-masked]>]
>
> *Sent:* Monday, June 10, 2013 10:09 AM
> *To:* Richard Barnes
> *Subject:* RE: [jose] Possible change to protected field****
>
> ** **
>
> *From:* Richard Barnes [mailto:[email-masked] <[email-masked]>]

---

From: Mike Jones <[email-masked]>
Date: Mon, 10 Jun 2013 21:36:03 +0000

My point is that if you just base64url encode the string - line the one below - CRLFs and all, you don't have to do any of the canonicalization steps to make the representation JSON-string-safe in the first place.  And you're using the same tool you're already using for encoding every other piece of binary content that needs to be encoded in some way for JOSE.

Otherwise, we also still have to worry about "innocuous" transformations that might during transport like space to non-breaking-space, tab to spaces, spaces being added at the end of lines (which HTML mail transports are notorious for), etc.

If we apply the KISS principle and use the same encoding in this case that we do in all other cases, we don't have to even consider whether those and other content integrity threats that we may not have even thought of are going to hurt anyone in practice, because they can never arise.

                                                                -- Mike

From: Richard Barnes [mailto:[email-masked]]
Sent: Monday, June 10, 2013 2:30 PM
To: Mike Jones
Cc: Jim Schaad; [email-masked]
Subject: Re: [jose] Possible change to protected field

Mike: They already have to do that stuff before they base64-encode it.  All Jim is talking about is *not* doing the base64.

On Mon, Jun 10, 2013 at 4:25 PM, Mike Jones <[email-masked]<mailto:[email-masked]>> wrote:
It's a form of canonicalization when you take one representation of a string and apply a set of rules to transform it to a more stable/transmissible string.  In this case, we'd be proposing that developers have to translate CRLF to \u000DU\u00A, and other similar canonicalization transformations.  That's a kind of canonicalization.  Worse, it's one that I don't believe there's standard library support for, so developers would have to write it for every implementation and platform.  They'd hate us, and with good reason.

{"alg": "RSA-OAEP",
"enc": "A256GCM",
"kid": "7"
}

                                                            -- Mike

From: Jim Schaad [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Monday, June 10, 2013 10:52 AM
To: Mike Jones; 'Richard Barnes'
Cc: [email-masked]<mailto:[email-masked]>

Jim

                                                                -- Mike

                                                                Best wishes,
                                                                -- Mike

--Richard

Jim

---

From: Richard Barnes <[email-masked]>
Date: Mon, 10 Jun 2013 17:55:28 -0400

Oh, I see.  There actually is a new transformation here:
OLD: (json) --> UTF-8 --> base64
NEW: (json) --> UTF-8 --> json-string

I'm not totally convinced that this is actually a new thing, that is, that
JSON.stringify(protect) won't produce a JSON-safe string.  In the JSON
parsers I'm looking at, the output of JSON.stringify() is a JSON-safe
string.

However, in the course of looking at that question, I came across another
thing that I think is a better reason to abandon this proposal.  Namely,
the JSON parsers in both Chrome and Firefox (possible others; haven't
tried) choke on parsing strings that have control characters.  So while the
following line executes with no problem:
   var x = {a:"one line \n two lines"}
calling JSON.parse(JSON.stringify(x)) results in either an error (Chrome)
or an incorrect parse (Firefox).

So regardless of "canonicalization" concerns (which aren't really about
canonicalization), it doesn't seem to me that this is a change that would
be very deployable.  Guess that answers Jim's question about "whether it's
even possible" :)

--Richard

On Mon, Jun 10, 2013 at 5:36 PM, Mike Jones <[email-masked]>wrote:

>  My point is that if you just base64url encode the string – line the one
> below – CRLFs and all, you don’t have to do any of the canonicalization
> steps to make the representation JSON-string-safe in the first place.  And
> you’re using the same tool you’re already using for encoding every other
> piece of binary content that needs to be encoded in some way for JOSE.****
>
> ** **
>
> Otherwise, we also still have to worry about “innocuous” transformations
> that might during transport like space to non-breaking-space, tab to
> spaces, spaces being added at the end of lines (which HTML mail transports
> are notorious for), etc.****
>
> ** **
>
> If we apply the KISS principle and use the same encoding in this case that
> we do in all other cases, we don’t have to even consider whether those and
> other content integrity threats that we may not have even thought of are
> going to hurt anyone in practice, because they can never arise.****
>
> ** **
>
>                                                                 -- Mike***
> *
>
> ** **
>
> *Sent:* Monday, June 10, 2013 2:30 PM
>
> *To:* Mike Jones
> Mike: They already have to do that stuff before they base64-encode it.
>  All Jim is talking about is *not* doing the base64.****
>
> ** **
>
> On Mon, Jun 10, 2013 at 4:25 PM, Mike Jones <[email-masked]>
> wrote:****
>
> *Cc:* [email-masked]****

---

From: "Manger, James H" <[email-masked]>
Date: Tue, 11 Jun 2013 11:45:57 +1000

>>> I was thinking about proposing that we make a change to the content of the protected field in the JWS JSON serialization format.  If we encoded this as a UTF8 string rather than the base64url encoded UTF8 string, then the content would be smaller.

This would work. Mike's canonicalization worries are misplaced. Richard's Chrome/Firefox "bug" seems wrong.

Example: JOSE originator creates the following header, with U+000A for line ending.

{"alg": "RSA-OAEP",
"enc": "A256GCM",
"kid": "7"
}

This header consists of 19+1+17+1+10+1+1+1=51 characters (including 3 spaces and 4 newlines). Put these 51 chars as the value of the "protected" field.

{"protected":"{\"alg\": \"RSA-OAEP\",\n\"enc\": \"A256GCM\",\n\"kid\": \"7\"\n}\n"}

They appear as 51+16=67 chars inside the quotes of the "protected" field: 4 newlines are replaced with 4 n chars; plus 16 slashes are added. This step could have used \u0022 instead of \", or \u000A or \u000a instead of \n, or escaped any of the other 35 chars. But that doesn't matter since this escaping will be removed by the first JSON parse at the other end, which will gives the original 51 chars to its caller (in whatever native string type the language uses).

If the originator used CRLF instead (giving 55 chars), the transmitted data will be {"protected":"...\r\n...\r\n...\r\n...\r\n"}. The first JSON parse at the recipient will recreate the correct 55 chars. No problem.

There is plenty of potential to confuse a human doing this manually as there are 2 layers of JSON string escaping. It is well-defined enough for a computer (or spec) though.

As to whether this is worthwhile...
In this specific example the base64url overhead would be +17 chars, merely 1 more char than the overhead of omitting the base64url step.

A more valuable change would be to apply the crypto operations AFTER removing the base64url encoding. That is, apply the crypto to a UTF-8-encoded JSON header, instead of to the US-ASCII encoding of the base64url-encoded UTF-8-encoded JSON header. Implementations always need the UTF-8 form at some point anyway. It means the crypto is applied to fewer bytes so it is faster. It means an idea like Jim's that doesn't need base64url to maintain the correct bytes in transmission doesn't need to artificially add a base64url-encoding step at the receiver just to do the crypto.

P.S.
>> the JSON parsers in both Chrome and Firefox (possible others; haven't tried) choke on parsing strings that have control characters

JSON does not allow unescaped control chars when representing strings so choking is the correct behaviour. JSON.parse(JSON.stringify(x)) works for me in Chrome. JSON.stringify does NOT leave unescaped control chars in quotes in the result.

--
James Manger

---

From: Richard Barnes <[email-masked]>
Date: Tue, 11 Jun 2013 10:10:30 -0400

On Mon, Jun 10, 2013 at 9:45 PM, Manger, James H <
[email-masked]> wrote:

> the content would be smaller.
>
>
> This header consists of 19+1+17+1+10+1+1+1=51 characters (including 3
> spaces and 4 newlines). Put these 51 chars as the value of the "protected"
> field.
>
> They appear as 51+16=67 chars inside the quotes of the "protected" field:
> 4 newlines are replaced with 4 n chars; plus 16 slashes are added. This
> step could have used \u0022 instead of \", or \u000A or \u000a instead of
> \n, or escaped any of the other 35 chars. But that doesn't matter since
> this escaping will be removed by the first JSON parse at the other end,
> which will gives the original 51 chars to its caller (in whatever native
> string type the language uses).
>
> If the originator used CRLF instead (giving 55 chars), the transmitted
> data will be {"protected":"...\r\n...\r\n...\r\n...\r\n"}. The first JSON
> parse at the recipient will recreate the correct 55 chars. No problem.
>
> There is plenty of potential to confuse a human doing this manually as
> there are 2 layers of JSON string escaping. It is well-defined enough for a
> computer (or spec) though.
>
>
> In this specific example the base64url overhead would be +17 chars, merely
> 1 more char than the overhead of omitting the base64url step.
>
>
> A more valuable change would be to apply the crypto operations AFTER
> removing the base64url encoding. That is, apply the crypto to a
> UTF-8-encoded JSON header, instead of to the US-ASCII encoding of the
> base64url-encoded UTF-8-encoded JSON header. Implementations always need
> the UTF-8 form at some point anyway. It means the crypto is applied to
> fewer bytes so it is faster. It means an idea like Jim's that doesn't need
> base64url to maintain the correct bytes in transmission doesn't need to
> artificially add a base64url-encoding step at the receiver just to do the
> crypto.
>
>
> P.S.
> tried) choke on parsing strings that have control characters
>
> JSON does not allow unescaped control chars when representing strings so
> choking is the correct behaviour. JSON.parse(JSON.stringify(x)) works for
> me in Chrome. JSON.stringify does NOT leave unescaped control chars in
> quotes in the result.
>

I have heard multiple independent reports of my incorrectness in this
experiment.  So I retract my assertion of infeasibility, with regard to
control characters.

However, there's still an issue with code points that don't have to be
escaped. Consider the following example:

var x = '{"a":"caf\\u00e9"}';
var x2 = JSON.stringify(JSON.parse(x))

In a quick test of the environments I have on hand (several browsers +
Node.JS + PHP + Perl), I see a wide variety of behaviors, most of which
would break the signature.  The browsers and Node all converted the
"\u00e9" sequence to an unescaped "é" character, while Perl (creatively)
rendered it directly as a Latin-1 0xe9 byte.  Only PHP rendered it back to
the escape sequence.  Test code here:
<http://www.ipv.sx/jose/json_test.html>

Now, even if we think that re-encoding is a problem, there's the question
of whether we think the risk is acceptable to achieve better human
readability.  I'm pretty split, myself.

--Richard

---

From: "Jim Schaad" <[email-masked]>
Date: Tue, 11 Jun 2013 10:12:17 -0700

From: Richard Barnes [mailto:[email-masked]] 
Sent: Tuesday, June 11, 2013 7:11 AM
To: Manger, James H
Cc: Mike Jones; Jim Schaad; [email-masked]
Subject: Re: [jose] Possible change to protected field

This would work. Mike's canonicalization worries are misplaced. Richard's
Chrome/Firefox "bug" seems wrong.

Example: JOSE originator creates the following header, with U+000A for line
ending.

{"alg": "RSA-OAEP",
"enc": "A256GCM",
"kid": "7"
}

This header consists of 19+1+17+1+10+1+1+1=51 characters (including 3 spaces
and 4 newlines). Put these 51 chars as the value of the "protected" field.

{"protected":"{\"alg\": \"RSA-OAEP\",\n\"enc\": \"A256GCM\",\n\"kid\":
\"7\"\n}\n"}

They appear as 51+16=67 chars inside the quotes of the "protected" field: 4
newlines are replaced with 4 n chars; plus 16 slashes are added. This step
could have used \u0022 instead of \", or \u000A or \u000a instead of \n, or
escaped any of the other 35 chars. But that doesn't matter since this
escaping will be removed by the first JSON parse at the other end, which
will gives the original 51 chars to its caller (in whatever native string
type the language uses).

If the originator used CRLF instead (giving 55 chars), the transmitted data
will be {"protected":"...\r\n...\r\n...\r\n...\r\n"}. The first JSON parse
at the recipient will recreate the correct 55 chars. No problem.

There is plenty of potential to confuse a human doing this manually as there
are 2 layers of JSON string escaping. It is well-defined enough for a
computer (or spec) though.

As to whether this is worthwhile...
In this specific example the base64url overhead would be +17 chars, merely 1
more char than the overhead of omitting the base64url step.

A more valuable change would be to apply the crypto operations AFTER
removing the base64url encoding. That is, apply the crypto to a
UTF-8-encoded JSON header, instead of to the US-ASCII encoding of the
base64url-encoded UTF-8-encoded JSON header. Implementations always need the
UTF-8 form at some point anyway. It means the crypto is applied to fewer
bytes so it is faster. It means an idea like Jim's that doesn't need
base64url to maintain the correct bytes in transmission doesn't need to
artificially add a base64url-encoding step at the receiver just to do the
crypto.

P.S.

JSON does not allow unescaped control chars when representing strings so
choking is the correct behaviour. JSON.parse(JSON.stringify(x)) works for me
in Chrome. JSON.stringify does NOT leave unescaped control chars in quotes
in the result.

In a quick test of the environments I have on hand (several browsers +
Node.JS + PHP + Perl), I see a wide variety of behaviors, most of which
would break the signature.  The browsers and Node all converted the "\u00e9"
sequence to an unescaped "é" character, while Perl (creatively) rendered it
directly as a Latin-1 0xe9 byte.  Only PHP rendered it back to the escape
sequence.  Test code here:

Now, even if we think that re-encoding is a problem, there's the question of
whether we think the risk is acceptable to achieve better human readability.
I'm pretty split, myself.

Do you thing from your test that this would also be a problem if you did

Base64(JSON.stringify(JSON.parse(Unbase64(x)))

This would appear to be a general re-creation problem even if we use base64
encoding

Jim

--Richard

--
James Manger

---

From: Richard Barnes <[email-masked]>
Date: Tue, 11 Jun 2013 13:34:14 -0400

On Tue, Jun 11, 2013 at 1:12 PM, Jim Schaad <[email-masked]> wrote:

> *Sent:* Tuesday, June 11, 2013 7:11 AM
> *To:* Manger, James H
> *Cc:* Mike Jones; Jim Schaad; [email-masked]
>
> [email-masked]> wrote:****
>
> Example: JOSE originator creates the following header, with U+000A for
> line ending.****
>
>
> {"alg": "RSA-OAEP",
> "enc": "A256GCM",
> "kid": "7"
> }****
>
> P.S.****
> tried) choke on parsing strings that have control characters****
>
> quotes in the result.****
>
> ** **
>
> control characters.****
> escaped. Consider the following example:****
>
> ** **
>
> var x = '{"a":"caf\\u00e9"}';****
>
> var x2 = JSON.stringify(JSON.parse(x))****
>
> ** **
>
> the escape sequence.  Test code here:****
>
> <http://www.ipv.sx/jose/json_test.html>****
>
> ** **
>
> readability.  I'm pretty split, myself.****
>
> ** **
>
> Do you thing from your test that this would also be a problem if you did**
> **
>
> ** **
>
> Base64(JSON.stringify(JSON.parse(Unbase64(x)))****
>
> ** **
>
> This would appear to be a general re-creation problem even if we use
> base64 encoding
>

The base64 part in your line actually isn't relevant.  It's actually the
outer parse/stringify (on the overall JWE object) that we're worried about
changing the string.

The control case for the experiment would be:

var x = '{"a":"validBase64=="}';
var x2 = JSON.stringify(JSON.parse(x))

That works just fine with my standard test suite.
<http://www.ipv.sx/jose/json_test.html>

--Richard

> James Manger****

---

From: "Jim Schaad" <[email-masked]>
Date: Tue, 11 Jun 2013 11:49:35 -0700

Richard,

Let’s not confuse the text representation and the memory representation.
All that matters is that what would be presented to the hash function is the
byte sequence

0x63 0x61 0x66 0xe9

It does not matter what the escaped text string looks like as long as
everyone can read it in and get the correct byte sequence.

Jim

From: Richard Barnes [mailto:[email-masked]] 
Sent: Tuesday, June 11, 2013 10:34 AM
To: Jim Schaad
Cc: Manger, James H; Mike Jones; [email-masked]
Subject: Re: [jose] Possible change to protected field

 

{"alg": "RSA-OAEP",
"enc": "A256GCM",
"kid": "7"
}

P.S.

--Richard

Jim

--Richard

--
James Manger