Subject: [jose] #27: member names MUST be unique needs additional text

From: "jose issue tracker" <[email-masked]>
Date: Tue, 25 Jun 2013 06:30:04 -0000

#27: member names MUST be unique needs additional text

 Based on the discussions on the JSON mailing list, it is clear that there
 will never be a requirement on parsers that member names at a lexical
 scope need to be unique.

 Based on the discussions on the JSON mailing list, it is clear that if
 there are two members with the same name, it is not predictable if you
 will get an error, both elements the first element or the last element.
 It will be highly parser dependent.

 From this we need to include additional text on why names need to be
 unique and how to go about checking that they are.  We will also need to
 include advice about what to do if a system is unable to provide the
 uniqueness requirement.

-- 
-------------------------+-------------------------------------------------
 Reporter:               |      Owner:  draft-ietf-jose-json-web-
  [email-masked] |  [email-masked]
     Type:  defect       |     Status:  new
 Priority:  major        |  Milestone:
Component:  json-web-    |    Version:
  signature              |   Keywords:
 Severity:  -            |
-------------------------+-------------------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/jose/trac/ticket/27>
jose <http://tools.ietf.org/jose/>

---

From: "jose issue tracker" <[email-masked]>
Date: Wed, 26 Jun 2013 00:41:12 -0000

Comment (by [email-masked]):

 The JWS draft currently says:

         The Header Parameter Names within the JWS Header MUST be unique;
         JWSs with duplicate Header Parameter Names MUST be rejected.

 How about changing this to:

         The Header Parameter Names within the JWS Header MUST be unique;
         JWSs with duplicate Header Parameter Names MUST be rejected.
         This is necessary to prevent attacks in which the same JWS might
 be interpreted
         in different ways by different implementations and to prevent
 attackers
         from hiding extra content in duplicate member values.
         If the platform’s JSON parser does not reject input with duplicate
 member names,
         the input will first need to be separately parsed to reject these
 invalid inputs
         before using the platform’s parser.

  signature              |  Resolution:
 Keywords:               |

Ticket URL: <http://trac.tools.ietf.org/wg/jose/trac/ticket/27#comment:1>
jose <http://tools.ietf.org/jose/>

---

From: "Jim Schaad" <[email-masked]>
Date: Wed, 26 Jun 2013 11:29:52 -0700

<no hat>

I consider myself to be reasonably competent in both English and Technical English.  I have no idea what I am supposed to be doing to deal with the text below.  Does this mean that I need to write an independent parser?  What about cases where it is coming in on a stream and I don't get to see the data before the parse occurs?  How are they interpreted differently?  What exactly is this supposed to be addressing.  Much of this could be skipped when we said don't do it.  Since this is no longer a viable statement due to the state of parsers, we need to be more explicit and say what is going on.

No I don't consider the suggested text to be adequate.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> jose issue tracker
> Sent: Tuesday, June 25, 2013 5:41 PM
> To: [email-masked];
> [email-masked]
> Cc: [email-masked]
> Subject: Re: [jose] #27: member names MUST be unique needs additional text
> 
>          This is necessary to prevent attacks in which the same JWS might  be
> interpreted
>          in different ways by different implementations and to prevent  attackers
>          If the platform s JSON parser does not reject input with duplicate
> member names,
>          the input will first need to be separately parsed to reject these  invalid
> inputs
>          before using the platform s parser.
> 
> --
> -------------------------+----------------------------------------------
> -------------------------+---
> -------------------------+----------------------------------------------
> -------------------------+---
> 
> _______________________________________________
> jose mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jose

---

From: Mike Jones <[email-masked]>
Date: Wed, 26 Jun 2013 19:29:59 +0000

The operable sentence in my suggested text below is this one: "If the platform's JSON parser does not reject input with duplicate member names, the input will first need to be separately parsed to reject these invalid inputs before using the platform's parser".  In other words, if the JSON parser in the development platform you are using does not reject inputs with duplicate member names, you will need to write a separate JSON parser that detects this invalid input and rejects it.

This parser could either just be a validator, returning TRUE or FALSE for whether the JSON is valid for JOSE - in which case you'd then pass any inputs validating as TRUE to the platform's JSON parser, or it could be a replacement parser, in which case your code would not use the development platform's JSON parser at all.  I suspect people would be more likely to do the former than the latter, but both approaches are equivalent.

BTW, if it's your sense that there's a problem occurring in the JSON working group with respect to enabling strict JSON parsing, we probably need to become active there.  For instance, even if the spec allows duplicate member names like the ECMA spec does, the RFC could recommend or require that parsers support a "strict" mode, which rejects these unnecessarily lax inputs.  Then JOSE implementations could use that.

				-- Mike

-----Original Message-----
From: Jim Schaad [mailto:[email-masked]] 
Sent: Wednesday, June 26, 2013 11:30 AM
To: 'jose issue tracker'; [email-masked]; Mike Jones
Cc: [email-masked]
Subject: RE: [jose] #27: member names MUST be unique needs additional text

<no hat>

> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of jose issue tracker
>          This is necessary to prevent attacks in which the same JWS 
> might  be interpreted
>          If the platform s JSON parser does not reject input with 
> duplicate member names,
>          the input will first need to be separately parsed to reject 
> these  invalid inputs
> -------------------------+--------------------------------------------
> -------------------------+--
> -------------------------+--------------------------------------------
> -------------------------+--
> Ticket URL: 
> <http://trac.tools.ietf.org/wg/jose/trac/ticket/27#comment:1>

---

From: "Jim Schaad" <[email-masked]>
Date: Wed, 26 Jun 2013 20:18:29 -0700

I would strongly encourage everyone in the JOSE list to join the JSON list and read what was is there and participate.

That being said there are situations where you might not be able to do an independent parser.  And in fact what you have stated is that there is now a requirement that a JOSE library supply a JSON parser.  Is that really what we want to say?  How far from creating a set of generator and parser requirements that are non standard and requiring a parser/generator are we from that?

> From: Mike Jones [mailto:[email-masked]]
> Sent: Wednesday, June 26, 2013 12:30 PM
> To: Jim Schaad; [email-masked]
> Cc: [email-masked]
> The operable sentence in my suggested text below is this one: "If the
> platform's JSON parser does not reject input with duplicate member names,
> the input will first need to be separately parsed to reject these invalid inputs
> before using the platform's parser".  In other words, if the JSON parser in the
> development platform you are using does not reject inputs with duplicate
> member names, you will need to write a separate JSON parser that detects
> this invalid input and rejects it.
> 
> This parser could either just be a validator, returning TRUE or FALSE for
> whether the JSON is valid for JOSE - in which case you'd then pass any inputs
> validating as TRUE to the platform's JSON parser, or it could be a replacement
> parser, in which case your code would not use the development platform's
> JSON parser at all.  I suspect people would be more likely to do the former
> than the latter, but both approaches are equivalent.
> 
> BTW, if it's your sense that there's a problem occurring in the JSON working
> group with respect to enabling strict JSON parsing, we probably need to
> become active there.  For instance, even if the spec allows duplicate member
> names like the ECMA spec does, the RFC could recommend or require that
> parsers support a "strict" mode, which rejects these unnecessarily lax inputs.
> Then JOSE implementations could use that.
> 
> 				-- Mike
> 
> I consider myself to be reasonably competent in both English and Technical
> English.  I have no idea what I am supposed to be doing to deal with the text
> below.  Does this mean that I need to write an independent parser?  What
> about cases where it is coming in on a stream and I don't get to see the data
> before the parse occurs?  How are they interpreted differently?  What
> exactly is this supposed to be addressing.  Much of this could be skipped
> when we said don't do it.  Since this is no longer a viable statement due to the
> state of parsers, we need to be more explicit and say what is going on.
>

---

From: Mike Jones <[email-masked]>
Date: Thu, 27 Jun 2013 04:07:15 +0000

The alternative is to say that producers MUST NOT produce JSON with duplicate member names and that consumers are strongly RECOMMENDED to reject input with duplicate member names and leave it at that.

I understand not wanting to require customer parsers if the underlying platform parsers are deficient.  But I really hope that the JSON WG will have more sense than to allow the current madness to continue.  As I understood it, the primary motivation for the JSON WG in the first place was to change the SHOULD to a MUST.  If they don't do that, I'm really not sure what the point of it is.

				-- Mike

P.S.  I am a subscriber to the JSON mailing list, but unfortunately, the volume got so high a few weeks ago that trying to keep up with it became an impediment to getting real work done.  At that point I created a rule to file those messages into a folder, which I only rarely look at.  Has that situation improved?

-----Original Message-----
From: Jim Schaad [mailto:[email-masked]] 
Sent: Wednesday, June 26, 2013 8:18 PM
To: Mike Jones; [email-masked]
Cc: [email-masked]
Subject: RE: [jose] #27: member names MUST be unique needs additional text

> platform's JSON parser does not reject input with duplicate member 
> names, the input will first need to be separately parsed to reject 
> these invalid inputs before using the platform's parser".  In other 
> words, if the JSON parser in the development platform you are using 
> does not reject inputs with duplicate member names, you will need to 
> write a separate JSON parser that detects this invalid input and rejects it.
> 
> This parser could either just be a validator, returning TRUE or FALSE 
> for whether the JSON is valid for JOSE - in which case you'd then pass 
> any inputs validating as TRUE to the platform's JSON parser, or it 
> could be a replacement parser, in which case your code would not use 
> the development platform's JSON parser at all.  I suspect people would 
> be more likely to do the former than the latter, but both approaches are equivalent.
> 
> BTW, if it's your sense that there's a problem occurring in the JSON 
> working group with respect to enabling strict JSON parsing, we 
> probably need to become active there.  For instance, even if the spec 
> allows duplicate member names like the ECMA spec does, the RFC could 
> recommend or require that parsers support a "strict" mode, which rejects these unnecessarily lax inputs.
> To: 'jose issue tracker'; 
> [email-masked];
> Mike Jones
> Cc: [email-masked]
> I consider myself to be reasonably competent in both English and 
> Technical English.  I have no idea what I am supposed to be doing to 
> deal with the text below.  Does this mean that I need to write an 
> independent parser?  What about cases where it is coming in on a 
> stream and I don't get to see the data before the parse occurs?  How 
> are they interpreted differently?  What exactly is this supposed to be 
> addressing.  Much of this could be skipped when we said don't do it.  
> Since this is no longer a viable statement due to the state of parsers, we need to be more explicit and say what is going on.
> 
> > -------------------------+------------------------------------------
> > -------------------------+------------------------------------------

---

From: Tim Bray <[email-masked]>
Date: Wed, 26 Jun 2013 22:23:45 -0700

I think it’s a nice clean minimal solution to say that producers MUST NOT
generate dupes, end of story.  I don’t think saying anything beyond that
adds value. -T

On Wed, Jun 26, 2013 at 9:07 PM, Mike Jones <[email-masked]>wrote:

> The alternative is to say that producers MUST NOT produce JSON with
> duplicate member names and that consumers are strongly RECOMMENDED to
> reject input with duplicate member names and leave it at that.
>
> I understand not wanting to require customer parsers if the underlying
> platform parsers are deficient.  But I really hope that the JSON WG will
> have more sense than to allow the current madness to continue.  As I
> understood it, the primary motivation for the JSON WG in the first place
> was to change the SHOULD to a MUST.  If they don't do that, I'm really not
> sure what the point of it is.
>
>                                 -- Mike
>
> P.S.  I am a subscriber to the JSON mailing list, but unfortunately, the
> volume got so high a few weeks ago that trying to keep up with it became an
> impediment to getting real work done.  At that point I created a rule to
> file those messages into a folder, which I only rarely look at.  Has that
> situation improved?
>
> I would strongly encourage everyone in the JOSE list to join the JSON list
> and read what was is there and participate.
>
> That being said there are situations where you might not be able to do an
> independent parser.  And in fact what you have stated is that there is now
> a requirement that a JOSE library supply a JSON parser.  Is that really
> what we want to say?  How far from creating a set of generator and parser
> requirements that are non standard and requiring a parser/generator are we
> from that?
>
> > Since this is no longer a viable statement due to the state of parsers,
> we need to be more explicit and say what is going on.
> >

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 27 Jun 2013 15:26:12 +1000

JSON parsers that accept duplicate names, but ignore all but the last, are too common to believe JOSE will not be implemented using those parsers. See "[Json] Call for real-world examples of how parsers deal with duplicate keys" http://www.ietf.org/mail-archive/web/json/current/msg00471.html.

I suspect the best we can do is say:

  A creator of a JOSE message MUST NOT put duplicate names in any JSON object in a JOSE header.
  To prevent attacks where a JOSE message is interpreted as different valid messages by different recipients, each recipient MUST either reject messages with duplicate names or accept only the last name.

Plus agitate in the JSON group to add related text to the next version of the JSON spec.

--
James Manger

> Jim Schaad
> Sent: Thursday, 27 June 2013 1:18 PM
> To: 'Mike Jones'; [email-masked]
> Cc: [email-masked]
> I would strongly encourage everyone in the JOSE list to join the JSON
> list and read what was is there and participate.
> 
> That being said there are situations where you might not be able to do
> an independent parser.  And in fact what you have stated is that there
> is now a requirement that a JOSE library supply a JSON parser.  Is that
> really what we want to say?  How far from creating a set of generator
> and parser requirements that are non standard and requiring a
> parser/generator are we from that?
> 
> > Since this is no longer a viable statement due to the state of
> parsers, we need to be more explicit and say what is going on.
> >

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 27 Jun 2013 15:39:05 +1000

> I think it’s a nice clean minimal solution to say that producers MUST
> NOT generate dupes, end of story.  I don’t think saying anything beyond
> that adds value. -T

Clean and minimal that may be, but it ignores the security issue. We don't want a malicious producer (who is so malicious they ignore a MUST) to create JOSE messages that a JOSE-compliant security layer accepts as "benign interpretation #1" so it passes the message on to the JOSE-compliant backend app that acts on "nasty interpretation #2".

--
James Manger

---

From: Tim Bray <[email-masked]>
Date: Wed, 26 Jun 2013 23:15:44 -0700

Well, you have a practical problem in that most implementers will want to
use a standard JSON library, which is good practice because it will be
well-debugged, and most libraries [sob] silently take care of dupe keys and
don’t have a way to tell the client software what happened. So if you want
truly predictable behavior, you’re forcing the use of hand-constructed JSON
parsers. And that sucks, because getting good performance in JSON parsing
is surprisingly hard, with dramatic performance differences between
implementations. So you’re forcing receiving software which wants to be
conformant to use a hand-rolled parser which will probably have lousy
performance and have other bugs which in fact may compromise security more
than dupe-key tricks could.  -T

On Wed, Jun 26, 2013 at 10:39 PM, Manger, James H <
[email-masked]> wrote:

> Clean and minimal that may be, but it ignores the security issue. We don't
> want a malicious producer (who is so malicious they ignore a MUST) to
> create JOSE messages that a JOSE-compliant security layer accepts as
> "benign interpretation #1" so it passes the message on to the
> JOSE-compliant backend app that acts on "nasty interpretation #2".
>
> --
> James Manger
>

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 27 Jun 2013 16:29:58 +1000

Most JSON libraries that silently accept [sob] dupe keys at least consistently use the last key. Allowing just that behaviour (or rejecting dupes), while forbidding acceptance of a non-last dupe (eg first key), is safe and allows most JSON libraries to be used. Since this accepts most libraries I think it is practical.

Hence I suggest:

>   A creator of a JOSE message MUST NOT put duplicate names in any JSON

> object in a JOSE header.

>   To prevent attacks where a JOSE message is interpreted as different

> valid messages by different recipients, each recipient MUST either

> reject messages with duplicate names or accept only the last name.

This isn’t “truly predictable” as a message with dupes might be accepted or rejected. The crucial point is that if a message is accepted it is predictable.

--
James Manger

From: Tim Bray [mailto:[email-masked]]
Sent: Thursday, 27 June 2013 4:16 PM
To: Manger, James H
Cc: Mike Jones; Jim Schaad; [email-masked]; [email-masked]
Subject: Re: [jose] #27: member names MUST be unique needs additional text

Well, you have a practical problem in that most implementers will want to use a standard JSON library, which is good practice because it will be well-debugged, and most libraries [sob] silently take care of dupe keys and don’t have a way to tell the client software what happened. So if you want truly predictable behavior, you’re forcing the use of hand-constructed JSON parsers. And that sucks, because getting good performance in JSON parsing is surprisingly hard, with dramatic performance differences between implementations. So you’re forcing receiving software which wants to be conformant to use a hand-rolled parser which will probably have lousy performance and have other bugs which in fact may compromise security more than dupe-key tricks could.  -T

--
James Manger

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 27 Jun 2013 09:16:10 +0200

On Jun 27, 2013, at 07:26, "Manger, James H" <[email-masked]> wrote:

> each recipient MUST either reject messages with duplicate names or accept only the last name.

Maybe only gatekeeper functions need validating parsers?

gatekeeper function: A processing step that derives (makes/validates)
an assertion from a JOSE object and then moves forward that JOSE
object unchanged to the next processing step (e.g., to preserve a
signature)

validating parser: A JSON parser that rejects JSON that might be
interpreted differently by another JSON parser

Are we sure that the duplicate key issue is the only one that needs to
be checked by a validating parser?  A number of candidate issues:

http://www.ietf.org/mail-archive/web/json/current/msg01068.html

E.g., what about issue 3.8?  If a gatekeeper function derives an
asssertion about a JOSE object mentioning twitter ID
10765432100123456789, but the next step thinks it is about
10765432100123458000?

https://dev.twitter.com/docs/twitter-ids-json-and-snowflake

The list in msg01068 also is not complete for our purposes, e.g., it
mentions normalization issues only in passing.  A validating parser
sure wants to reject anything that isn't already normalized Unicode.

Maybe gatekeeper functions are not such a bright idea?

Grüße, Carsten

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 27 Jun 2013 18:08:55 +1000

> Maybe only gatekeeper functions need validating parsers?

Doubt it. How can a JOSE library tell if it will ever be used by a gateway... better treat all libraries as potential gateways.

> an assertion from a JOSE object and then moves forward that JOSE object
> unchanged to the next processing step (e.g., to preserve a
> signature)
> 

But now a validating parser MUST reject dupes, as others might interpret them differently, so the validating parser cannot be built from many standard JSON libraries.

> 10765432100123456789, but the next step thinks it is about

That would be a problem, but one for the app using JOSE to fix, not JOSE itself. I don't think any JOSE fields use JSON numbers as integers; and if they did the JOSE spec should add a "MUST be under 2^53" rule. At least in this case it is clearer what is at fault.

> sure wants to reject anything that isn't already normalized Unicode.

Not sure if that is a problem. JOSE might avoid those issues (almost by accident) by only defining field names that use ASCII letters. If it turns out that a bunch of JSON parsers "legitimately" accept {"alg":"ABC","\u24D0\u24DB\u24D6":"XYZ"} as having an "alg" value of XYZ (due to some normalization) we would need to amend JOSE to explicitly forbid it. I doubt that isn't necessary.

["\u24D0\u24DB\u24D6" = "ⓐⓛⓖ"]

> Maybe gatekeeper functions are not such a bright idea?

They are everywhere. Firewalls, proxies, WAFs, gateways; any tiered architecture... 
Gateways are fine. Potentially ambiguous messages are the idea that is not so bright. 

It is not just "inline" gateways. You don't want your audit tool, or support tool, that might look at messages after they have been processed to see a different interpretation. You don't want a court disagreeing about a message years later.

--
James Manger

---

From: Tim Bray <[email-masked]>
Date: Thu, 27 Jun 2013 08:20:11 -0700

The approach just seems wrong.  If we require that conforming
implementations MUST NOT emit dupes, then authors of receiving software can
just pick the best-performing parser with the nicest API, because they
*all* perform correctly when there are no dupes.

Who’s going to own the responsibility for making the authoritative finding
that of the many JSON parsers in the Java ecosystem, these ones, but not
those ones, are usable in JOSE applications?  And suppose one that has been
previously OK is quietly optimized on GitHub to run 30% faster and as a
side-effect silently ignores dupes?

The benefit of using JSON is that there is widely deployed software to
handle it.  It is a known problem that sloppy spec drafting has allowed
various kinds of problems to occur when dupe keys are received. “Doctor, it
hurts when I do this.” “So... don’t do that.”

Are we also going to specify the behavior of receiving software when the
JSON has a misplaced quotation mark?  A missing trailing “}”?

 -T

On Wed, Jun 26, 2013 at 11:29 PM, Manger, James H <
[email-masked]> wrote:

> Most JSON libraries that silently accept [sob] dupe keys at least
> consistently use the last key. Allowing just that behaviour (or rejecting
> dupes), while forbidding acceptance of a non-last dupe (eg first key), is
> safe and allows most JSON libraries to be used. Since this accepts most
> libraries I think it is practical.****
>
> ** **
>
> Hence I suggest:****
>
> ** **
>
> >   A creator of a JOSE message MUST NOT put duplicate names in any JSON**
> **
>
> > object in a JOSE header.****
>
> >   To prevent attacks where a JOSE message is interpreted as different***
> *
>
> > valid messages by different recipients, each recipient MUST either****
>
> > reject messages with duplicate names or accept only the last name.****
>
> ** **
>
> This isn’t “truly predictable” as a message with dupes might be accepted
> or rejected. The crucial point is that if a message is accepted it is
> predictable.****
>
> ** **
>
> --****
>
> James Manger****
>
> ** **
>
> *From:* Tim Bray [mailto:[email-masked]]
> *Sent:* Thursday, 27 June 2013 4:16 PM
> *To:* Manger, James H
> *Cc:* Mike Jones; Jim Schaad; [email-masked];
> [email-masked]
>
> *Subject:* Re: [jose] #27: member names MUST be unique needs additional
> text****
>
> ** **
>
> than dupe-key tricks could.  -T****
>
> ** **
>
> [email-masked]> wrote:****
>
> > that adds value. -T****
> James Manger****
> ** **

---

From: John Bradley <[email-masked]>
Date: Thu, 27 Jun 2013 11:49:03 -0400

The problem is not wanting to leave a hole for malicious signers to create JWT that may be misinterpreted by the receiver.  Accidental interop problems from bad signers are not a big concern for me and are covered by MUST NOT emit dups.

Accepting malformed JSON as the envelope may lead to real security issues.

This may be more of an issue for the JSON encoding than the compact as there you need to be careful the body you process for the signature is the body that the application uses.

What happens if you receive.

{"protected":<integrity-protected shared header contents>",
      "unprotected":<non-integrity-protected shared header contents>",
      "payload":"<original payload contents>",
      "payload":"<bad payload contents inserted by attacker>"
      "signatures":[
       {"header":"<per-signature unprotected header 1 contents>",
        "signature":"<signature 1 contents>"},
       ...
       {"header":"<per-signature unprotected header N contents>",
        "signature":"<signature N contents>"}],
     }

If the JOSE library validates on the first version of "payload" and the application using a different JSON parser only sees the second "payload" we have a wrapping attack similar to xmldsig.

I can't off the top of my head think of what you could do in the compact serialization that would lead to a successful attack given that the signature is over the base64url encoded segments, that prevents attackers from modifying the JSON while maintaining integrity.

It is a real issue for the JSON serialization though.

John B.

On 2013-06-27, at 11:20 AM, Tim Bray <[email-masked]> wrote:

> The approach just seems wrong.  If we require that conforming implementations MUST NOT emit dupes, then authors of receiving software can just pick the best-performing parser with the nicest API, because they *all* perform correctly when there are no dupes.  
> 
> Who’s going to own the responsibility for making the authoritative finding that of the many JSON parsers in the Java ecosystem, these ones, but not those ones, are usable in JOSE applications?  And suppose one that has been previously OK is quietly optimized on GitHub to run 30% faster and as a side-effect silently ignores dupes?  
> 
> The benefit of using JSON is that there is widely deployed software to handle it.  It is a known problem that sloppy spec drafting has allowed various kinds of problems to occur when dupe keys are received. “Doctor, it hurts when I do this.” “So... don’t do that.”
> 
> Are we also going to specify the behavior of receiving software when the JSON has a misplaced quotation mark?  A missing trailing “}”?
> 
>  -T
> 
> 
> 
> On Wed, Jun 26, 2013 at 11:29 PM, Manger, James H <[email-masked]> wrote:
> On Wed, Jun 26, 2013 at 10:39 PM, Manger, James H <[email-masked]> wrote:
>

---

From: Mike Jones <[email-masked]>
Date: Thu, 27 Jun 2013 16:42:33 +0000

I've come to believe that James' proposed language does a good job of making security-appropriate restrictions while allowing the use of existing parsers.  It was:

A creator of a JOSE message MUST NOT put duplicate names in any JSON object in a JOSE header.  To prevent attacks where a JOSE message is interpreted as different valid messages by different recipients, each recipient MUST either reject messages with duplicate names or accept only the last name.

I might change "or accept only the last name" to "or use a parser that always uses the lexically last name for any duplicate member names, per [ECMAScript]" for clarity, but the intent is the same.

Would that work for people?

                                                            -- Mike

From: John Bradley [mailto:[email-masked]]
Sent: Thursday, June 27, 2013 8:49 AM
To: Tim Bray
Cc: Manger, James H; Mike Jones; Jim Schaad; [email-masked]; [email-masked]
Subject: Re: [jose] #27: member names MUST be unique needs additional text

      "signatures":[

       ...

     }

John B.

On 2013-06-27, at 11:20 AM, Tim Bray <[email-masked]<mailto:[email-masked]>> wrote:

Who's going to own the responsibility for making the authoritative finding that of the many JSON parsers in the Java ecosystem, these ones, but not those ones, are usable in JOSE applications?  And suppose one that has been previously OK is quietly optimized on GitHub to run 30% faster and as a side-effect silently ignores dupes?
The benefit of using JSON is that there is widely deployed software to handle it.  It is a known problem that sloppy spec drafting has allowed various kinds of problems to occur when dupe keys are received. "Doctor, it hurts when I do this." "So... don't do that."
Are we also going to specify the behavior of receiving software when the JSON has a misplaced quotation mark?  A missing trailing "}"?
 -T

On Wed, Jun 26, 2013 at 11:29 PM, Manger, James H <[email-masked]<mailto:[email-masked]>> wrote:
Most JSON libraries that silently accept [sob] dupe keys at least consistently use the last key. Allowing just that behaviour (or rejecting dupes), while forbidding acceptance of a non-last dupe (eg first key), is safe and allows most JSON libraries to be used. Since this accepts most libraries I think it is practical.

> reject messages with duplicate names or accept only the last name.

This isn't "truly predictable" as a message with dupes might be accepted or rejected. The crucial point is that if a message is accepted it is predictable.

--
James Manger

From: Tim Bray [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, 27 June 2013 4:16 PM
To: Manger, James H
Cc: Mike Jones; Jim Schaad; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

Well, you have a practical problem in that most implementers will want to use a standard JSON library, which is good practice because it will be well-debugged, and most libraries [sob] silently take care of dupe keys and don't have a way to tell the client software what happened. So if you want truly predictable behavior, you're forcing the use of hand-constructed JSON parsers. And that sucks, because getting good performance in JSON parsing is surprisingly hard, with dramatic performance differences between implementations. So you're forcing receiving software which wants to be conformant to use a hand-rolled parser which will probably have lousy performance and have other bugs which in fact may compromise security more than dupe-key tricks could.  -T

On Wed, Jun 26, 2013 at 10:39 PM, Manger, James H <[email-masked]<mailto:[email-masked]>> wrote:
> I think it's a nice clean minimal solution to say that producers MUST
> NOT generate dupes, end of story.  I don't think saying anything beyond
> that adds value. -T

--
James Manger

---

From: "Jim Schaad" <[email-masked]>
Date: Thu, 27 Jun 2013 09:49:38 -0700

I am attacking you.  Why would I care about what the restrictions on producers are?

From: Tim Bray [mailto:[email-masked]] 
Sent: Wednesday, June 26, 2013 10:24 PM
To: Mike Jones
Cc: Jim Schaad; [email-masked]; [email-masked]
Subject: Re: [jose] #27: member names MUST be unique needs additional text

I think it’s a nice clean minimal solution to say that producers MUST NOT generate dupes, end of story.  I don’t think saying anything beyond that adds value. -T

On Wed, Jun 26, 2013 at 9:07 PM, Mike Jones <[email-masked]> wrote:

                                -- Mike

---

From: "Jim Schaad" <[email-masked]>
Date: Thu, 27 Jun 2013 10:07:04 -0700

When the volume is high is when one needs to keep track of the what is going on.  If the volume is low then you can ignore it.

There have not been any recent discussions on parsers but I don't know what you mean by improved so I cannot make a statement one way or the other on that.  I have always had the mail moved into a folder and go in and read it in one big shot once or twice a day unless I am currently involved in a discussion.  I find that technique works well.

Jim

> Sent: Wednesday, June 26, 2013 9:07 PM
> platform parsers are deficient.  But I really hope that the JSON WG will have
> more sense than to allow the current madness to continue.  As I understood
> it, the primary motivation for the JSON WG in the first place was to change the
> SHOULD to a MUST.  If they don't do that, I'm really not sure what the point of
> it is.
> 
> 				-- Mike
> 
> P.S.  I am a subscriber to the JSON mailing list, but unfortunately, the volume
> got so high a few weeks ago that trying to keep up with it became an
> impediment to getting real work done.  At that point I created a rule to file
> those messages into a folder, which I only rarely look at.  Has that situation
> improved?
> 
> independent parser.  And in fact what you have stated is that there is now a
> requirement that a JOSE library supply a JSON parser.  Is that really what we
> want to say?  How far from creating a set of generator and parser
> > Since this is no longer a viable statement due to the state of parsers, we
> need to be more explicit and say what is going on.
> >

---

From: "Jim Schaad" <[email-masked]>
Date: Thu, 27 Jun 2013 10:10:12 -0700

Except for the parsers which only return the nth name (according to the JSON
list they exist) it at least provides a degree of predictability.

The JSON list would point out that you want to allow duplicate field names
for the purposes of putting comments into output using the name "//".

From: Mike Jones [mailto:[email-masked]] 
Sent: Thursday, June 27, 2013 9:43 AM
To: John Bradley; Tim Bray
Cc: Manger, James H; Jim Schaad; [email-masked];
[email-masked]
Subject: RE: [jose] #27: member names MUST be unique needs additional text

I've come to believe that James' proposed language does a good job of making
security-appropriate restrictions while allowing the use of existing
parsers.  It was:

A creator of a JOSE message MUST NOT put duplicate names in any JSON object
in a JOSE header.  To prevent attacks where a JOSE message is interpreted as
different valid messages by different recipients, each recipient MUST either
reject messages with duplicate names or accept only the last name.

I might change "or accept only the last name" to "or use a parser that
always uses the lexically last name for any duplicate member names, per
[ECMAScript]" for clarity, but the intent is the same.

                                                            -- Mike

Cc: Manger, James H; Mike Jones; Jim Schaad; [email-masked];

The problem is not wanting to leave a hole for malicious signers to create
JWT that may be misinterpreted by the receiver.  Accidental interop problems
from bad signers are not a big concern for me and are covered by MUST NOT
emit dups.

This may be more of an issue for the JSON encoding than the compact as there
you need to be careful the body you process for the signature is the body
that the application uses.

If the JOSE library validates on the first version of "payload" and the
application using a different JSON parser only sees the second "payload" we
have a wrapping attack similar to xmldsig.

I can't off the top of my head think of what you could do in the compact
serialization that would lead to a successful attack given that the
signature is over the base64url encoded segments, that prevents attackers
from modifying the JSON while maintaining integrity.

John B.

The approach just seems wrong.  If we require that conforming
implementations MUST NOT emit dupes, then authors of receiving software can
just pick the best-performing parser with the nicest API, because they *all*
perform correctly when there are no dupes.  

Who's going to own the responsibility for making the authoritative finding

The benefit of using JSON is that there is widely deployed software to
handle it.  It is a known problem that sloppy spec drafting has allowed
various kinds of problems to occur when dupe keys are received. "Doctor, it
hurts when I do this." "So... don't do that."

Are we also going to specify the behavior of receiving software when the
JSON has a misplaced quotation mark?  A missing trailing "}"?

 -T

On Wed, Jun 26, 2013 at 11:29 PM, Manger, James H
<[email-masked]> wrote:

libraries I think it is practical.

> reject messages with duplicate names or accept only the last name.

This isn't "truly predictable" as a message with dupes might be accepted or
rejected. The crucial point is that if a message is accepted it is
predictable.

--

James Manger

From: Tim Bray [mailto:[email-masked]] 
Sent: Thursday, 27 June 2013 4:16 PM
To: Manger, James H
Cc: Mike Jones; Jim Schaad; [email-masked];
[email-masked]

 

Well, you have a practical problem in that most implementers will want to
use a standard JSON library, which is good practice because it will be
well-debugged, and most libraries [sob] silently take care of dupe keys and
don't have a way to tell the client software what happened. So if you want
truly predictable behavior, you're forcing the use of hand-constructed JSON
parsers. And that sucks, because getting good performance in JSON parsing is
surprisingly hard, with dramatic performance differences between
implementations. So you're forcing receiving software which wants to be
conformant to use a hand-rolled parser which will probably have lousy
performance and have other bugs which in fact may compromise security more
than dupe-key tricks could.  -T

Clean and minimal that may be, but it ignores the security issue. We don't
want a malicious producer (who is so malicious they ignore a MUST) to create
JOSE messages that a JOSE-compliant security layer accepts as "benign
interpretation #1" so it passes the message on to the JOSE-compliant backend
app that acts on "nasty interpretation #2".

--
James Manger

---

From: Richard Barnes <[email-masked]>
Date: Thu, 27 Jun 2013 18:47:36 -0400

On Thu, Jun 27, 2013 at 12:42 PM, Mike Jones <[email-masked]>wrote:

>  I’ve come to believe that James’ proposed language does a good job of
> making security-appropriate restrictions while allowing the use of existing
> parsers.  It was:****
>
> ** **
>
> object in a JOSE header.  To prevent attacks where a JOSE message is
> interpreted as different valid messages by different recipients, each
> recipient MUST either reject messages with duplicate names or accept only
> the last name.****
>
> ** **
>
> I might change “or accept only the last name” to “or use a parser that
> [ECMAScript]” for clarity, but the intent is the same.****
>
> ** **
>
> Would that work for people?****
>
> ** **
>
>                                                             -- Mike
>

That's close.  I don't think we can allow the "lexically last" because
elements in a JSON dictionary can be re-ordered.  Suppose there a helpful
intermediary that sorts a JSON dictionary in descending alphabetical order
of values.
When it gets: { "payload": "bad stuff",  "payload": "good stuff", ... }
It emits: { "payload": "good stuff",  "payload": "bad stuff", ... }

The only way to avoid the ambiguity is to reject objects with duplicate
keys.

I realize that this renders most JSON parsers incompatible.  Implementors
will need to add a secondary check.  I feel your pain.  But hopefully this
pain will be temporary; the ambiguity will be fixed by the JSON WG and
ECMA, and parsers will be updated.  Even in the short run, it doesn't seem
like it would be that hard to mod existing parsers to have a "strict" flag.

"""
A creator of a JOSE message MUST NOT put duplicate names in any JSON object
in a JOSE header.  To prevent attacks where a JOSE message is interpreted
as different valid messages by different recipients, each recipient MUST
reject messages.  Implementations MUST NOT select the lexically last name,
as required by [ECMAScript], since this could lead them to select the
incorrect value if an intermediary re-orders the fields in a dictionary.
 If an implementation uses a JSON parser that does not reject objects with
duplicate fields, then it MUST verify that there are no dictionaries with
duplicate fields in the JSON object.
"""

--Richard

> ****
>
> ** **
>
> *From:* John Bradley [mailto:[email-masked]]
> *Sent:* Thursday, June 27, 2013 8:49 AM
> *To:* Tim Bray
> *Cc:* Manger, James H; Mike Jones; Jim Schaad; [email-masked];
> JWT that may be misinterpreted by the receiver.  Accidental interop
> problems from bad signers are not a big concern for me and are covered by
> MUST NOT emit dups.****
>
> ** **
>
> ****
> This may be more of an issue for the JSON encoding than the compact as
> there you need to be careful the body you process for the signature is the
> body that the application uses.****
>
> ** **
>
> What happens if you receive.****
>
> ** **
>
> {"protected":<integrity-protected shared header contents>",****
>
>       "unprotected":<non-integrity-protected shared header contents>",****
>
>       "payload":"<original payload contents>",****
>
>       "payload":"<bad payload contents inserted by attacker>"****
>
>       "signatures":[****
>
>        {"header":"<per-signature unprotected header 1 contents>",****
>
>         "signature":"<signature 1 contents>"},****
>
>        ...****
>
>        {"header":"<per-signature unprotected header N contents>",****
>
>         "signature":"<signature N contents>"}],****
>
>      }****
>
>  ** **
>
> have a wrapping attack similar to xmldsig.****
>
> ** **
>
> from modifying the JSON while maintaining integrity.****
>
> ** **
>
> It is a real issue for the JSON serialization though.****
>
> ** **
>
> John B.****
>
> ** **
>
> On 2013-06-27, at 11:20 AM, Tim Bray <[email-masked]> wrote:****
>
>
>
> ****
>
> side-effect silently ignores dupes?  ****
>
> hurts when I do this.” “So... don’t do that.”****
>
> JSON has a misplaced quotation mark?  A missing trailing “}”?****
>
>  -T****
>
> ** **
>
> ** **
>
>  ****
>  ****
>  ****
>  ****
>  ****
> [email-masked]****
>
>
>  ****
>  ****
>  ****

---

From: Richard Barnes <[email-masked]>
Date: Thu, 27 Jun 2013 18:55:56 -0400

In a JSON-formatted JWS, the top-level JSON fields are not covered by the
signature (only the protected header).  See the examples in my previous
mail and in John's.
--Richard

On Thu, Jun 27, 2013 at 6:51 PM, Mike Jones <[email-masked]>wrote:

>  If an intermediary reorders the fields the signature validation will
> fail, so I’m not worried about that.****
>
> ** **
>
>                                                                 -- Mike***
> *
>
> ** **
>
> *From:* Richard Barnes [mailto:[email-masked]]
> *Sent:* Thursday, June 27, 2013 3:48 PM
> *To:* Mike Jones
> *Cc:* John Bradley; Tim Bray; Jim Schaad; Manger, James H; [email-masked];
> On Thu, Jun 27, 2013 at 12:42 PM, Mike Jones <[email-masked]>
> wrote:****
>
>                                                             -- Mike****
> of values. ****
> When it gets: { "payload": "bad stuff",  "payload": "good stuff", ... }***
> *
>
> It emits: { "payload": "good stuff",  "payload": "bad stuff", ... }****
>
> ** **
>
> keys.****
> """****
> recipient MUST reject messages.  Implementations MUST NOT select the
> lexically last name, as required by [ECMAScript], since this could lead
> them to select the incorrect value if an intermediary re-orders the fields
> in a dictionary.  If an implementation uses a JSON parser that does not
> reject objects with duplicate fields, then it MUST verify that there are no
> dictionaries with duplicate fields in the JSON object.****
>
> """****
>
> ** **
>
> --Richard ****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
>  ****
>
>   ****
>
> https://www.ietf.org/mailman/listinfo/jose****
>
>  ** **
>

---

From: Mike Jones <[email-masked]>
Date: Thu, 27 Jun 2013 23:00:17 +0000

Yes - but this is only a problem for the JSON Serialization - and then, only for fields that are not integrity protected.

If you don't care enough about the content of the header fields to integrity protect them, you clearly don't care whether an intermediary can tamper with them.  Reordering duplicate member names is the only the tip of the iceberg of the tampering that not integrity protecting header fields enables.  The intermediary could delete them or completely rewrite them!

Defending only against field reordering is like putting your finger in a hole in the dike when the floodgates next door are already wide open.  Your field will definitely flood when the sea rises.

                                                                -- Mike

From: Richard Barnes [mailto:[email-masked]]
Sent: Thursday, June 27, 2013 3:56 PM
To: Mike Jones
Cc: John Bradley; Tim Bray; Jim Schaad; Manger, James H; [email-masked]; [email-masked]
Subject: Re: [jose] #27: member names MUST be unique needs additional text

In a JSON-formatted JWS, the top-level JSON fields are not covered by the signature (only the protected header).  See the examples in my previous mail and in John's.
--Richard

On Thu, Jun 27, 2013 at 6:51 PM, Mike Jones <[email-masked]<mailto:[email-masked]>> wrote:
If an intermediary reorders the fields the signature validation will fail, so I'm not worried about that.

                                                                -- Mike

From: Richard Barnes [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, June 27, 2013 3:48 PM
To: Mike Jones
Cc: John Bradley; Tim Bray; Jim Schaad; Manger, James H; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

On Thu, Jun 27, 2013 at 12:42 PM, Mike Jones <[email-masked]<mailto:[email-masked]>> wrote:
I've come to believe that James' proposed language does a good job of making security-appropriate restrictions while allowing the use of existing parsers.  It was:

                                                            -- Mike

That's close.  I don't think we can allow the "lexically last" because elements in a JSON dictionary can be re-ordered.  Suppose there a helpful intermediary that sorts a JSON dictionary in descending alphabetical order of values.
When it gets: { "payload": "bad stuff",  "payload": "good stuff", ... }
It emits: { "payload": "good stuff",  "payload": "bad stuff", ... }

The only way to avoid the ambiguity is to reject objects with duplicate keys.

I realize that this renders most JSON parsers incompatible.  Implementors will need to add a secondary check.  I feel your pain.  But hopefully this pain will be temporary; the ambiguity will be fixed by the JSON WG and ECMA, and parsers will be updated.  Even in the short run, it doesn't seem like it would be that hard to mod existing parsers to have a "strict" flag.

"""
A creator of a JOSE message MUST NOT put duplicate names in any JSON object in a JOSE header.  To prevent attacks where a JOSE message is interpreted as different valid messages by different recipients, each recipient MUST reject messages.  Implementations MUST NOT select the lexically last name, as required by [ECMAScript], since this could lead them to select the incorrect value if an intermediary re-orders the fields in a dictionary.  If an implementation uses a JSON parser that does not reject objects with duplicate fields, then it MUST verify that there are no dictionaries with duplicate fields in the JSON object.
"""

--Richard

From: John Bradley [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, June 27, 2013 8:49 AM
To: Tim Bray
Cc: Manger, James H; Mike Jones; Jim Schaad; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

      "signatures":[

       ...

     }

John B.

--
James Manger

--
James Manger

_______________________________________________
jose mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/jose

---

From: Mike Jones <[email-masked]>
Date: Thu, 27 Jun 2013 22:51:47 +0000

-- Mike

                                                            -- Mike

"""

--Richard

      "signatures":[

       ...

     }

John B.

--
James Manger

--
James Manger

---

From: Richard Barnes <[email-masked]>
Date: Thu, 27 Jun 2013 19:10:34 -0400

The goal here is not to defend against attackers.  It's to prevent benign
intermediaries from causing breakage.

The point here is that a JOSE implementation needs to get the same
understanding of a JOSE object regardless of which of the multiple
equivalent forms of the JSON form it gets.  This is about having a
well-specified protocol, for which transformations that produce equivalent
JSON objects should produce equivalent JOSE objects.  That is, the
following process should be lossless:
JOSE --stringify--> JSON --reorder-fields--> JSON --parse--> JOSE

If that's not the case, then we've built a broken protocol.

--Richard

On Thu, Jun 27, 2013 at 7:00 PM, Mike Jones <[email-masked]>wrote:

>  Yes – but this is only a problem for the JSON Serialization – and then,
> only for fields that are not integrity protected.****
>
> ** **
>
> If you don’t care enough about the content of the header fields to
> integrity protect them, you clearly don’t care whether an intermediary can
> tamper with them.  Reordering duplicate member names is the only the tip of
> the iceberg of the tampering that not integrity protecting header fields
> enables.  The intermediary could delete them or completely rewrite them!**
> **
>
> ** **
>
> Defending only against field reordering is like putting your finger in a
> hole in the dike when the floodgates next door are already wide open.  Your
> field will definitely flood when the sea rises.****
>
> ** **
>
>                                                                 -- Mike***
> *
>
> ** **
>
> *Sent:* Thursday, June 27, 2013 3:56 PM
>
> *To:* Mike Jones
> mail and in John's.****
> --Richard****
> On Thu, Jun 27, 2013 at 6:51 PM, Mike Jones <[email-masked]>
> wrote:****
>
> If an intermediary reorders the fields the signature validation will fail,
> so I’m not worried about that.****
>
>  ****
>
>                                                                 -- Mike***
> *
>
>  ****
>

---

From: Mike Jones <[email-masked]>
Date: Thu, 27 Jun 2013 23:18:34 +0000

It's not lossless only for illegal inputs - those with duplicate members.  If your goal isn't to defend against attackers, then it's safe to assume that the producer produced a legal input - hence no problem in practice.

From: Richard Barnes [mailto:[email-masked]]
Sent: Thursday, June 27, 2013 4:11 PM
To: Mike Jones
Cc: John Bradley; Tim Bray; Jim Schaad; Manger, James H; [email-masked]; [email-masked]
Subject: Re: [jose] #27: member names MUST be unique needs additional text

The goal here is not to defend against attackers.  It's to prevent benign intermediaries from causing breakage.

The point here is that a JOSE implementation needs to get the same understanding of a JOSE object regardless of which of the multiple equivalent forms of the JSON form it gets.  This is about having a well-specified protocol, for which transformations that produce equivalent JSON objects should produce equivalent JOSE objects.  That is, the following process should be lossless:
JOSE --stringify--> JSON --reorder-fields--> JSON --parse--> JOSE

--Richard

On Thu, Jun 27, 2013 at 7:00 PM, Mike Jones <[email-masked]<mailto:[email-masked]>> wrote:
Yes - but this is only a problem for the JSON Serialization - and then, only for fields that are not integrity protected.

                                                                -- Mike

To: Mike Jones

                                                                -- Mike

                                                            -- Mike

"""

--Richard

      "signatures":[

       ...

     }

John B.

--
James Manger

--
James Manger