Subject: Re: [jose] Whether implementations must understand all JOSE header fields

From: Mike Jones <[email-masked]>
Date: Tue, 18 Dec 2012 22:03:44 +0000

Hi James,

Per the response I just sent to Axel, when we discussed this issue in the past, only one person was in favor of a must-ignore-by-default behavior, and yet that’s what you’re advocating below.  See http://www.ietf.org/mail-archive/web/jose/current/msg01216.html.  Of course, it’s always possible you could change minds, but I suspect that if you want to influence the outcome of this issue, the highest-payoff strategy would be to discuss syntax options that require people to take special action to ignore a not-understood field, rather than the other way around.

                                                                My two cents worth…
                                                                -- Mike

From: Manger, James H [mailto:[email-masked]]
Sent: Tuesday, December 18, 2012 4:59 AM
To: Mike Jones; [email-masked]
Subject: RE: [jose] Whether implementations must understand all JOSE header fields

My comments in red.

Currently the JOSE specs contain the following language declaring that all header (and some other) fields must be understood:  “Implementations MUST understand the entire contents of the header; otherwise, the … MUST be rejected.”  There's currently an open issue about whether all header fields should be required to be understood by implementations using them or whether to relax this requirement in some circumstances.  We would like the working group to consider the following potential resolutions to the issue.

1.  Maintain the current requirement that all header fields must be understood by implementations using them.

PRO:  This is a simple rule…

It is not as simple as ignoring unknown fields, which all code will do naturally.

Are the “MUST understand or reject” rules well defined? Obviously if a totally unknown field is present the JOSE message must be rejected.

How about if a defined field is in an unexpected place, eg “zip” in a JWE header? Presumably that has to be rejected so it is not sufficient for code to check any JOSE messages against one list.

How about a JWE with an “epk” field when “alg”:”RSA-OAEP”? Presumably that also has to be rejected so checking against a static list of JWE fields is not sufficient.

Does the type or value of a field need to be checked if an implementation doesn’t need to use it? For example, does an implementation that already has a given cert (so it doesn’t need to use a “x5u” value) need to confirm that the “x5u” value is a string, or is a valid URI, or actually resolves to the same cert? Probably not. I assume it is sufficient to understand that a field can be ignored, without actually understanding its actual syntax or value is a specific message.

…designed to result in secure implementations.

Secure in what way? Is this aimed specifically at preventing new fields being ignored in case they change the semantics of existing fields?

PRO: New fields are allowed to change the semantics of existing fields because implementations not understanding the changed semantics will fail-safe due to seeing an unknown field.

(or is this a CON for encouraging dangerous extensions?)

…Representations of existing JWS & JWE objects are unchanged.

CON:  Extensibility is limited to cases where all participating parties understand new fields that are introduced.

CON: Because code can successfully process JOSE message without implementing the “MUST understand or reject” rule it is hard to have confidence that it is implemented (or implemented well).

2A.  Take an alternative approach where, by default, all fields must be understood, but where an explicit list of fields can be designated as “may be ignored if not understood”.  For instance, an “ign” (may be ignored) member could be defined whose value is an array listing the member names that may be safely ignored if not understood.  An example using this field in a JWS header is:

    {"alg":"ES256",

     "ign":["notes"],

     "notes":"This signature need not be checked when the moon is full"

    }

(Obviously, the "ign" field MUST NOT contain the value "ign".)

PRO:  This would enable adding non-critical header fields without breaking implementations.  It would maintain the current semantics, while allowing explicit exceptions to be made in cases where the fields are not security-critical and the structure has a meaningful interpretation without them.  Representations of existing JWS & JWE objects are unchanged.

CON:  Requires coordination of field names and may-be-ignored list, adding some implementation complexity.

2B.  Take an alternative approach where instead of one header object, there would be two header objects.  Implementations would be required to understand all fields in the first header object.  Implementations would be allowed to ignore fields in the second header object if not understood.  An example of this approach, adding the second header object after the first in a JWS, would be:

    eyJ0eXAiOiJKV1QiLA0KICJhbGciOiJIUzI1NiJ9

    .

    eyJub3RlcyI6IlRoaXMgc2lnbmF0dXJlIG5lZWQgbm90IGJlIGNoZWNrZWQgd2hlbiB0aGUgbW9vbiBpcyBmdWxsIn0

    .

    eyJpc3MiOiJqb2UiLA0KICJleHAiOjEzMDA4MTkzODAsDQogImh0dHA6Ly9leGFtcGxlLmNvbS9pc19yb290Ijp0cnVlfQ

    .

    dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk

The first, third, and fourth fields are from the example JWS in the current spec. The new second field is the base64url encoded representation of the string '{"notes":"This signature need not be checked when the moon is full"}'.

PRO:  May-be-ignored fields are easily distinguishable from must-be-understood fields.

CON:  Implementations must consult/merge two sets of headers to understand the meaning/structure of the object, adding some implementation complexity.  Existing JWS & JWE object representations would become invalid.

3. Have a field that indicates the semantics of the JOSE message. For instance the field could be called “!” and defined values could be “sig”, “mac”, “enc”… for signed-with-an-asymmetric-key, MACed-with-a-shared-secret, encrypted-(AEAD)-with-shared-secret…. The definition of a value for the “!” field indicates the other fields it uses. Recipients MUST ignore any unrecognized fields. Originators adding a new field that changes the semantics of a message MUST use a new value for the “!” field.

PRO: Code will implement this rule naturally, simply by switching on the “!” field value.

PRO: It is extensible. Extra fields can be defined and deployed without requiring originators and recipients to be upgraded simultaneously.

CON: Arbitrary extra data of a originator’s choosing can be included as unknown fields in messages that recipients will accept. Though this is true even with a “MUST understand or reject” rule (eg whitespace in the JSON header, junk in a “x5u” URI fragment…).

After a discussion period, we believe that it would be useful for the chairs to make two formal consensus calls of the following kind:

FIRST POLL:  Should all header fields be critical for implementations to understand?

YES – All header fields must continue to be understood by implementations or the input must be rejected.

NO – A means of listing that specific header fields may be safely ignored should be defined.

ALTERNATIVE FIRST POLL FOR EXISTING IMPLEMENTORS: Have you implemented the “MUST understand or reject” rule? Which situations do you reject (eg “foo”; “zip” in JWS; “epk” when “alg”:”RSA-OAEP”; “jku”:<non-string>; “jku”:”<invalid URI>”; …?

ALTERNATIVE SECOND POLL: How real is the risk that future groups will deliberately design & deploy new fields that change the semantics of JOSE messages depending on whether or not the new field is understood, without realizing the security consequences?

ALTERNATIVE THIRD POLL: Is JOSE so complete and stable that extensions are unlikely to be needed?

SECOND POLL:  Should the result of the first poll be "NO", which syntax would you prefer for designating the header fields that may be ignored if not understood?

A – Define a header field that explicitly lists the fields that may be safely ignored if not understood.

B – Introduce a second header, where implementations must understand all fields in the first but they may ignore not-understood fields in the second.

                                                Thanks for considering these questions,

                                                Richard Barnes, John Bradley, Mike Jones

--
James Manger

---

From: Richard Barnes <[email-masked]>
Date: Tue, 18 Dec 2012 17:27:03 -0500

Maybe my opinion wasn't reflected clearly, but I agree with James that "ignore by default" is the reality for basically everything these days, especially at the formatting layer, where this stuff sits.  Your average JSON parser isn't even paying attention to what the field names are.  Even XML schemas, which can supposedly reject unknowns, are littered with wide-open extension points in practice (<xs:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/>).  

Applications can always require that certain fields be present.  Having any notion of criticality at this layer is only a convenience that saves applications from having to fix or pre-negotiate criticality values.

(And, before it comes up: PKIX is an application protocol.  CMS has no notion of criticality.)  

--Richard

On Dec 18, 2012, at 5:03 PM, Mike Jones <[email-masked]> wrote:

> _______________________________________________
> jose mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jose

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 19 Dec 2012 09:41:46 +1100

Mike,

Could you at least expand a bit on how you expect a “MUST understand or reject” rule will work in practice?
For instance, does the spec wording forbid “zip” in a JWS? “epk” when “alg”:”RSA-OAEP”? “x5u”:7? “x5u”:”hello world”? Will implementations be consistent?

Perhaps you have an implementation of this rule already and can tell us what it does?

I would also appreciate your thoughts on whether someone can extend JWS by specifying: when the “zip” field is present in a JWS the content is compressed.
That looks legitimate with the current “MUST understand or reject” rule. I think it is likely, however, that such an extension will cause security problems. Some implementations will confirm that “zip” is on the list of known standard fields, but will ignore it in the JWS processing code.

--
James Manger

From: Mike Jones [mailto:[email-masked]]
Sent: Wednesday, 19 December 2012 9:04 AM
To: Manger, James H; [email-masked]
Subject: RE: [jose] Whether implementations must understand all JOSE header fields

Hi James,

                                                                My two cents worth…
                                                                -- Mike

From: Manger, James H [mailto:[email-masked]]
Sent: Tuesday, December 18, 2012 4:59 AM
To: Mike Jones; [email-masked]<mailto:[email-masked]>
Subject: RE: [jose] Whether implementations must understand all JOSE header fields

My comments in red.

    {"alg":"ES256",

     "ign":["notes"],

    }

    .

    .

    .

--
James Manger

---

From: Mike Jones <[email-masked]>
Date: Tue, 18 Dec 2012 23:23:03 +0000

Yes, I believe the spec wording does forbid “zip” in a JWS, because it’s not a field with a defined meaning in that context – just as “epk” has no defined meaning with “alg”:”RSA-OAEP”.  I agree that more explicit wording to that effect can and probably should be added.  If you’d like to suggest some, please have at it.

One would define “zip” in a JWS in the normal way that extensions are made to IETF specs – you’d write a spec defining the meaning for it and register the “zip” value in the IANA JSON Web Signature and Encryption Header Parameters registry for JWS usage.

                                                                Best wishes,
                                                                -- Mike

From: Manger, James H [mailto:[email-masked]]
Sent: Tuesday, December 18, 2012 2:42 PM
To: Mike Jones; [email-masked]
Subject: RE: [jose] Whether implementations must understand all JOSE header fields

Mike,

--
James Manger

From: Mike Jones [mailto:[email-masked]]
Sent: Wednesday, 19 December 2012 9:04 AM
To: Manger, James H; [email-masked]<mailto:[email-masked]>
Subject: RE: [jose] Whether implementations must understand all JOSE header fields

Hi James,

                                                                My two cents worth…
                                                                -- Mike

My comments in red.

    {"alg":"ES256",

     "ign":["notes"],

    }

    .

    .

    .

--
James Manger

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 19 Dec 2012 16:12:03 +1100

> One would define “zip” in a JWS in the normal way that extensions are made to IETF specs – you’d write a spec defining the meaning for it and register the “zip” value in the IANA JSON Web Signature…

Lets ignore the process for now (IETF, IANA…), the issue is whether this style of extension is sensible: an extension that indicates a change of semantics by the presence of a new field. A style that the “MUST understand everything” rule encourages.
My argument is that this style is too dangerous a way to indicate new semantics. When changing semantics (as “zip” does) it would be much better to change a field we are confident implementations will be looking at, as opposed to relying on them to notice a new field.

--
James Manger

---

From: Dirkjan Ochtman <[email-masked]>
Date: Wed, 19 Dec 2012 09:29:36 +0100

On Wed, Dec 19, 2012 at 6:12 AM, Manger, James H
<[email-masked]> wrote:
> Lets ignore the process for now (IETF, IANA…), the issue is whether this
> style of extension is sensible: an extension that indicates a change of
> semantics by the presence of a new field. A style that the “MUST understand
> everything” rule encourages.
>
> My argument is that this style is too dangerous a way to indicate new
> semantics. When changing semantics (as “zip” does) it would be much better
> to change a field we are confident implementations will be looking at, as
> opposed to relying on them to notice a new field.

I fully agree with this.

I think the better style is somewhat like this: implementation MUST
use the value of the "alg" header to determine how to interpret the
contents. Any possible alg value may have different header fields that
are required to implement that algorithm. This may be too simplistic,
but it's much more in line with what implementors are prone to doing,
and closer to the expected semantics of JSON processing.

Cheers,

Dirkjan

---

From: Mike Jones <[email-masked]>
Date: Wed, 19 Dec 2012 19:48:23 +0000

“zip” is a perfect example of indicating a change of semantics with the presence of a new field.  The processing of a JWE without a “zip” field is different than the processing of it with one.  An implementation must understand the field to use the resulting JWE.  The same would be true of any JWS that used a “zip” extension.

It would never be safe to ignore this field, whether defined as part of the base spec, or defined as an extension later.

                                                                -- Mike

P.S.  On a related topic, see the thread “Reducing the size of JWS payloads” for a discussion on why compression won’t work for JWS payloads that are JWEs.

From: Manger, James H [mailto:[email-masked]]
Sent: Tuesday, December 18, 2012 9:12 PM
To: Mike Jones; [email-masked]
Subject: RE: [jose] Whether implementations must understand all JOSE header fields

--
James Manger

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 20 Dec 2012 12:39:21 +1100

> It would never be safe to ignore this field, whether defined as part of the base spec, or defined as an extension later.

I agree.
When I looked at a handful of publicly available JOSE implementations quite a while ago none enforced the “MUST understand everything” rule — so a zip-for-JWS extension would be dangerous for them. One implementation later added code to enforce the rule using one fixed list of all header fields defined in JW* — so a zip-for-JWS extension would be dangerous for that too.

We know the “MUST understand everything” rule makes it hard to deploy non-critical extensions.
It seems to me that the rule also makes it dangerous to deploy critical extensions.
About the only safe way to define a critical extension will be to define a prefix for “alg” values (eg "alg": "zip;RSA1_5"). I hope that isn’t the most practical option we leave for anyone wanting to extend JOSE.

--
James Manger

---

From: Mike Jones <[email-masked]>
Date: Thu, 20 Dec 2012 01:51:20 +0000

The prefix for “alg” values would have exactly the same property as adding a new header parameter.  If the new alg value was not understood, the JWS would be useless to the receiver.  (Just as it would be useless if a new header value was used, if not understood.)  I’m not sure why the former is any more usable than the latter.

                                                                Cheers,
                                                                -- Mike

From: Manger, James H [mailto:[email-masked]]
Sent: Wednesday, December 19, 2012 5:39 PM
To: Mike Jones; [email-masked]
Subject: RE: [jose] Whether implementations must understand all JOSE header fields

I agree.

--
James Manger

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 20 Dec 2012 13:08:20 +1100

> The prefix for “alg” values would have exactly the same property as adding a new header parameter. If the new alg value was not understood, the JWS would be useless to the receiver.

I am very confident that every (reasonable) JOSE implementation will notice the “alg” value, ie fail-safe on an unrecognized “alg” value.

> (Just as it would be useless if a new header value was used, if not understood.)  I’m not sure why the former is any more usable than the latter.

I have far far less confidence that every (reasonable) JOSE implementation will notice an unexpected field, particularly a known field in an unexpected context.

--
James Manger
                                                                Cheers,
                                                                -- Mike

I agree.

--
James Manger

---

From: Mike Jones <[email-masked]>
Date: Thu, 20 Dec 2012 02:12:02 +0000

Fair enough, but I am also very confident that every implementation trying to use a compressed payload where it was expecting an uncompressed one would quickly notice the difference and then do what it takes to get their implementation fixed/updated. ☺

                                                            -- Mike

From: Manger, James H [mailto:[email-masked]]
Sent: Wednesday, December 19, 2012 6:08 PM
To: Mike Jones; [email-masked]
Subject: RE: [jose] Whether implementations must understand all JOSE header fields

--
James Manger
                                                                Cheers,
                                                                -- Mike

I agree.

--
James Manger

---

From: Dirkjan Ochtman <[email-masked]>
Date: Thu, 20 Dec 2012 08:47:02 +0100

Yeah, that makes sense to me.

Also, even having a small set of must-understand fields (alg and zip
and maybe 1 or 2 others) is very different from having a
must-understand-all policy.

Cheers,

Dirkjan

On Thu, Dec 20, 2012 at 3:12 AM, Mike Jones <[email-masked]> wrote:
> Fair enough, but I am also very confident that every implementation trying
> to use a compressed payload where it was expecting an uncompressed one would
> quickly notice the difference and then do what it takes to get their
> implementation fixed/updated. J
>
>
>
>                                                             -- Mike
>
>
>
>> The prefix for “alg” values would have exactly the same property as adding
>> a new header parameter. If the new alg value was not understood, the JWS
>> would be useless to the receiver.
>
>
>
> I am very confident that every (reasonable) JOSE implementation will notice
> the “alg” value, ie fail-safe on an unrecognized “alg” value.
>
>
>
>> (Just as it would be useless if a new header value was used, if not
>> understood.)  I’m not sure why the former is any more usable than the
>> latter.
>
>
>
> I have far far less confidence that every (reasonable) JOSE implementation
> will notice an unexpected field, particularly a known field in an unexpected
> context.
>
>
>
> --
>
> James Manger
>
>                                                                 Cheers,
>
>                                                                 -- Mike
>
>
>
>> “zip” is a perfect example of indicating a change of semantics with the
>> presence of a new field.  The processing of a JWE without a “zip” field is
>> different than the processing of it with one.  An implementation must
>> understand the field to use the resulting JWE.  The same would be true of
>> any JWS that used a “zip” extension.
>
>>
>
>> It would never be safe to ignore this field, whether defined as part of
>> the base spec, or defined as an extension later.
>
>
>
> I agree.
>
> When I looked at a handful of publicly available JOSE implementations quite
> a while ago none enforced the “MUST understand everything” rule — so a
> zip-for-JWS extension would be dangerous for them. One implementation later
> added code to enforce the rule using one fixed list of all header fields
> defined in JW* — so a zip-for-JWS extension would be dangerous for that too.
>
>
>
> About the only safe way to define a critical extension will be to define a
> prefix for “alg” values (eg "alg": "zip;RSA1_5"). I hope that isn’t the most
> practical option we leave for anyone wanting to extend JOSE.
>
>
>
> --
>
> James Manger
>
>