Subject: [OAUTH-WG] Signatures spec proposal, take 2

From: Dirk Balfanz <[email-masked]>
Date: Thu, 23 Sep 2010 15:39:29 -0700

Hi guys,

sorry it took a while, but here is an updated proposal. It's still in three
parts:

Part I is about "JSON Tokens" that can be used for all sorts of things, not
just OAuth:
http://balfanz.github.com/jsontoken-spec/draft-balfanz-jsontoken-00.html

Part II is about how to embed an OAuth token and (some parts of) an HTTP
request into a JSON Token:
http://balfanz.github.com/jsontoken-spec/draft-balfanz-signedoauth2-00.html

Part III is how to use signatures instead of client secrets for assertions
in OAuth:
http://balfanz.github.com/jsontoken-spec/draft-balfanz-clientassertions-00.html

Diffs from the last specs are:

- JSON Tokens are now just a profile of Magic Signatures, which John Panzer
has helpfully extended for this purpose
- There was a vulnerability to masquerading attacks in the last proposal,
which is addressed in this proposal by adding a data_type parameter that is
part of the signature, but _not_ part of the payload.
- no more support of X.509 certs - the only supported format for discovered
public keys is now the Magic Key format. We'll give people tools (which are
quite easy to write) to convert their self-signed or CA-issued certs to
magic keys.
- The specs are now formatted as I-Ds.

Comments, please!

Dirk.

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Thu, 23 Sep 2010 18:51:28 -0700

In part II, how is the signature bound to the HTTP request URI? I see the method and body, but not the request URI.

EHL

On 9/23/10 3:39 PM, "Dirk Balfanz" <[email-masked]> wrote:

Hi guys,

sorry it took a while, but here is an updated proposal. It's still in three parts:

Part I is about "JSON Tokens" that can be used for all sorts of things, not just OAuth:
http://balfanz.github.com/jsontoken-spec/draft-balfanz-jsontoken-00.html

Part II is about how to embed an OAuth token and (some parts of) an HTTP request into a JSON Token:
http://balfanz.github.com/jsontoken-spec/draft-balfanz-signedoauth2-00.html

Part III is how to use signatures instead of client secrets for assertions in OAuth:
http://balfanz.github.com/jsontoken-spec/draft-balfanz-clientassertions-00.html

- JSON Tokens are now just a profile of Magic Signatures, which John Panzer has helpfully extended for this purpose
- There was a vulnerability to masquerading attacks in the last proposal, which is addressed in this proposal by adding a data_type parameter that is part of the signature, but _not_ part of the payload.
- no more support of X.509 certs - the only supported format for discovered public keys is now the Magic Key format. We'll give people tools (which are quite easy to write) to convert their self-signed or CA-issued certs to magic keys.
- The specs are now formatted as I-Ds.

Comments, please!

Dirk.

---

From: Dirk Balfanz <[email-masked]>
Date: Thu, 23 Sep 2010 19:07:31 -0700

On Thu, Sep 23, 2010 at 6:51 PM, Eran Hammer-Lahav <[email-masked]>wrote:

>  In part II, how is the signature bound to the HTTP request URI? I see the
> method and body, but not the request URI.
>

The request URI goes in the audience parameter that's defined in part I.

Dirk.

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Thu, 23 Sep 2010 19:32:35 -0700

First, thanks for putting this together. It provides a pretty comprehensive story about the signed tokens proposal. I don't have much to add at this point to the drafts.

My position, which I have expressed many times before about this approach is:

Any solution involving repeating the HTTP request elements in some sort of envelope will require validation .In this case, you need to define all the rules for validating the 'audience' parameter.

This approach makes it too tempting for server developers to ignore the HTTP request and only use the values of the 'audience' and 'method' parameters. In order to validate those parameters, the server needs to perform some sort of normalization (which eventually leads to the same signature mismatch problems). Not looking at the HTTP request will lead to security problems because it will allow bypassing existing HTTP firewall rules which operate on the request URI and HTTP method.

As a matter of personal taste, this smells too much like SOAP. And that's a really bad thing.

I am not in favor of any solution that replicates components of the HTTP request within the request (as is the case here). Recent experience shows that with a little help and good libraries, developers can figure out how to normalize and sign an HTTP request successfully, when motivated.

EHL

On 9/23/10 7:07 PM, "Dirk Balfanz" <[email-masked]> wrote:

On Thu, Sep 23, 2010 at 6:51 PM, Eran Hammer-Lahav <[email-masked]> wrote:
In part II, how is the signature bound to the HTTP request URI? I see the method and body, but not the request URI.

Dirk.

EHL

On 9/23/10 3:39 PM, "Dirk Balfanz" <[email-masked] <http://[email-masked]> > wrote:

Hi guys,

Comments, please!

Dirk.

---

From: "Richard L. Barnes" <[email-masked]>
Date: Thu, 23 Sep 2010 22:59:53 -0400

Basically, this argument comes down to which validation pattern you  
prefer.

OPTION 1:
0. Input = object + signature
1. Verify match between object data and HTTP fields
2. Verify signature over the object

OPTION 2:
0. Input = signature
1. Construct object data from HTTP fields
2. Verify signature over the object

So the choice is between memcmp and memcpy :)

As Eran noted, there is some security advantage in option 2, since it  
forces the data in the signed object to correspond to the actual HTTP  
request instead of relying on the recipient to verify.  I would prefer  
that pattern, for that reason.

That's not to say that the JSON token proposal is without any  
utility.  For example, option 2 could take as input a JSON token with  
the 'data' element removed; the recipient would then reconstruct the  
'data' element from the HTTP request and verify the  token.

As an editorial note, it would be very helpful if the JSON token  
document could be made a little more stand-alone w.r.t. the Magic  
Signatures draft.  An example would also help, just to illustrate the  
overall structure of the token.

--Richard

On Sep 23, 2010, at 10:32 PM, Eran Hammer-Lahav wrote:

> First, thanks for putting this together. It provides a pretty  
> comprehensive story about the signed tokens proposal. I don’t have  
> much to add at this point to the drafts.
>
> Any solution involving repeating the HTTP request elements in some  
> sort of envelope will require validation .In this case, you need to  
> define all the rules for validating the ‘audience’ parameter.
>
> This approach makes it too tempting for server developers to ignore  
> the HTTP request and only use the values of the ‘audience’ and  
> ‘method’ parameters. In order to validate those parameters, the  
> server needs to perform some sort of normalization (which eventually  
> leads to the same signature mismatch problems). Not looking at the  
> HTTP request will lead to security problems because it will allow  
> bypassing existing HTTP firewall rules which operate on the request  
> URI and HTTP method.
>
> As a matter of personal taste, this smells too much like SOAP. And  
> that’s a really bad thing.
>
> I am not in favor of any solution that replicates components of the  
> HTTP request within the request (as is the case here). Recent  
> experience shows that with a little help and good libraries,  
> developers can figure out how to normalize and sign an HTTP request  
> successfully, when motivated.
>
> EHL
>
>
> On Thu, Sep 23, 2010 at 6:51 PM, Eran Hammer-Lahav <[email-masked] 
> > wrote:
> In part II, how is the signature bound to the HTTP request URI? I  
> see the method and body, but not the request URI.
>
> On 9/23/10 3:39 PM, "Dirk Balfanz" <[email-masked] <http://[email-masked] 
> > > wrote:
>
> Hi guys,
>
> - JSON Tokens are now just a profile of Magic Signatures, which John  
> Panzer has helpfully extended for this purpose
> - There was a vulnerability to masquerading attacks in the last  
> proposal, which is addressed in this proposal by adding a data_type  
> parameter that is part of the signature, but _not_ part of the  
> payload.
> - no more support of X.509 certs - the only supported format for  
> discovered public keys is now the Magic Key format. We'll give  
> people tools (which are quite easy to write) to convert their self- 
> signed or CA-issued certs to magic keys.
> _______________________________________________
> OAuth mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/oauth

---

From: Dirk Balfanz <[email-masked]>
Date: Fri, 24 Sep 2010 09:45:35 -0700

On Thu, Sep 23, 2010 at 7:59 PM, Richard L. Barnes <[email-masked]> wrote:

> Basically, this argument comes down to which validation pattern you prefer.
>
> OPTION 1:
> request instead of relying on the recipient to verify.  I would prefer that
> pattern, for that reason.
>

I think it's more robust to verify than to generate. In option 2, you have
to "guess" what the signer actually signed. To be fair, you have pretty good
signals, like the HTTP request line, the Host: header, etc., but in the end,
you don't _know_ that the signer really saw the same thing when they
generated the signature. I can't help but feeling that that's a bit of a
hack. In option 1, you always know what the signer saw when they generated
the signature, and it's up to you (the verifier) to decide whether that
matches your idea of what your endpoint looks like.

>
> That's not to say that the JSON token proposal is without any utility.  For
> example, option 2 could take as input a JSON token with the 'data' element
> removed; the recipient would then reconstruct the 'data' element from the
> HTTP request and verify the  token.
>
> As an editorial note, it would be very helpful if the JSON token document
> could be made a little more stand-alone w.r.t. the Magic Signatures draft.
>

Yeah, we were going back and forth on this. Should we just reference Magic
Signatures? Should we copy-and-paste? etc. The MS spec itself is more
complex, allows for multiple signatures (although not in the compact
serialization, which we use), has other serializations, etc. I thought that
it would be a bit of work for the reader to figure out exactly which pieces
of the MS spec they would have to understand in order to implement JSON
Tokens. I agree, though, that it is a bit, umh, unusual, to put excerpts of
another spec into your spec (especially if that ends up being 80% of your
spec :-).

>  An example would also help, just to illustrate the overall structure of
> the token.
>

Agreed.

Dirk.

> First, thanks for putting this together. It provides a pretty comprehensive
> story about the signed tokens proposal. I don’t have much to add at this
> point to the drafts.
>
> Any solution involving repeating the HTTP request elements in some sort of
> envelope will require validation .In this case, you need to define all the
> rules for validating the ‘audience’ parameter.
>
> This approach makes it too tempting for server developers to ignore the
> HTTP request and only use the values of the ‘audience’ and ‘method’
> parameters. In order to validate those parameters, the server needs to
> perform some sort of normalization (which eventually leads to the same
> signature mismatch problems). Not looking at the HTTP request will lead to
> security problems because it will allow bypassing existing HTTP firewall
> rules which operate on the request URI and HTTP method.
>
> As a matter of personal taste, this smells too much like SOAP. And that’s a
> really bad thing.
>
> I am not in favor of any solution that replicates components of the HTTP
> request within the request (as is the case here). Recent experience shows
> that with a little help and good libraries, developers can figure out how to
> normalize and sign an HTTP request successfully, when motivated.
>
> EHL
>
>

---

From: "Richard L. Barnes" <[email-masked]>
Date: Fri, 24 Sep 2010 13:08:55 -0400

> I think it's more robust to verify than to generate. In option 2,  
> you have to "guess" what the signer actually signed. To be fair, you  
> have pretty good signals, like the HTTP request line, the Host:  
> header, etc., but in the end, you don't _know_ that the signer  
> really saw the same thing when they generated the signature. I can't  
> help but feeling that that's a bit of a hack. In option 1, you  
> always know what the signer saw when they generated the signature,  
> and it's up to you (the verifier) to decide whether that matches  
> your idea of what your endpoint looks like.

Generating does not imply guessing: The signer can specify what he  
signed without providing the data directly.  Quoting from another  
thread:
"
1. Signer computes signature sig_val over data object:
   { user_agent: "Mozilla", method: "GET" }
2. Signer sends { signed_fields: ['user_agent', 'method'], sig:  
sig_val }
3. Recipient reconstructs data object using signed_fields
4. Recipient verifies sig_val == sign(reconstructed_object)
"

If the spec is written properly, the recipient should be able to look  
at the names of the fields ('user_agent', 'method') and use them to  
reconstruct the original object.

The idea of allowing signed fields to change en route to the server  
strikes me as a little odd.  Sure, you could ignore the method, path,  
and host values in HTTP and just act on the enveloped data, but at  
that point, why not just do away with the overhead of HTTP and run the  
whole thing over TCP?

--Richard

---

From: Dirk Balfanz <[email-masked]>
Date: Fri, 24 Sep 2010 11:38:17 -0700

On Fri, Sep 24, 2010 at 10:08 AM, Richard L. Barnes <[email-masked]> wrote:

> Generating does not imply guessing: The signer can specify what he signed
> without providing the data directly.  Quoting from another thread:
> "
> 2. Signer sends { signed_fields: ['user_agent', 'method'], sig: sig_val }
> If the spec is written properly, the recipient should be able to look at
> the names of the fields ('user_agent', 'method') and use them to reconstruct
> the original object.
>

User-agent and method are well-defined both for senders and receivers of
HTTP requests. What's less well-defined is the URL, which is what Eran is
objecting to. So in practice, it looks more like this:

1. Signer generates URL using some library, e.g.:
    paramsMap = new Map();
    paramsMap.put('param1', 'value1');
    paramsMap.put('param2', 'value2');

    uri = new UriBuilder()
      .setScheme(Scheme.HTTP)
      .setHost('WWW.foo.com')
      .setPath('/somePath')
      .setQueryParams(paramsMap)
      .build().toString();
   // uri now looks something like "
http://WWW.foo.com/somePath?param1=value1&param2=value2"

2. They then use a different library to send the HTTP request
    request = new GetRequest();
    request.setHeader('signed-token', sign('GET', uri));
    request.execute(uri);

The problem is that we don't know what the execute method on GetRequest does
with the URI. It probably will use a library (possibly different from the
one used in step 1) to decompose the URI back into its parts, so it can
figure out whether to use SSL, which host and port to connect to, etc. Is it
going to normalize the hostname to lowercase in the process? Is it going to
escape the query parameters? Is it going to add ":80" to the Host:-header
because that's the port it's going to connect to? Is it going to put the
query parameters into a different order?, etc., all of which would cause the
recipient of the message to put back together a _different_ URI from what
the sender saw.

OAuth1 therefore defined a bunch of rules on how to "normalize" the URI to
make sure that both the sender and the receiver saw the same URI even if the
http library does something funny. Many people thought that those rules were
too complicated. There is currently an argument over whether or not the
complexity of the rules can be hidden in libraries, and I'm personally a bit
on the fence on this. What I _do_ object to, more on a philosophical level,
is that we can never know for sure what the http library is doing to the
request, and that therefore we can never be sure whether the normalization
rules we have come up with cover all the crazy libraries out there. There is
a symmetric problem on the receiver side - where the servlet APIs may or may
not have messed with the parameters before you get to reconstruct the URI.

The JSON token proposal does something simpler: you get to see the URI as
the sender saw it (in this case with the uppercase WWW, without the :80,
etc.), and you get to decide whether that matches your endpoint. So instead
of wondering in what order the signer saw the query parameters when he
signed them, and whether they were escaped or not, you simply check that all
the query parameters that he signed (as evidenced in the JSON token) are
indeed present in the HTTP request, and vice versa, etc. It's a comparable
amount of work, but it seems cleaner, less hacky to me.

Dirk.

> The idea of allowing signed fields to change en route to the server strikes
> me as a little odd.  Sure, you could ignore the method, path, and host
> values in HTTP and just act on the enveloped data, but at that point, why
> not just do away with the overhead of HTTP and run the whole thing over TCP?
>
> --Richard
>

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 24 Sep 2010 13:31:51 -0700

Most developers will do one of two things:

1. Completely ignore the HTTP request and just look at the signed data.
2. Come up with some simple rules to normalize both the request URI and signed data and compare them to make sure they are the same.

Why is this bad?

- This proposal recreates SOAP, only using JSON instead of XML. It treats HTTP as a stupid transport where the HTTP fields are more of a liability than an asset. This design would work much better if you had a single endpoint API with a structured payload that is signed using a magic signature. All this HTTP request construction nonsense goes away and you don't need to repeat parameters, methods, etc. This is exactly why Roy Fielding, who knows a little bit about HTTP, rejected this approach of duplicating request data in the transmission.

- At the time the application gets to look at the signed data and the HTTP request header, it is already too late for outer security layers to do their jobs. A firewall configured to block requests to certain URI paths or HTTP methods will fail to stop requests were the internal signed data is different (and potentially harmful) than the external HTTP request header. This moved the burden of enforcing such a security policy from the firewall to the application layer aware of the OAuth signature format.

- Any kind of URI normalization and comparison is hard, and will result in some failures. OAuth 1.0a has shown this difficulty, but recent experience has completely reversed the narrative this proposal is based on, that developers cannot figure it out. As I said many times before, developers just need the right motivation to spend an extra hour to figure out why their signatures fail (yes, an hour!) and providers should stop treating their developers badly and provide quality debugging information.

- I am pretty sure that the vast majority of platforms these days allow the application full access to the raw request header as sent over the wire. Before we go and design another solution like the OAuth 1.0a signature process which was based on limitations in platforms such as PHP4, we should do our homework and determine how hard it really is to access the raw request header. This proposal is an optimization to a problem that does not really exist.

- OAuth 2.0 now fully supports bearer tokens which removes the requirement to use signatures. I think it is perfectly fine to raise the deployment bar for the more advance cases where signatures are required. Let's be honest here - most of the applications currently using OAuth 1.0a will be happy with 2.0 without signatures.

- By working closer to the HTTP layer, we not only improve HTTP, but also give one more reason for HTTP libraries to add native OAuth signature support.

This is not a complete list.

EHL

On 9/24/10 11:38 AM, "Dirk Balfanz" <[email-masked]> wrote:

On Fri, Sep 24, 2010 at 10:08 AM, Richard L. Barnes <[email-masked]> wrote:
I think it's more robust to verify than to generate. In option 2, you have to "guess" what the signer actually signed. To be fair, you have pretty good signals, like the HTTP request line, the Host: header, etc., but in the end, you don't _know_ that the signer really saw the same thing when they generated the signature. I can't help but feeling that that's a bit of a hack. In option 1, you always know what the signer saw when they generated the signature, and it's up to you (the verifier) to decide whether that matches your idea of what your endpoint looks like.

Generating does not imply guessing: The signer can specify what he signed without providing the data directly.  Quoting from another thread:
"

If the spec is written properly, the recipient should be able to look at the names of the fields ('user_agent', 'method') and use them to reconstruct the original object.

User-agent and method are well-defined both for senders and receivers of HTTP requests. What's less well-defined is the URL, which is what Eran is objecting to. So in practice, it looks more like this:

    uri = new UriBuilder()
      .setScheme(Scheme.HTTP)
      .setHost('WWW.foo.com <http://WWW.foo.com> ')
      .setPath('/somePath')
      .setQueryParams(paramsMap)
      .build().toString();
   // uri now looks something like "http://WWW.foo.com/somePath?param1=value1&param2=value2"

The problem is that we don't know what the execute method on GetRequest does with the URI. It probably will use a library (possibly different from the one used in step 1) to decompose the URI back into its parts, so it can figure out whether to use SSL, which host and port to connect to, etc. Is it going to normalize the hostname to lowercase in the process? Is it going to escape the query parameters? Is it going to add ":80" to the Host:-header because that's the port it's going to connect to? Is it going to put the query parameters into a different order?, etc., all of which would cause the recipient of the message to put back together a _different_ URI from what the sender saw.

OAuth1 therefore defined a bunch of rules on how to "normalize" the URI to make sure that both the sender and the receiver saw the same URI even if the http library does something funny. Many people thought that those rules were too complicated. There is currently an argument over whether or not the complexity of the rules can be hidden in libraries, and I'm personally a bit on the fence on this. What I _do_ object to, more on a philosophical level, is that we can never know for sure what the http library is doing to the request, and that therefore we can never be sure whether the normalization rules we have come up with cover all the crazy libraries out there. There is a symmetric problem on the receiver side - where the servlet APIs may or may not have messed with the parameters before you get to reconstruct the URI.

The JSON token proposal does something simpler: you get to see the URI as the sender saw it (in this case with the uppercase WWW, without the :80, etc.), and you get to decide whether that matches your endpoint. So instead of wondering in what order the signer saw the query parameters when he signed them, and whether they were escaped or not, you simply check that all the query parameters that he signed (as evidenced in the JSON token) are indeed present in the HTTP request, and vice versa, etc. It's a comparable amount of work, but it seems cleaner, less hacky to me.

Dirk.

The idea of allowing signed fields to change en route to the server strikes me as a little odd.  Sure, you could ignore the method, path, and host values in HTTP and just act on the enveloped data, but at that point, why not just do away with the overhead of HTTP and run the whole thing over TCP?

--Richard