Subject: Online Protocol Testing Experience

From: "Tschofenig, Hannes (NSN - FI/Espoo)" <[email-masked]>
Date: Fri, 17 May 2013 12:31:29 +0000

Hi all,

when concerns about the lack of interoperability surfaced mid last year 
in the OAuth working group we (Derek, and myself) tried to figure out 
whether we should schedule a face-to-face interop and/or to develop an 
online test suite. We got in touch with Lucy Lynch (ISOC) and she helped 
us to find developers to work with us on the test software.

Roland Hedberg, one of the guys working on the project for OAuth 
testing, presented his ongoing work in the OAuth working group, see
http://www.ietf.org/proceedings/86/slides/slides-86-oauth-2.pdf

OAuth is a bit more complex since it involves more than two parties and 
we were looking for a test framework that could be re-used to develop 
the desired results more quickly. To our surprise we couldn't find 
anything that we could easily re-use and most of the test frameworks 
were really focused on different types of tests. Of course, we might 
have looked at the wrong projects.

Here is how it works at the moment (and I will post a more complete 
step-by-step introduction in the next few days):
* Imagine you have developed an OAuth-based identity management server 
(that contains an OAuth 2.0 authorization server) and it runs somewhere 
on the Internet (or in your lab). You don't need to have access to the 
source code to execute the tests.
* You download the scripts that Roland & Co had developed and configure 
them. Of course you will have to create an account at your IdP as well.
* You run the test scripts against the authorization server and the 
script plays the other OAuth 2.0 parties in the exchange. The script contains a number 
of test cases (around 60+ at the moment) and determines whether the IdP 
responds correctly in the exchanges.

I know that these ideas have come up in other working groups in the past 
already (such as in SCIM, which also has a test server up and 
running).

It would be interesting to hear what others have been doing and what 
worked for you or what didn't.

Ciao
Hannes