Subject: HTTP compliance

From: Duane Wessels <[email-masked]>
Date: Sun, 4 Jun 2000 18:45:06 -0600

During the workshop in Lisbon, many people mentioned the need for HTTP
compliance tests.  An analogy was made to the "connect-a-thon" that Sun
has organized for NFS implementations.  One nice feature of the
connect-a-thon is that participating companies can not use the results
to claim compliance in their marketing literature.  It seems to be an
engineering-only event to ensure that products interoperate with each
other.

My group would very much like to take on this role for HTTP
compliance/interoperation testing.  We are interested in developing
software for vendors and others to use.  We are willing to host
"HTTP-a-thon" events if vendors would find it useful.

Unfortunately, we probably don't have the resources at this time to
pull it off.    We need money to hire a programmer/coordinator.  What
follows is our plan for making it happen:

1) We create a "consortium" of sorts to fund the development.
   The members are vendors and others interested in HTTP compliance
   software and tests.  Given that the discussion originated in
   the context of caching, we expect many members to be companies
   that have caching products.

2) Companies pay an annual membership fee of $20k (+/- $5k).
   Our goal is to collect about $200/year for this project.
   This money pays for salary, equipment, travel, etc. 
   
3) Development is driven by the needs and desires of the members.
   That is, members decide which features and tests are most
   important.  I envision something democratic -- one vote per
   member.

4) My group implements the desired features, and provides support for
   the software.

5) Members have immediate access to software releases.  Software is
   released to non-members after a period of six months (or so).
   Members are bound to specific terms of use.  Results may not be used
   in marketing literature.

This is our proposal; nothing has been set in stone.  I'm eager to find
out if vendors and others think its a workable model.  Feel free to
follow up on this list, or reply to me personally.

Duane W.

---

From: Mark C Nottingham <[email-masked]>
Date: Sun, 4 Jun 2000 20:02:59 -0700

Duane,

I (and everyone I talked to) think that you guys are a natural for this sort
of project, mostly because you're already in the testing business. As long
as the focus stays on the engineering level, and away from making it
competitive, this will be great.

Others will have more intelligent things to say about the organisation of
funding, membership, etc., but it would be nice if vendors who aren't
directly participating weren't cut out (e.g., six-month code delay), as
they're generally the people who need the most help.

How is the connect-a-thon organised in this respect?

A few points for discussion; 

* What are the target devices for such tests?
Describing server and client as well as proxy/surrogate tests is useful, but
much more work, and it may turn out quite a bit harder (and more disruptive)
to pull the vendors on board.

I'd recommend starting with proxy/surrogate devices, and eventually
expanding into clients and servers if there's enough interest.

* What kinds of tests?
The most obvious goal is to prove full 1.1 compliance, according to stated
RFC2119 requirements. I think most people realise that doing so is actually
a very difficult task, because of all of the possible permutations of HTTP
mechanisms, etc that present themselves.

As I said in Lisbon, my interest is primarily in assuring that caches
operate in a predictable way to content providers; while we could test to
make sure that each message is well-formed, etc., IMHO that won't be
necessary, as it's already in the best interests of the vendor to do so.

It may be more realistic to define a relatively small group of relevant
tests to start with, making it easier to get a first go at it out quickly,
to gauge response and hopefully have some positive effects ASAP.

Finally, I'd be very surprised if any of the commercial vendors who showed
interest didn't have a pre-existing compliance engine that they use
interally. If one could be convinced/arm-twisted that it would be in their
best interests to donate it (or a portion), we'd save a lot of effort.

Cheers,

> follows is our plan for making it happen:
[snip]

-- 
Mark Nottingham, Senior Developer
Akamai Technologies (San Mateo, CA)

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 5 Jun 2000 07:42:03 -0600 (MDT)

> they're generally the people who need the most help.

We would be more than happy to cut nobody out. However, there are two
problems that are hard to solve if you allow free timely access to the
tool:

	- a not-for-marketing license would be nearly impossible
	  to enforce; a company or a magazine are likely to violate
	  the licensing terms (knowingly or not) and publish claims
	  of [non]compliance.

	- there is not much incentive for vendors to pay for the
	  development if they can get the tool anyway (just look
	  at the number of Polygraph sponsors versus the number of
	  caching vendors who use the tool)

> How is the connect-a-thon organised in this respect?

It will probably up to the participants to decide. IMO, connect-a-thon
will not be very useful for these kinds of tests because the tool will
be testing mostly compliance with standards and current practice rather
than compatibility with similar software. I suspect that most of the
tests can be done in isolation so connect-a-thon will not be required.
 
> expanding into clients and servers if there's enough interest.

I would make it demand driven. If there is enough interest/support from
the server/client community, the tool can include server/client tests.
 
> mechanisms, etc that present themselves.

Full HTTP/1.1 compliance is a nice goal, but probably not very
practical. Again, since the tool is designed for product developers, I
would leave the scope for them to decide on. Initial goals should
probably be more modest to get some visible progress sooner.
 
> to gauge response and hopefully have some positive effects ASAP.

Agree.
 
> best interests to donate it (or a portion), we'd save a lot of effort.

I am less optimistic after talking to several vendors :). However, we
definitely should try to use the results of past developments if any.

Thanks,

Alex.

---

From: "Oskar Batuner" <[email-masked]>
Date: Mon, 5 Jun 2000 11:42:25 -0400

Folks, for me it looks like we are talking about two different
possible developments:

1. HTTP compliance tests.
2. Connect-a-thone.

A while ago I was in NFS development and connect-a-thon was a very
important part of testing. But the way it worked had nearly nothing to
do with the protocol compliance testing. Different vendors came together
with equipment and software, point clients to other people’s
servers and run a group of standardized tests. If our new client has
problem with the current version of let's say HP-UX - it does not matter
who is in compliance, it just means that WE do have a problem.
If such problem is found when both versions are not yet released – you
talk to other party, and also look how this specific test works with
other people implementations.

The primary goal of such tests is inter-vendor compatibility and
"current industry practices" definitely dominate over puristic protocol
compliance issues.

What kind of possible even/alliance are we trying to create?
Both have the right to exist.

The first one, compliance tests, is kind of more straightforward,
one can just write a set of tests with limited interaction with
cache/service vendors. Still to be really useful such tests
should take into account current practices, the way HTTP is
practically implemented, not just seat in the clean
room with nothing but the set of RFC's.

I agree with Alex that there is no need for all of us to come
together for such testing. Tests can be made available as
a software package - or alternately run remotely by configuring
the test lab and system under test to point to one another. The
latter form creates more controllable environment and may prevent
some problems Alex is talking about.

The form of organization Duane is proposing is best suited exactly
for this type of testing.

Connect-a-thone may be also a good thing to have. It is especially
useful for cache/layerX switch/router vendors - current environment
becomes extremely complicated. Too many different devices available,
and understanding the source of the problem when interacting with
specific device often requires a help of vendor engineers - not just
sales support. Service vendors may be interested in both testing
possible elements of infrastructure and preventing problems for
service customers.

The connect-a-thone event may be opened to non-members for some
registration fee.

My two cents.

- Oskar

>-----Original Message-----
>From: Alex Rousskov [mailto:[email-masked]]
>Sent: Monday, June 05, 2000 9:42 AM
>To: Mark C Nottingham
>Cc: [email-masked]
>Subject: Re: HTTP compliance
>
>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 5 Jun 2000 08:48:35 -0700

> definitely should try to use the results of past developments if any.

*grin*
I meant in no way to imply optimism by that statement. 

-- 
Mark Nottingham
http://www.mnot.net/

---

From: Mark C Nottingham <[email-masked]>
Date: Mon, 5 Jun 2000 10:37:33 -0700

Connect-a-thon has been mentioned not as a similar type of test or model for
testing, but as a model for how the participants approach the tests.

I also agree that people shouldn't have to be in the same room. Regarding
current practice vs. clean room, this is good, *as long as* the tests vary
from the specification. i.e., it would be good to test for optimal
operation, etc., but to declare that common practice supercedes an explicit
requirement in the specification is IMHO folly.

Cheers,