Subject: [rohc] Some comments on SigComp interoperability testing...

From: Mark West <[email-masked]>
Date: Thu, 25 Aug 2005 14:11:46 +0100 (BST)

Hi all,

I think that there are a few things worth discussing around SigComp
interoperability testing.  Specifically, what is there to test and how to
do it...

I believe that I am on record (during the run-up to the previous tests) as
stating that Interop testing is more about verifying the assumptions made
by compressors.  And I still believe that to be largely true.

Abbie will send a mail shortly, summarising what we understand to be the
functionality of SigComp and those functions which we believe were
successfully tested at SIPit.

One of the most obvious questions surrounds the use of bytecode: do we
have to test every instruction (and what does this mean)?  The assumption
for SIPit last time was that everyone who did 'serious' interoperability
testing had already run and passed the torture tests.

Three examples:

* If an algorithm makes use of, say, LSHIFT and demonstrates
interoperability, but no algorithm (tested) happens to make use of RSHIFT,
does that mean that RSHIFT cannot be considered as a 'tested' instruction?
(If not LSHIFT / RSHIFT, how about DIVIDE / REMAINDER..?)

* SORT-* instructions have been tested in the torture-tests but have not
been used in any interoperability test.  Actually the SORT-* instructions
are not heavily exercised by the torture tests.  However, what is the
difference between sending some bytecode generated in a torture test and
some bytecode from a 'real' compressor..?

* The MULTILOAD instruction *has* been used in bytecode which has
interoperated.   However, to the best of my knowledge, none of this code
has tested the properties of MULTILOAD when the instruction overwrites
itself.  However, the torture tests explicitly verify this case.

I believe that it is unreasonable to insist that every instruction is used
in bytecode that has been shown to interoperate (we are, after all,
limited by the algorithms that are implemented).  However, it clearly is
reasonable to require a certain level of confidence in the execution of
the instructions (possibly beyond what is in the torture tests).  A good
example is the stack, which was not tested in the last interoperability
test: should we test *every* stack instruction; or simply show that the
stack *concept* is interoperable..?

Note that, if we are testing with 'real' SigComp implementations, I'd
expect a compressor to only send the (limited) bytecode that is necessary
to decompress messages: it is not practical to have arbitrary bits of
bytecode exchanged between arbitrary peers.  You could generate specific
test bytecode and send it to an implementation -- but that's, essentially,
what the torture tests do!

Also, many of the test cases are concerned with failures ('if this is
attempted, then this failure should result...'), but a well-behaved
compressor won't generate many (any?!) of these failures...  Again, it is
a torture-test like approach (injecting specific test messages) which has
to be used.

I think that all we can usefully do in an interoperability test is:
- ensure that correct usage of the state handling functionality is
properly exercised;
- test a respectable subset of the available UDVM instructions;
- test message formats and the like;
- identify any 'orphan' functionality -- such as the mysterious 'S-bit'
which, basically, no-one could see much use for...

Comments?

Cheers,

Mark.

--
Mark A. West, Senior Consultant Engineer
Roke Manor Research Ltd., Romsey, Hants.  SO51 0ZN
Phone +44 (0)1794 833311   Fax  +44 (0)1794 833433

_______________________________________________
Rohc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rohc