Subject: [tap] What is a test result? Ideas from other testing systems.

From: Michael G Schwern <[email-masked]>
Date: Tue, 23 Sep 2008 18:31:46 -0700

As part of Test::Builder2 I've been looking at a lot of other testing
frameworks other than TAP in order to make TB2 flexible enough to provide the
core for any testing style.  This is important to TAP because TAP is really
more a test serialization format than a testing style.

One of the differences I come across is that TAP has a test result (pass or
fail) and directives (skip, todo) which can be combined.  You can have a
passing skip test or a failing todo test.  This leads to awkward questions
like what does a failing skip test mean?  It also leads to awkward designs
like a test result having a "should it be considered a pass" attribute and
separate a "did it say it passed" result.  For example, "not ok #TODO" says it
failed, but it's considered a pass.

Most others treats them as all results.  Let's look at dejagnu [1] (which is
an extension of POSIX testing) vs TAP.

Dejagnu		TAP
-------------------------------
PASS		ok
FAIL		not ok
XFAIL 		not ok # TODO
XPASS 		ok # TODO
UNSUPPORTED	ok # SKIP
UNRESOLVED	???
UNTESTED	???

By having atomic test results they avoid the "not ok # SKIP" edge case.  Also
notice they have two states we don't.  UNRESOLVED means the results cannot be
evaluated by the program and requires human intervention.

The dejagnu docs explain...

    Here are some of the ways a test may wind up UNRESOLVED:

    * A test's execution is interrupted.
    * A test does not produce a clear result.
    * A test depends on a previous test, which fails.
    * The test was set up incorrectly.

In TAP-land we'd probably just fail the test.  The POSIX testing framework is
much more spec-coverage oriented.

UNTESTED is the opposite of a TODO test.  A TODO test means there's a test,
but no feature.  UNTESTED means there's a feature but no test.  Again, being
converted with spec-coverage, POSIX wants to explicitly know that a piece of
the spec is uncovered.  This is an interesting result which TAP has no easy
way of simulating.  Also note that it's neither a pass nor a fail.  Its
another state.

NUnit [2], a cleverly designed .Net framework, has interesting test modifiers
like this:

Category	A set of tests which can be included and excluded together.
		For example "LongRunning"

Culture		Only run in the given culture (locale)

Explicit	Only run this test if explicitly asked for.

Ignore		Don't run the test.  This is an alternative to
		commenting the test out, like a TODO + SKIP.

Platform	Only run on the given platform

Property	Assign key/value pairs to a test.  Much like YAML
		diagnostics.

TAP can simulate most of this with SKIPs, but its interesting to see what they
built in.  I suppose at this level it's more at the test library level then
the TAP level.

One framework which I can't recall at the moment, distinguishes between "there
is an unfixed bug" and "there is an unimplemented feature".  TAP lumps these
together as TODO but they are really different concerns.

Another (perhaps the same one) distinguishes between the status of the test
(pass/fail), should it cause the run to be considered a failure and should it
report the result to the user.  For example, in TAP a "not ok #TODO" is a
failure, but it does not cause the test run to fail and the result should not
be reported.  What TAP is missing is the "informational test" or probe, the
one which does not cause the test to fail (and thus halt installation in Perl
land) but IS reported.

Test::Builder2 is being designed with all this in mind.  A test result is
encapsulated as a Test::Builder2::Result object of the correct subclass.
Currently this is Test::Builder2::Result::Pass, Fail, Skip and Todo, [3] but
the result logic is pluggable so new result types can be added.  Currently the
results are designed like TAP with directives, a pass/fail field and an
interpretation.  I'm moving away from that and using an extension of the POSIX
result.  Each result has the three basic properties, 1) did it fail? 2) does
it fail the suite? 3) should the user be informed.  Instead of a "Todo" result
there's a "Bug" (1. Yes  2.  No.  3.  No), "BugFixed" (1. No.  2. No.  3.
Yes), "MissingFeature" (1. Yes  2. No  3. No), "MissingFeatureImplemented" (1.
No  2.  No.  3. Yes) and so on.  Note that Bug/BugFixed and
MissingFeature/MissingFeatureImplemented have the same basic behaviors, but
subtly different meanings that someone might want to exploit.

Fascinating stuff.

[1] Dejagnu/POSIX docs
    http://www.gnu.org/software/dejagnu/manual/x47.html

[2] NUnit  http://www.nunit.org/index.php?p=docHome&r=2.4.8

[3]
http://code.google.com/p/test-more/source/browse/trunk/lib/Test/Builder2/Result.pm

-- 
Defender of Lexical Encapsulation

_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Wed, 24 Sep 2008 04:34:56 +0200

* Michael G Schwern <[email-masked]> [2008-09-24 03:35]:
> Test::Builder2 is being designed with all this in mind. A test
> result is encapsulated as a Test::Builder2::Result object of
> the correct subclass. Currently this is
> Test::Builder2::Result::Pass, Fail, Skip and Todo

Might it be more useful to cast these as roles of a ::Result
rather than subclasses?

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael G Schwern <[email-masked]>
Date: Tue, 23 Sep 2008 22:45:54 -0700

> Might it be more useful to cast these as roles of a ::Result

That's how chromatic did it in Perl 6, and I might do it that way.  Except
that implies you can have the "Pass" and the "Fail" role?  Weird.  But maybe
you model an "ok # SKIP" as having the "Pass" and the "Skip" role.  That
doesn't jive too well with the idea that a skip is just a skip, it's not a
"pass with skip".

Anyhow, we'll see.  I'm sort of slapping things together quickly at this stage.

(Honestly it's because I still think in subclasses.)

-- 
Don't try the paranormal until you know what's normal.
    -- "Lords and Ladies" by Terry Prachett
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: "David E. Wheeler" <[email-masked]>
Date: Tue, 23 Sep 2008 23:33:28 -0700

> Fascinating stuff.

Indeed, thanks for sharing.

Best,

David

---

From: Ovid <[email-masked]>
Date: Wed, 24 Sep 2008 00:51:06 -0700 (PDT)

--- On Wed, 24/9/08, Michael G Schwern <[email-masked]> wrote:

> This leads to awkward questions like what does a failing skip test mean?

To my mind, if a test is skipped, it's not run.  Therefore a failing skip test should be a parse error.

That being said, "unsupported" could be like a skip test in that we have this feature, but we don't know that it works on platform X, therefore if it works for you, great, but we don't support it.  That means what we would normally skip could be run, but the results are not guaranteed.  In that respect, a failing skip test is just that:  a failing skip test.  However, I think a distinction between unsupported and skip is important.

> an extension of POSIX testing) vs TAP.

In reading through the docs, this almost sounds like a conceptual mistake.  If you call "perror", the status of the next test is UNRESOLVED.  The idea is that if you have a serious error, the subsequent test results cannot be trusted.  To my mind, this seems to overcomplicate things for no clear benefit that I can see.  If tests are failing, you can't trust the results and the UNRESOLVED message seems superfluous.  However, there could easily be a lot more going on that I don't understand.

You also might want to read my questions about this topic and some of the answers:

  http://stackoverflow.com/questions/91585/what-do-you-need-from-a-test-harness
  http://is.gd/336f

Other thoughts:  I notice that DejaGnu automatically supports a log and I think that this is good (tempted to write Test::Log).  Seemless integration with something like Log::Log4Perl could be beneficial.  I'm trying to do something conceptually similar with App::Prove::History, but I'm using an SQLite database to back it up because test results over time are important.  That's raised a couple of interesting issues.

First, how do I identify a test program, much less a test?  Currently I'm going by filename, but if someone renames a file, I lose.  I've thought about optional Subversion and Git integration (and others, but CVS fails) which can track via source control the file renames, but that's waaaaay down in the priority queue.

There's also the question of "what's a test?"  If you're on this list, you probably know that XUnit has test methods, each of which is considered a test, and has various asserts in each method.  The asserts are *not* tests.  We can resolve part of the issue there with nested TAP, but that brings me back to a more immediate problem.

People want to know the start and end times (or duration) of each test.  Here's why this fails with Test::More:

  ok $some_val;  # duration is useless
  my $next_val = super_expensive_function();
  is $next_val, $expected; # duration is still useless

I could potentially hook the start time to be the end time of the previous test and that will mitigate it, but if you load expensive fixtures on start up, you would need something like this to start accurate timing:

  load_expensive_fixtures();
  start_timing();
  ok $some_val;  # duration is useless
  my $next_val = super_expensive_function();
  is $next_val, $expected; # duration is still useless

But you know, that's just clunky.  And super_expensive_function() *is* part of the test but Test::More and friends really don't treat it that way.  Aside from test duration, are there other implications of this we don't know about?  If that function throws an error or warning, should that be logged with the test?

This could be much ado about nothing, but in case it's not, I figured I'd lay it out there for people to see.

Cheers,
Ovid

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Wed, 24 Sep 2008 13:24:01 +0200

> Indeed, thanks for sharing.

Yes, I forgot to mention. Thanks for the effort (research and
writeup), Michael.

Regards,

---

From: Michael Peters <[email-masked]>
Date: Wed, 24 Sep 2008 10:06:54 -0400

Ovid wrote:

> I'm trying to do something conceptually similar with App::Prove::History, but I'm using an 
> SQLite database to back it up because test results over time are important.  That's raised a 
> couple of interesting issues.

One question I've had while watching the progress on App::Prove::History is how important is 
historical test data from one machine? Is this to be used with each developer having a separate 
history or is more for a smoker machine?

> First, how do I identify a test program, much less a test?  Currently I'm going by filename, 
> but if someone renames a file, I lose.  I've thought about optional Subversion and Git integration 
> (and others, but CVS fails) which can track via source control the file renames, but that's 
> waaaaay down in the priority queue.

This is a feature I've been wanting to do for Smolder for some time. It would be nice to answer such 
questions as "When was the last time this particular test passed on this platform?". Tinderbox does 
this now and it solves it by giving each test assertion a uid. This has to be assigned on the test 
level, not the harness level though. Since as you point out the file name can change. So can the 
assertion's position or the assertion's description. While a pain to do for your tests, it's the 
only way I can think of to really track this in a general way. A uid would be easy to embed as TAP 
YAML diagnostic.

>   ok $some_val;  # duration is useless

Are you saying it's useless here because it will basically be zero? If you put the calculations that 
resulted in $some_val after the start_timing() call would it still be useless?

>   is $next_val, $expected; # duration is still useless

Why is duration useless here?

But overall I think you might be mixing 2 different concepts. Benchmarking and testing. If you want 
to see how your application either speeds up or slows down over time, then create some benchmark 
scripts that store the results into an SQLite db and run them periodically. You could even run them 
with every svn checkout so you know exactly who slowed stuff down. Similar to smoke testing, but 
orthogonal IMO.

-- 
Michael Peters
Plus Three, LP

---

From: Ovid <[email-masked]>
Date: Wed, 24 Sep 2008 07:20:12 -0700 (PDT)

> history or is more for a smoker machine?

I suspect that a lot of uses will simply come out of the new technology.  One example which springs to mind:

We sometimes get permission to refactor by arguing that as our test suite grows larger and slower, developers are running the suite less often and more failing code gets checked in.  This is anecdotal exerience, though.  We can't back it up with numbers, but we could with historical results.

Also, I would like to eventually run tests against my entire installation (this is blocked while waiting for some BackPan indexing work from brian d foy).  If you upgrade a module Some-Module/t/foo.t fails, you might want to know if it was failing before the upgrade.  Then you could at least hope you haven't introduced *new* bugs.

I'm sure other uses will evolve over time.  (Particularly if people help me with my crappy code :)

> A uid would be easy to embed as TAP YAML diagnostic.

Except they have to be manually generated and embedded in the code, don't they?

> Are you saying it's useless here because it will
> basically be zero? If you put the calculations that 

I think it might be useful then, but again, it's far down on my list right now.
 
> Why is duration useless here?

It's only useless there if you're timing the is() assertion.  If you use start_timing() or start timing after the last test, then it becomes useful.

> But overall I think you might be mixing 2 different
> concepts. Benchmarking and testing.

But it might be nice to select tests where their timing has differed dramatically from, say, a test run a few weeks ago.  Without being able to uniquely identify tests, though, that's pretty tricky.

Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael Peters <[email-masked]>
Date: Wed, 24 Sep 2008 10:24:16 -0400

> Except they have to be manually generated and embedded in the code, don't they?

Yes. You can't generate them from the harness because it doesn't have enough knowledge. Even hooking 
into source control wouldn't be enough. It could follow name changes in files, but it wouldn't know 
how to track a logical test assertion over time.

I never said it was a fun way to do it, just that it was the only reliable way that I've been able 
to see after thinking about it for way too long.

-- 
Michael Peters
Plus Three, LP

---

From: Eric Wilhelm <[email-masked]>
Date: Wed, 24 Sep 2008 13:00:35 -0700

# from Michael Peters
# on Wednesday 24 September 2008 07:24:

>Yes. You can't generate them from the harness because it doesn't have
> enough knowledge. Even hooking into source control wouldn't be
> enough. It could follow name changes in files, but it wouldn't know

Rather than clutter the code with uid's, simply tracking the filename 
and assertion description is a good initial identifier.

And, in the same way that you use version-control as an external 
notation for the changing filename, you could use some sort of external 
changeset for the assertion descriptions.

That is, your filename tracking basically uses this historical 
association:

  filename_today => filename_yesterday => filename_last_friday

Then you e.g. create an identifier for that file and connect all of 
those filenames (per timestamp) to that identifier.

Same thing for the assertions.  Potentially, you could even use the diff 
to track the assertion renaming, but the worst case is that you allow 
some external tool to create the historical associations - which means 
the user can always add the history if/when they decide to care.

Note that given a filename+timestamp, the assertion does not need a 
UUID, but simply an id which is unique within that scope.

So, instead of paying for the tracking of everything up-front by 
scattering data throughout the code, the user could use a unique 
assertion name only for the ones they care about up-front.

That is: the relations are setup in such a way that they can decide to 
care about a specific something later, and perhaps designed to minimize 
the amount of annotation that has to be added at that point, but should 
not depend on knowing everything from the very beginning.

This would also make a nice way to add backdated tracking to an 
unmodified historical test suite because your tool didn't exist last 
year ;-)

--Eric
-- 
"Time flies like an arrow, but fruit flies like a banana."
--Groucho Marx
---------------------------------------------------
    http://scratchcomputing.com
---------------------------------------------------
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Ovid <[email-masked]>
Date: Thu, 25 Sep 2008 00:23:12 -0700 (PDT)

> and assertion description is a good initial identifier.

<snip>

I've been thinking about this and at first I thought it wouldn't work because people often have no test names or lots of duplicate test names.

Then I realized that those would be a hell of a lot easier to fix than UUIDs or a similar strategy, so maybe this could be a first pass solution.  (Still going to play havoc with the db schema, though).

Cheers,
Ovid

---

From: Eric Wilhelm <[email-masked]>
Date: Thu, 25 Sep 2008 01:40:41 -0700

# from Ovid
# on Thursday 25 September 2008:

>I've been thinking about this and at first I thought it wouldn't work
> because people often have no test names or lots of duplicate test
> names.

I was pondering that same issue, but the logic all leads to either you 
had clean test names or you didn't and you probably can't be bothered 
to go through every revision of the test suite and apply a unique name 
(and most version control tools don't really allow that), so using your 
new tool to make a trackback through an existing project history would 
be a non-starter if it depended on that.

Whereas if you 'wing it', you provide utility where the tests already 
had distinct names (those are probably the important ones anyway.)  And 
I think you can leave the door open to creating formal associations 
later.

So, say per-revision, you have the assertion name, line number, file id, 
and a pointer to the result from the previous revision (plus ya know, 
that pass/fail raison d'être stuff.)  The assertion name is not a 
unique identifier, all that matters is the pointer to the previous 
result.  If the name is unique within the file scope, you can set the 
pointer.  But for anything else leave it unset, or potentially use the 
line number if the file hasn't changed.

The rest is basically 'hard'.  But an external tool (I'm thinking a 
diff-based analysis, but you might have an intern handy) could go back 
through using the file id+line number to connect the dots.

Depending on use cases (i.e. report generation vs interactive), a user 
could get away with zero names assuming that their dot connector had 
done everything right.  (If I move an assertion in a way that fools the 
dot connector, I just give it a name before moving it or create the 
pointer manually, etc.)

> (Still going to play havoc with the db schema, though). 

That which doesn't kill our schema... ;-)

--Eric
-- 
Introducing change is like pulling off a bandage: the pain is a memory
almost as soon as you feel it.
--Paul Graham
---------------------------------------------------
    http://scratchcomputing.com
---------------------------------------------------
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael G Schwern <[email-masked]>
Date: Thu, 25 Sep 2008 18:35:30 -0400

> skip test should be a parse error.

Which is why its an awkward question, why does the potential for a
non-sensical state exist at all?  Of course, we know the answer "for
historical reasons".  It didn't break backwards compatibility.  If I
were putting TAP together today I'd probably make "skip" a first-class
test result alongside "ok" and "not ok".

The important consideration being that the way TAP traditionally
separates "pass/fail" and "directive" has its problems.  Not every
test can be said to pass or fail.

> In reading through the docs, this almost sounds like a conceptual
> mistake.  If you call "perror", the status of the next test is
> UNRESOLVED...
*snip*

Don't worry about the details of what Dejagnu does with it.  The
important thing is that this is a test result we have no good way of
expressing in TAP.  "I can't figure out what the result is, call a
human".  We'd probably just call that a fail.

Consider what you could do with that.  When installing a module, when
a test suite gets an UNRESOLVED result, rather than just failing, it
could tell the user so, display them the information and ask the user
what they want to do with it.  For example, DBD::mysql keeps failing
to install on my machine because it can't seem to get access to my
test database.  Rather than failing it would be nice if it could just
ask me what to do, at which point I could maybe fix the access
permissions and let it try again.

I bring this up because TAP is the Test Anything Protocol, and here's
something it can't express.

> Other thoughts: I notice that DejaGnu automatically supports a log
> and I think that this is good (tempted to write Test::Log).
> Seemless integration with something like Log::Log4Perl could be
> beneficial.  I'm trying to do something conceptually similar with
> App::Prove::History, but I'm using an SQLite database to back it up
> because test results over time are important.  That's raised a

Something that's happening as Test::Builder2 gets redesigned to handle
non-TAP styles is it's gaining the ability to handle the
responsibilities normally handed by Test::Harness.  This is because
most test systems the test program does all the work and it's damn
convenient, because it gives you, the test author, complete control
with complete information.  The downside is it gives the person
running the test less control, and they're the ones who best know how
they want the results presented.

PHPUnit draws a line down the middle.  They handle displaying their own
results, but they also have a rich set of logging facilities that can output
in several formats (TAP is one of them).

Anyhow, currently Test::Log could be done by giving Test::Builder a
tied filehandle to output to.  That tied handle would then pass the
TAP to STDOUT/STDERR as normal and then do something else with it,
like write it to a file or a database.

This approach, doing the logging in the test program rather than the
test harness, works well when the people running the tests are the
same people writing the tests.  Tests for internal projects.

> First, how do I identify a test program, much less a test?
> Currently I'm going by filename, but if someone renames a file, I
> lose.  I've thought about optional Subversion and Git integration
> (and others, but CVS fails) which can track via source control the
> file renames, but that's waaaaay down in the priority queue.

You could do a checksum of the file's contents.  That survives across
renames.  If the file was modified you can't usefully compare the
results from a previous run... at least if you're being a stickler
about it.  You could do something clever and run the programs through
a filter that removes any irrelevant code -- whitespace, comments, maybe
test names and diagnostic strings -- before checksumming.

You could just save the whole test file alone with the results.  That
seems the simplest and most reliable.  Disk is cheap.  Alternatively,
if you have the information, you could just point to a revision in
version control or whatever is necessary to recreate the code that was
run.  That has the benefit of being able to store the entire state of
the code.

"What is a test" and "when does a test change" are subjective problems, so in
order to solve it one first needs to know why you need the answers to those
questions.  If the end result is to be able to reproduce the test result, that
means you need to archive enough information to reproduce the entirety of the
code base and the environment.  If it's just so you can identify the test,
then maybe repository URL, line and revision are all you need.

> There's also the question of "what's a test?"  If you're on this
> list, you probably know that XUnit has test methods, each of which
> is considered a test, and has various asserts in each method.  The
> asserts are *not* tests.  We can resolve part of the issue there
> with nested TAP, but that brings me back to a more immediate
> problem.
>
> People want to know the start and end times (or duration) of each
> test.  Here's why this fails with Test::More:
>
> I could potentially hook the start time to be the end time of the
> previous test and that will mitigate it, but if you load expensive
> fixtures on start up, you would need something like this to start
> accurate timing:
>

> But you know, that's just clunky.  And super_expensive_function()
> *is* part of the test but Test::More and friends really don't treat
> it that way.  Aside from test duration, are there other implications
> of this we don't know about?  If that function throws an error or
> warning, should that be logged with the test?

Since TAP is just a stream of results, and Perl tests are just
programs, it's not possible to automate picking a "unit" out.  You need
some sort of structure.  The test author has to define the "unit", where one
"unit" ends and another begins.  You can't just guess except in some very
specific circumstances.  That way lies the halting problem.

In typical Perl testing, the .t file is your basic unit and that's
what you'd time.  If that's not a fine enough granularity, the stock
answer is that your .t files are too large.  If you're smashing .t
files together, you can still remember where the file boundaries were
and do the timing like that.

Another approach is something like Test::More::Behaviours [1].  It
provides a nice halfway between xUnit test methods, simple lexical
blocks of tests and the blat of unstructured code in a .t file.  It
defines the start and end of a group of tests so you have a point to
do setup and teardown which could include timing.

If you're doing the timing on the TAP interpreter end, you're out of
luck.  The .t file is the only unit you have.

Maybe this can be solved by allowing timing points in the TAP results?
Nothing more than a timestamp which can be manually entered.  That way the
program can communicate its units to the TAP interpreter.  This will likely be
part of nested TAP.

[1]  http://github.com/rija/test-more-behaviours/tree/master which used to be
Test::More::LikeXUnit and might have come out of the BBC.

-- 
Whip me, beat me, make my code compatible with VMS!
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap