Subject: [tap] Grammar errors vs test failure. Can we put pass and fail in the grammar?

From: Michael G Schwern <[email-masked]>
Date: Tue, 20 Jan 2009 18:09:17 -0800

example...

1..0 SKIP foo
ok 1

1..1
not ok 1 # SKIP

1..1
ok 0

1..2 TODO 2 5;
ok 1
not ok 2

1..5 SKIP foo

We can specify all this as being illegal in the grammar.  For example, this
partial grammar specifies that a skip all can not have any test results.

Stream          = Header (Plan Body / Body Plan / Skip-All) Footer
Header          = [Comments] [Version]
Footer          = [Comments]

Plan            = "1.." Number-Of-Tests [SP Plan-Todo] EOL
Skip-All        = "1..0 SKIP" *(SP Reason) EOL
Plan-Todo       = "todo" 1*(SP Test-Number) ";"   ; Obsolete

But now this:

1..0 SKIP foo
ok 1

Is not a parsable, but failing, stream.  It is now unparsable.  It is not TAP.
 A TAP displayer can't do anything with it except spit it back at the user.

So as convenient as it would be to strictly spell out all the ways to produce
nonsense TAP in the grammar, I'm against it if it means making the nonsense
completely unparsable.  However, it would be nice to formally specify what a
passing and failing TAP document is.

What if Stream was split into Passing-Stream and Failing-Stream?  And then we
wrote out all the ways a stream can be considered failing?  Then a TAP parser
simply tries to parse a TAP document with Passing-Stream.  If that works, its
a pass!  If not they verify it's valid TAP with Stream and call it a fail. [1]

Obviously this will produce a much more complicated grammar, but I think
there's huge value here in making it very easy to write a TAP parser.  Much of
the complexity of TAP is in figuring out if it failed.

[1] I predict it's easier to spell out all the ways a TAP stream can pass then
all the ways it can fail.  Thus writing out a Passing grammar as opposed to a
Failing grammar.

-- 
We do what we must because we can.
For the good of all of us,
Except the ones who are dead.
    -- Jonathan Coulton, "Still Alive"

_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap