Subject: Re: [tap] [offlist] Re: Parse error vs failure

From: Aristotle Pagaltzis <[email-masked]>
Date: Sun, 1 Feb 2009 09:57:41 +0100

* Michael G Schwern <[email-masked]> [2009-02-01 00:05]:
> Aristotle Pagaltzis wrote:
> > Basically. I think we should differentiate between
> > syntactical and semantic errors.
>
> Assuming a semantic error is treated as a failure, I'll take
> that as a yes.

Yes. :-)

I just didn’t quite like the “failure-with-warning” term: it
feels a bit too prescriptive. In practice most processors will
treat semantic errors that way, but I’d prefer to do the same as
the Atom WG, where we said we just tell you *exactly* what the
bits mean but what to do with them is up to you.

Although I guess “failure” can be seen as merely a meaning for
the stream as a whole, in which case “failure with warning” would
be another, in which case this all would be mere nitpicking. I
don’t care about the precise terminology though, as long as the
principle is kept in mind: don’t prescribe processor behaviour,
just describe document meaning.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Ovid <[email-masked]>
Date: Sun, 1 Feb 2009 05:36:45 -0800 (PST)

----- Original Message ----

> From: Michael G Schwern <[email-masked]>

> In Oslo, when we came across an edge case that was considered nonsense, the
> tendency was to say that its a parse error.  For example, "not ok 4 # SKIP",
> because why would you fail a test that didn't run?  Also "1..0" without a
> skip, because why would you not run any tests and yet not skip the test?

Your arguments made sense, but I have a question:  what does 'not ok 4 # SKIP' even mean?  To my mind, that's a parse error because something has gone horribly wrong or the person writing the TAP producer is terribly confused.  There's no way the parser can understand what the intent is, so making it a parse error makes sense to me because if *I* can't parse it, how can a TAP consumer?

I know I'm the odd man out here and I don't feel strongly enough about this to fight it :)

Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Steffen Schwigon <[email-masked]>
Date: Sun, 01 Feb 2009 21:41:35 +0100

Ovid <[email-masked]> writes:
>> In Oslo, when we came across an edge case that was considered
>> nonsense, the tendency was to say that its a parse error.  For
>> example, "not ok 4 # SKIP", because why would you fail a test that
>> didn't run?  Also "1..0" without a skip, because why would you not
>> run any tests and yet not skip the test?
>
> Your arguments made sense, but I have a question: what does 'not ok
> 4 # SKIP' even mean?  To my mind, that's a parse error because
> something has gone horribly wrong or the person writing the TAP
> producer is terribly confused.  There's no way the parser can
> understand what the intent is, so making it a parse error makes
> sense to me because if *I* can't parse it, how can a TAP consumer?

Example story:

I, for instance, mark failing results *dynamically* as "# TODO" under
cirumstances when I generically convert results from foreign test
suites into TAP. This way I keep them "ok enough" but can still
recognize the failure.

I once had the idea to use "# SKIP" to model slightly different
situations from those foreign suites.

I quickly reverted due to some amount of stupidity in it. But it's
anyway a nice niche for expressing test results.

I therefore wouldn't strongly forbid them, particularly not early in
the grammar.

Kind regards,
Steffen 
-- 
Steffen Schwigon <[email-masked]>
Dresden Perl Mongers <http://dresden-pm.org/>
German Perl-Workshop 2009 <http://www.perl-workshop.de/en/2009>
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Mon, 2 Feb 2009 06:20:27 +0100

* Ovid <[email-masked]> [2009-02-01 14:40]:
> Your arguments made sense, but I have a question: what does
> 'not ok 4 # SKIP' even mean? To my mind, that's a parse error
> because something has gone horribly wrong or the person writing
> the TAP producer is terribly confused.

You seem to have parsed it successfully far enough to discover
a contradiction in its meaning. In this way it seems to me that
`not ok 4 # SKIP` is different from `os9aso2kk2100#asz`, which
doesn’t mean anything at all.

Regards,

---

From: Andy Armstrong <[email-masked]>
Date: Mon, 2 Feb 2009 12:15:00 +0000

On 31 Jan 2009, at 20:18, Michael G Schwern wrote:
> Consensus?  Generally prefer failure-with-warning over parse error  
> for valid
> but sort of nonsense output?

Yeah, they're semantic rather than syntactic errors.

-- 
Andy Armstrong, Hexten

---

From: Andy Armstrong <[email-masked]>
Date: Mon, 2 Feb 2009 12:15:35 +0000

On 31 Jan 2009, at 21:37, Aristotle Pagaltzis wrote:
> * Michael G Schwern <[email-masked]> [2009-01-31 21:20]:
>> Generally prefer failure-with-warning over parse error for
>> valid but sort of nonsense output?
>
> Basically. I think we should differentiate between syntactical
> and semantic errors.

Oops. Note to self: read all of your mail backlog before replying :)

-- 
Andy Armstrong, Hexten

---

From: Michael G Schwern <[email-masked]>
Date: Tue, 03 Feb 2009 22:57:40 -0800

> about this to fight it :)

I think the important thing here is the difference between a syntax
error and a semantic error.  Let me illustrate:

"The color eight died" is syntactically correct.  There's the noun
"eight", modified by the adjective "color" and the verb "died" which
"the color eight" did.  There is meaning there to be parsed.  You can
draw a sentence diagram for it.  It is English.

"The color eight died" is a semantic error.  It is nonsense.  Eight is
not a color and numbers don't die.

You can have something which is syntactically correct but semantic
nonsense, and there is still meaning in semantic nonsense.

"not ok 4 # SKIP" is syntactically correct.  It applies to test #4.
That test failed.  The code was not run.

"not ok 4 # SKIP" is semantic nonsense.  You don't fail a test you
never ran.

Pragmatically, something has gone terribly wrong and the consumer
should flag that line.  So the desired result is still there for a
syntax error or a semantic error.  I hope that's clear, it still
results in an error.

This is, to a certain extent, splitting hairs, but that's part of what
formalizing TAP is about.  However, there are practical implementation
advantages to making it a semantic error.  Consider a grammar-based TAP
parser, rather than using ad-hoc regexes.

* Simpler grammar.

A test result line can be described as:

Test-Result [SP Test-Number] [SP Description] [SP # SP Directive [SP Reason]]

Allowing all combinations of Test-Results with Directives.  Meaning is applied
after they are parsed.

Whereas if "not ok 4 # SKIP" is made a syntax error that exception has
to be written into the grammar so we need to spell out all the combinations:

Test-Result-Line = Simple-OK / Simple-Fail / Passing-TODO /
                   Failing-TODO / Passing-SKIP
Simple-OK       = "ok" [SP Test-Number] [SP Description]        ; pass
Simple-Fail     = "not ok" [SP Test-Number] [SP Description]    ; fail
Passing-TODO    = Simple-OK   SP TODO-Directive                 ; pass
Failing-TODO    = Simple-Fail SP TODO-Directive                 ; pass
Passing-SKIP    = Simple-OK   SP SKIP-Directive                 ; pass

Which you might want to do to grammatically determine passing or failing, but
it does leave us with no basic definition for a test result line.

* All syntax errors are ignored.

Since we ignore all syntax errors, a failing skip test must be detected and
treated specially.  That means it must be valid syntax!

Failing-SKIP    = Simple-Fail   SP SKIP-Directive               ; error

* Better error reporting.

This is the most pragmatic reason.  Since "not ok 4 # SKIP" is an
actual error, as opposed to an ignored line, you're going to
have to detect it and parse it anyway to inform the user.

Consider if you see:

TAP Version 14
1..3
ok 1
not ok 2 - no database # SKIP
  ---
  file: foo.t
  line: 42
  ...
ok 3

Now, what would you rather tell the user?  If it's just a syntax
error, then all you get is:

  Syntax error at TAP stream line 4.
  Diagnostics without test at line 5.

But if you parse and understand it, you can use that information to say:

  Nonsense failing skip test "no database" at foo.t line 42.

pointing the user at the problem.  Isn't that better?

-- 
Robrt:   People can't win
Schwern: No, but they can riot after the game.

---

From: Steffen Schwigon <[email-masked]>
Date: Wed, 04 Feb 2009 23:29:17 +0100

Michael G Schwern <[email-masked]> writes:
> "The color eight died" is a semantic error.  It is nonsense.  Eight
> is not a color and numbers don't die.
>

The English comparison is a good idea.

The sentence might still be called lyrics or poetry. No one would try
to disallow this in the English language just for semantic reasons.

I wonder why we cannot model the SKIP semantics similar to #TODO. With
results like "is_actual_ok" to access the raw "ok/not ok" but
interpret the line as "ok" for the stats.

>   Diagnostics without test at line 5.

That's why we all already agreed that we do not want it a syntax
error.

> pointing the user at the problem.  Isn't that better?

As I said, why not just "fix" the stats, similar to what #TODO does
and provide details via "is_actual_ok" and friends?

Kind regards,
Steffen