Subject: [tap] "ok 1 description" vs "ok 1 - description"

From: Michael G Schwern <[email-masked]>
Date: Fri, 23 Jan 2009 18:07:12 -0800

According to the standard, it's supposed to be this:

ok 9 here is some descriptive text
ok 10 and some more stuff

But Test::More does this:

ok 9 - here is some descriptive text
ok 10 - and some more stuff

Why?  Because it's a whole lot easier to read.  Technically the description is
"- here is a some descriptive text".  I would like that to not be so while
still retraining the spacing.

So, in the interest of human readability, I propose a small change to the
grammar from:

Test-Result     = Status [SP Test-Number] [SP Description]
                   [SP "#" SP Directive [SP Reason]] EOL

to

Test-Result     = Status [SP Test-Number] [SP -] [SP Description]
                   [SP "#" SP Directive [SP Reason]] EOL

The dash is simply ignored if it's there.

Yay or nay?

-- 
87. If the thought of something makes me giggle for longer than 15
    seconds, I am to assume that I am not allowed to do it.
    -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
           http://skippyslist.com/list/

_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Ovid <[email-masked]>
Date: Sat, 24 Jan 2009 01:55:06 -0800 (PST)

----- Original Message ----

> From: Michael G Schwern <[email-masked]>

> The dash is simply ignored if it's there.

So everyone who's written a TAP producer in another language and
didn't see fit to add that dash will now be producing invalid TAP?

Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: "H.Merijn Brand" <[email-masked]>
Date: Sat, 24 Jan 2009 10:57:16 +0100

> The dash is simply ignored if it's there.

Yay

-- 
H.Merijn Brand  http://tux.nl      Perl Monger  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.3, and 11.0, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Paul Johnson <[email-masked]>
Date: Sat, 24 Jan 2009 12:54:27 +0100

> didn't see fit to add that dash will now be producing invalid TAP?

I don't think so, since the dash is optional.  The only real complication
I can see is if anyone actually does want to start the description with a
dash, in which case they would now need two dashes.  TAP from those
producers you mention above would have any dash prefix in the description
silently eaten.  Still not ideal, but perhaps not the end of the world.

(For values of dash that include appropriate whitespace.)

-- 
Paul Johnson - [email-masked]
http://www.pjcj.net
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Steffen Schwigon <[email-masked]>
Date: Sat, 24 Jan 2009 19:09:36 +0100

Paul Johnson <[email-masked]> writes:
> I don't think so, since the dash is optional.  The only real
> complication I can see is if anyone actually does want to start the
> description with a dash, in which case they would now need two
> dashes.  TAP from those producers you mention above would have any
> dash prefix in the description silently eaten.  Still not ideal, but
> perhaps not the end of the world.

IMHO when the TAP grammar changes things that were stable for years
it's near to the end of the world.

I, for instance, evaluate test results via the description over *lots*
of archived TAP streams since nearly years. I don't want to become an
outlaw.

So I'm strongly with Ovid.

And I don't know why the grammar is about to be discussed at
all. Until now I thought we would only think about backwards
compatible extensions that fit into the currently "ignored non-tap"
design space.

It would be ok if we were inventing TAP for the first time, but with
this 20 years old thingie I would be very carefully.

Kind regards,
Steffen 
-- 
Steffen Schwigon <[email-masked]>
Dresden Perl Mongers <http://dresden-pm.org/>
German Perl-Workshop 2009 <http://www.perl-workshop.de/en/2009>
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael G Schwern <[email-masked]>
Date: Sat, 24 Jan 2009 13:16:32 -0800

Steffen Schwigon wrote:
>> I don't think so, since the dash is optional.

Exactly, the dash is optional so if you leave it out that's fine.  Sorry for
not making that abundantly clear, I forget not everyone can read ABNF.

These two lines are semantically equivalent:

ok 1 foo
ok 1 - foo

They are both passing test number 1 with a description of "foo".

>> The only real

It's simple enough to deal with, uncommon enough not to be a worry and if you
screw it up it's just cosmetic (that is, it has no effect on passing or
failing).  So I'm not worried.

> of archived TAP streams since nearly years. I don't want to become an

Out of curiosity, what are you using to create these TAP streams and how do
they format the description?

> this 20 years old thingie I would be very carefully.

Seeing as how Test::More is the widest producer of TAP, and it's being doing
"ok 1 - description" for years now, and that the dash is optional so nothing
is being made illegal, all of the above would seem to actually be an argument
in favor of adding the optional dash to the grammar as it better describes
reality.

And it produces a nicer to read TAP moving forward, which is important, too.

As a point of reference, we are discussing the grammar exactly because we're
trying to write down reality.  But reality can be hard to nail down when you
try to formalize conventions.  Little niggling things pop up and assumptions
have to be examined.  Does "# SKIP" have to be capitalized or is "# skip" ok,
too? [1]  Do we allow more than one space between tokens? [2]  What's the
character set? [3]  Is "1..0\n" with no skip a passing test? [4]  Is it even
gramatically correct? [5]  What is the TAP end-of-line sequence? [6]  And, of
course, do we enshrine the "ok 1 - description" convention?

All of these little things suddenly become really important when you're trying
to write it all down, in detail, such that others can repeat your work.

To further drill this point into the ground, I noticed a grammatical ambiguity
when generating valid, but nonsense, TAP.  I came across stuff like this:

ok 8lasdjf;akjd;lfj

Which looked like a mistake in the Test-Number grammar until I realized that
was a Test-Result with no Test-Number.  "8lasdjf;akjd;lfj" was the
Description.  This suggests that the following is ambiguous:

ok 8

Is that test 8 passing?  Or is that a test with no number and a description of
"8"?  How do you produce a test with no number and a description of 8?
Without the dash in the grammar, you can't.  You have to say "ok
8not-a-test-number".  You can't even say "ok 8 not a test number" because
that's test number 8 with the description "not a test number".

Un-numbered tests are not just a curiosity (though numeric test descriptions
probably are).  They are important for unsynchronized, parallel testing such
as threads and forking.  The dash allows the test producer to remove this
ambiguity by saying "ok - 8".

Is this important?  Well, TAP is going from being a public interface, to being
a "published" interface.  I mean that in the sense Martin Fowler gave it to
mean "we no longer have control over all implementations and users" and thus
refactoring becomes impossible and backwards compatibility takes on a whole
new meaning.  It also means that people who have no connection to the TAP
community are going to write TAP parsers and producers based purely on the
spec and do REALLY CRAZY SHIT with any crack we leave in the protocol and then
we have to support it FOREVER. [7]

So I assure you that when I bring these things up, I'm trying to nail down the
Jell-O of reality.  I wholeheartedly agree with all my very being that we need
a spec of the current reality of TAP separate from any proposed extensions.

End of unsolicited rant.

[1] Answer, its case insensitive because it's easier to do that in ABNF and
TAP has traditionally been pretty inconsistent about enforcing case.

[2] Still up in the air.  I'm leaning towards more than one space as it gives
more flexibility in formatting for human readability and most parsers (by
which I mean Test::Harness and TAP::Parser) allow more than one space anyway.

[3] Recent discussion seems pretty strong that it's UTF8.

[4] No, though there's no strong reason it couldn't be.

[5] Yes, it is not a parse error.

[6] Newline?  Carriage return + newline?  Both?  This needs to be nailed down
if we're going to pass TAP streams around from machine to machine.

[7] To see this illustrated, please watch rjbs' brilliant rant against mail
formats, "Email Hates The Living".  Also it's funny.
http://video.google.com/videoplay?docid=7054401183589794595

-- 
'All anyone gets in a mirror is themselves,' she said. 'But what you
gets in a good gumbo is everything.'
    -- "Witches Abroad" by Terry Prachett
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: "David E. Wheeler" <[email-masked]>
Date: Sat, 24 Jan 2009 13:43:18 -0800

> didn't see fit to add that dash will now be producing invalid TAP?

I always include the dash in the JavaScript and PostgreSQL ports,  
because that's what Test::More does.

David

---

From: Steffen Schwigon <[email-masked]>
Date: Sat, 31 Jan 2009 23:24:48 +0100

Sorry for the late answer, I seem to be out of sync with the rest of
the world ...

Michael G Schwern <[email-masked]> writes:
> Steffen Schwigon wrote:
>> IMHO when the TAP grammar changes things that were stable for
>> years it's near to the end of the world.
>>
>> I, for instance, evaluate test results via the description over
>> *lots* of archived TAP streams since nearly years. I don't want
>> to become an outlaw.
>
> Out of curiosity, what are you using to create these TAP streams
> and how do they format the description?

They are created within a heterogeneous environment with Perl
Test::More, Python PyTAP, and "echo/print" from Shell or C.

To process the TAP, however, we use the Perl toolchain around
TAP::Parser.

> Seeing as how Test::More is the widest producer of TAP, and it's
> being doing "ok 1 - description" for years now, and that the dash
> is optional so nothing is being made illegal, all of the above
> would seem to actually be an argument in favor of adding the
> optional dash to the grammar as it better describes reality.
>

I see that the dash was meant optional for TAP. Maybe the
optionality of the dash changed in TAP::Parser 3? Admittedly, I
didn't try.

Anyway, the only useful current processing tool is TAP::Parser and
that slurps the dash as part of the description.

So it's probably not the producers like Test::*, PyTAP and
"print" but the history of the *consumer*, i.e. TAP::Parser,
that I'm worrying about.

Someone(tm) (not me, I know, sorry) could try out what happens
around CPAN (as the largest reference test suite) if the grammar is
changed in TAP::Parser.

> As a point of reference, we are discussing the grammar exactly
> because we're trying to write down reality.

I see. But wasn't the original plan to write down what the only
existing consumer toolchain aka. TAP::Parser does, because else no
one in the world has a toolchain?

Kind regards,
Steffen

---

From: Michael G Schwern <[email-masked]>
Date: Sat, 31 Jan 2009 15:41:01 -0800

> optionality of the dash changed in TAP::Parser 3? Admittedly, I

The last version of Test::Harness 2 before TAP::Parser strips the dash if its
there, you can see the code in Test::Harness::Point->from_test_line().
http://cpansearch.perl.org/src/PETDANCE/Test-Harness-2.64/lib/Test/Harness/Point.pm

But earlier versions of TH2 do not.

TAP::Parser (aka Test::Harness 3) does not strip the dash.

So we've already got some waffling around how the description is handled.  We
survived with hardly anyone noticing.

> one in the world has a toolchain?

What the existing consumers and producers do is being written down.  Within
that limit, the human readability and machine parsability of TAP and its
ability to extend in the future are being protected.

Presumably the consumers and producers are in sync, but in the case of the
description Test::More and TAP::Parser aren't.  Test::More intends it to be
stripped, TAP::Parser doesn't do the stripping.  Nobody really uses it for
anything but human readable display, and they probably strip as to their
preference, so it hasn't been a big deal.

It's important to remember that TAP::Parser was written based on what
Test::Harness 2, Test::More and Test.pm do.  Test::More was written based on
what Test::Harness 1 and Perl's core t/TEST TAP parser handled.  Test::Harness
1 was written based on t/TEST which was written to support the Perl core
tests.  Parsers effect consumers, consumers effect parsers.

TAP::Parser was the first attempt to try and make sense of all that historical
cruft from scratch, and this is the first attempt to formalize it.  I'd like
to not just enshrine whatever it is we're doing at the moment we write the
draft and actually put some thought into it.

As mentioned before, the interpretation of the description is open to some
interpretation because it's just an identifier for the human.  It doesn't
effect pass or fail.  For one legacy parser to interpret it as "- foo" and
another to say "foo" doesn't make much difference.  Going forward is what does.

In addition, the existence of the dash removes some ambiguities.  How is this
interpreted?

ok 1      foo

Is that "foo" or "     foo"?  Its an open question.  Because of the tendency
of TAP::Parser to use \s+ (ie. one or more whitespace characters) to separate
tokens it's probably interpreted as "foo".  I doubt that was by design but
rather an expedient way to write the regexes.  If we just went with what
TAP::Parser does we'd be left with no way to have a machine readable leading
space in a description.

As another example, some people like to indent sub tests.  Without a dash it
would come out like this:

ok 1 thing() didn't blow up
ok 2   it returned the right thing
ok 3   they lived happily ever after

Do we count the leading spaces as part of the description?  The dash removes
this ambiguity.

ok 1 - thing() didn't blow up
ok 2 -   it returned the right thing
ok 3 -   they lived happily ever after

The details of how a description is handled has a lot ambiguity and wiggle
room both in current implementation and in definition.  It is not yet boxed in
by what TAP::Parser currently does.

-- 
I am somewhat preoccupied telling the laws of physics to shut up and sit down.
	-- Vaarsuvius, "Order of the Stick"
           http://www.giantitp.com/comics/oots0107.html
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Steffen Schwigon <[email-masked]>
Date: Sun, 01 Feb 2009 01:52:14 +0100

Michael G Schwern <[email-masked]> writes:
> The last version of Test::Harness 2 before TAP::Parser strips the
> dash if its there
>

So it already flip-flop'ed in the short history

 TH2early (not stripped),
 TH2late  (stripped),
 TH3/TP   (not stripped),

?

I agree then that stripping the dash from description might not be so
end-of-world as I initially thought.

I'm maybe one of the very few who currently depend on the exact
descriptions but I can probably even change my evaluation tools.

Kind regards,
Steffen

---

From: Michael G Schwern <[email-masked]>
Date: Sat, 31 Jan 2009 18:57:16 -0800

> So it already flip-flop'ed in the short history

Yep, that's about right.  Nothing before TH2 did anything at all with the
description.

> descriptions but I can probably even change my evaluation tools.

Fortunately, it will be self consistent within one tool.

What are you doing with them?