Subject: Re: [tap] "ok 1 description" vs "ok 1 - description"

From: Steffen Schwigon <[email-masked]>
Date: Sun, 01 Feb 2009 21:28:32 +0100

Michael G Schwern <[email-masked]> writes:
> Steffen Schwigon wrote:
>> So it already flip-flop'ed in the short history
>> 
>>  TH2early (not stripped),
>>  TH2late  (stripped),
>>  TH3/TP   (not stripped),
>> 
>> ?
>
> Yep, that's about right.  Nothing before TH2 did anything at all
> with the description.

(I'm dreaming about TAP... my thoughts drip out incrementally...)

I'm getting unsure again because: It could also mean that not
stripping was ok and the stripping was an experiment that was reverted
with some reason with TH3. I don't know. Who doeas? $Ovid? @Andy? 
$You?

>> I agree then that stripping the dash from description might not be
>> so end-of-world as I initially thought.
>> 
>> I'm maybe one of the very few who currently depend on the exact
>> descriptions but I can probably even change my evaluation tools.
>
> Fortunately, it will be self consistent within one tool.

Hm, not neccessarily as I provide an API to evaluate the archived TAP
by descriptions. It would just work as long as I control the
evaluation environment, which I still do at the moment.

> What are you doing with them?

Archiving to track history of testruns and recognize result
changes. Probably similar to Smolder, I think, maybe more
finegrained. Therefore I care for single tests and their descriptions.

Kind regards,
Steffen 
-- 
Steffen Schwigon <[email-masked]>
Dresden Perl Mongers <http://dresden-pm.org/>
German Perl-Workshop 2009 <http://www.perl-workshop.de/en/2009>
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael G Schwern <[email-masked]>
Date: Tue, 03 Feb 2009 22:13:27 -0800

> with some reason with TH3. I don't know. Who doeas? $Ovid? @Andy? 

An intersection of the three.

The REALLY IMPORTANT thing to realize was that so few people do anything with
descriptions except maybe print them out that very little thought has been put
into their exact parsing.  So don't read too much into what the
implementations did/do.

TH2early didn't strip the description because it was the first TAP parser that
allowed you to do anything with the parsed results so it just wasn't considered.

TH2late did strip the description because it was conforming to what Test::More
did.

TH3 removed the stripping because it wasn't considered part of the fledging
protocol which we were just then starting to write down.

But in all that we never considered the simple possibility that the dash could
be an optional element.

> evaluation environment, which I still do at the moment.

Are you going to be passing interpretations of the descriptions back and forth
between different implementations?  That is, when does the interpretation of
the description matter to anything but a single implementation?

> finegrained. Therefore I care for single tests and their descriptions.

Looking at it another way, the WORST thing that happens with mixing a legacy
producer which isn't aware of the dash with a post-spec, dash stripping parser
is that the description might get interpreted wrong and you can't associate a
description with it's historical results (though that would indicate that the
test started using a different producer, which stretching this scenario even
further).  Since using the description as a unique test ID is already a
heuristic (something that was discussed in some depth at Oslo) your system
must already be resistant to this problem.

Finally, if the parser always strips the optional dash it should always come
out the same whether the producer uses a leading dash or not EXCEPT for one
contrived case.  When a description deliberately is supposed to have a leading
" -".  I can only think of a fairly contrived example, but if you wanted to
output, say, Morse code "- ... -.-", and your producer doesn't know about the
optional dash, it would say:

ok 1 - ... -.-

Whereas a compliant producer would say:

ok 1 - - ... -.-

A compliant parser would parse the former as "... -.-" and the latter as "-
... -.-".  Same intended meaning, different interpreted descriptions.

Is this edge case important?  It seems one simple way to allay your fears is
to simply go through your historical TAP archives and look for instances where
a description starts with "- ".  If all the descriptions in a stream start
with "- " assume it's Test::Builder output and ignore it.  If only some of
them do, then we've found our exception.

I'm interested to hear what the results are.

-- 
94. Crucifixes do not ward off officers, and I should not test that.
    -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
           http://skippyslist.com/list/

---

From: "H.Merijn Brand" <[email-masked]>
Date: Wed, 4 Feb 2009 09:07:14 +0100

> optional dash, it would say:

It would be nice to take this example in the docs, but then use

	- .- .--.

:)

-- 
H.Merijn Brand  http://tux.nl      Perl Monger  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.3, and 11.0, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

---

From: Michael Peters <[email-masked]>
Date: Wed, 04 Feb 2009 09:15:19 -0500

> optional dash, it would say:

What about a description that begins with a negative number? Like say:

   -1 < 0 ?

-- 
Michael Peters
Plus Three, LP

---

From: Leon Timmermans <[email-masked]>
Date: Wed, 4 Feb 2009 15:36:12 +0100

> What about a description that begins with a negative number? Like say:

As long as there is no space between the minus and the number, it
won't get chopped off.

Leon

---

From: Steffen Schwigon <[email-masked]>
Date: Wed, 04 Feb 2009 23:15:59 +0100

Hi!

[Don't worry. Just explanations follow, I will agree you in the end.]

[And I write some words bold, just for better readability, not
shouting.]

Michael G Schwern <[email-masked]> writes:
> Steffen Schwigon wrote:
>> Hm, not neccessarily as I provide an API to evaluate the archived
>> TAP by descriptions. It would just work as long as I control the
> Are you going to be passing interpretations of the descriptions back
> and forth between different implementations?

No, not back and forth, but through 3 independent layers:

 1. producer,
 2. abstract API (using TAP::Parser),
 3. query tools (using the API).

I am level 2, level 1 and 3 just use me.

> That is, when does the interpretation of the description matter to
> anything but a single implementation?

It's about beeing *the description is the interface*.

I do *not* provide TAP::Parser directly but provide an abstract API
which can search for a description and let others write third-party
tools which depend on what they themself pushed in the *beginning* of
the process queue and let them also query results at the *end* of the
queue about their own descriptions.

Then they depend on my middle layer (TAP::Parser) beeing *constant*
in these descriptions throughout the years.

Because if today they query results for "- DESCRIPTION" (because they
created it with dash) and TAP::Parser changes later then they will not
find the results because it's now just "DESCRIPTION".

Changing this behaviour was what I was worrying about in the beginning
of the discussion.

> Looking at it another way, the WORST thing that happens with mixing
> a legacy producer which isn't aware of the dash with a post-spec,
> dash stripping parser is that the description might get interpreted
> wrong and you can't associate a description with it's historical
> results

Near the truth. Indeed. It's about matching "historical results", yes.

My users own the "entry" and the "exit" of the queue. My task is to
provide the stable "middle layer".

> (though that would indicate that the test started using a different
> producer, which stretching this scenario even further).

Hm, not exactly. My problem is about starting to use a different
*middle layer* in an environment with different producers at the same
time.

> Since using the description as a unique test ID is already a
> heuristic (something that was discussed in some depth at Oslo) your
> system must already be resistant to this problem.

Fully correct. But *that* (i.e., create useful descriptions they can
later query again) is *their* responsibility, because they own "entry"
and "exit".

My layer is the middle one, therefore *my* task is about the stability
of the description.

> out the same whether the producer uses a leading dash or not

*EXACTLY*. That's why in the mean time I *FULLY AGREE* with you.

(I hope you made it through here. :-)

Because that would make my middle layer independent from the different
provider styles by "stabilising" or "harmonizing" the descriptions in
a defined way.

And all my reports from different producers would always be queryable
in the same way, independent from their "dash style".

> EXCEPT for one contrived case.  When a description deliberately is
> supposed to have a leading " -".  I can only think of a fairly
> contrived example, but if you wanted to output, say, Morse code

I *really* had to lough here. :-)

> Is this edge case important?

Of course. Supporting Morse descriptions is a strong MUST HAVE. :-)
No, not really.

> It seems one simple way to allay your fears is to simply go through
> your historical TAP archives and look for instances where a
> description starts with "- ".

I don't need to. For the above "stabilizing" effect I'm willing to fix
the third party tools around me to just query without dashes, once
that dash is optional. And in the mean time I need to be able to fuzzy
search anyway somehow.

Am I strange? Or is it my approach? I should probably print out this
mail and present it on the next yapc::europe. :-)

Kind regards,
Steffen