Subject: [tap] Solution to the character set problem: Don't solve it.

From: Michael G Schwern <[email-masked]>
Date: Tue, 20 Jan 2009 12:58:09 -0800

character set is TAP going to use?  ASCII?  UTF-8?  Full Unicode?  Sticky.

Fortunately, I have a solution.  TAP will have NO CHARACTER SET!  That's
right, just not going to specify it.  But not in a sniveling "the character
set must be agreed upon by the parties interchanged" way.  We just word it so
the user can use any character set.

The critical observation is this:  there are really only a handful of places
were free-form strings are allowed in TAP.  1) Directive reasons  2)
Descriptions  3) Comments and  4) Ignored lines.  All the rest is fixed
strings which we can strictly specify.  Rather than specifying these things as
what they can be, we can specify them as what they can *not* be.  This handily
lines up with how you'd implement it.

For example, look at a complete TAP test result line.

    ok 3 Basset hounds got long ears # TODO This is a corgi

In order to match the test description you simply look for everything between
the test number and the #.  A test description can contain everything *except*
a "#" or EOL.  Similarly, a directive reason is just anything from the
directive itself to the EOL.  You can put any gibberish in there you like, its
still parsable and can be passed through to the user's display.

This makes the protocol simpler and less restrictive.  It makes it easier to
test non-Unicode apps.  Let's say you're working on a Big5 [1] application,
you'd naturally want to display your test results in Big5.  If TAP went all
Unicode you'd have to translate your Big5 results into Unicode muddying their
value as diagnostics.  Similarly, if you're working on a Big5 application you
probably have your terminal set to handle Big5 thus reading raw TAP output
with Big5 descriptions is most natural.

This solution continues with TAP being able to test ANYTHING.

[1] https://secure.wikimedia.org/wikipedia/en/wiki/Big5

-- 
"I went to college, which is a lot like being in the Army, except when
 stupid people yell at me for stupid things, I can hit them."
    -- Jonathan Schwarz

_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael Peters <[email-masked]>
Date: Tue, 20 Jan 2009 16:11:19 -0500

> the user can use any character set.

My head still swims a little when talking about character sets, so I might be wrong, but I don't 
think this will work. Without an agreed upon character set, all you have are meaningless bytes.

> still parsable and can be passed through to the user's display.

But how do you tell if a certain byte (or sequence of bytes) maps to the "#" character. Without an 
encoding you can't tell. "#" is the same byte in Latin1, ASCII, UTF-8 and Big5 but is not the same 
byte (or bytes) in UTF-16, UTF-32, UCS-2 or EBCDIC.

-- 
Michael Peters
Plus Three, LP

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 20 Jan 2009 22:30:09 +0100

>the user can use any character set.

That would be problematic as far as the IETF is concerned, as per BCP 18
(currently RFC 2277): "All protocols MUST identify, for all character
data, which charset is in use." So if you do not pick a single encoding,
you have to have some way to identify which encoding is being used.

>still parsable and can be passed through to the user's display.

That is a bit of a stretch, you assume that the user's display accepts
essentially arbitrary byte input, that is not always the case, and may
in fact lead to security problems. Consider the case of generating HTML
based on some TAP stream, you cannot have random byte streams in HTML
documents, you have to convert them to characters first. Otherwise you
have a hard time e.g. to filter out malicious <script> elements.

>with Big5 descriptions is most natural.

But then of course if you do not have your terminal set to Big5 you get
gibberish, which you may or may not realize, and are less likely to know
how to fix (besides, as the Wikipedia article points out, there are some
variants of Big5, so your terminal's Big5 may be a different one than
the TAP generating application's "Big5").
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 20 Jan 2009 22:34:02 +0100

>byte (or bytes) in UTF-16, UTF-32, UCS-2 or EBCDIC.

I think the idea is to rule out using those encodings directly. This is
not uncommon, for example, e-mail / MIME messages depend on encodings
being ASCII-compatible (ascii characters map to ascii octets, ascii
octets map to ascii characters). Encodings like UTF-16 can only be used
of additional transformation takes place (e.g. quoted-printable or
base64 encoding).
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael G Schwern <[email-masked]>
Date: Tue, 20 Jan 2009 14:15:10 -0800

> But how do you tell if a certain byte (or sequence of bytes) maps to the
> "#" character. Without an encoding you can't tell. "#" is the same byte
> in Latin1, ASCII, UTF-8 and Big5 but is not the same byte (or bytes) in
> UTF-16, UTF-32, UCS-2 or EBCDIC.

The important grammatical parts are specified as bytes.  We already get that
by using ABNF.  "#" is always going to be the byte x23.  So the grammatical
parts of the spec are unambiguous.

Yes, this means if you output raw TAP on a non-ASCII compatible display it
will look funny, but if you have a non-ASCII display you likely already know
how to cope.

One simple way to cope is to run the TAP through a simple parser which
translates the raw ASCII TAP into your native character set changing, for
example, ASCII "ok" into your native "ok".

-- 
39. Not allowed to ask for the day off due to religious purposes, on the
    basis that the world is going to end, more than once.
    -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
           http://skippyslist.com/list/
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael G Schwern <[email-masked]>
Date: Tue, 20 Jan 2009 14:41:57 -0800

> * Michael G Schwern wrote:

Maybe I wasn't being clear, because your comment about a security hole via
<script> I don't get.

It's ASCII (or something ASCII compatible) for the grammatical parts with
reasons and descriptions just being byte streams minus certain bytes
(representing things like ASCII "#" and end of line).

Basically, you can think of descriptions and reasons as just being byte
streams with a special byte saying it's done.  The grammar looks like:

EOL             = LF / CRLF
Test-Result     = Status [SP Test-Number] [SP Description]
                   [SP "#" SP Directive [SP Reason]] EOL
Status          = "ok" / "not ok"       ; Whether the test succeeded or failed
Description     = 1*(%x01-0b %x0c %x0e-22 %x24-7f)
                                        ; Anything but EOL and #
Directive       = "SKIP" / "TODO"
Reason          = 1*(%01-0b %0c %0e-7f) ; Anything but EOL

So a Description is a stream of bytes that goes until a # or EOL.  A Reason
goes until EOL.  How those bytes are interpreted is up to the user.

> of additional transformation takes place (e.g. quoted-printable or

And I hate them so much.  It's a kludge to bring an ASCII protocol into a
post-ASCII world.  I'd rather design something straight in.

> the TAP generating application's "Big5").

That's ok.  If you're reading raw TAP involving Big5 and you weren't expecting
it... why were you reading it in the first place?  This is one of those cases
were I'm perfectly happy to use the ISO 8601 cop out of "will be agreed upon
by the parties involved in the exchange".  This is less odious then
restricting output to UTF-8 or something and have to do MIME encoding (which
I'm not sure how you'd wedge that into a description) if you want your
descriptions in, say, Japanese.

Fortunately, a program to translate raw TAP into your native format is trivial.

-- 
124. Two drink limit does not mean first and last.

---

From: Michael Peters <[email-masked]>
Date: Tue, 20 Jan 2009 17:55:20 -0500

> (representing things like ASCII "#" and end of line).

So we are specifying a character set. ASCII (or compatible in the lower bytes). And if I understand 
you correctly, you're saying the rest can be in whatever encoding you want. So basically TAP streams 
can be mixed-character sets, right? That sounds like a can of worms to me, especially if you have no 
way to specify what that other character set is.

Also, since we aren't giving any meaning to the bytes, what if the x23 byte (#) is the 2nd byte in a 
multi-byte character? How will our character encoding ignorant parser pick up on this?

-- 
Michael Peters
Plus Three, LP

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Wed, 21 Jan 2009 00:24:21 +0100

]:
> When writing the TAP spec the problem of character sets comes
> up. What character set is TAP going to use? ASCII? UTF-8? Full
> Unicode? Sticky.
>
> Fortunately, I have a solution. TAP will have NO CHARACTER SET!

That’s not a solution, it’s a non-solution.

Just saying “it’s all octets as far as TAP is concerned” will
lead to the same mess as Unix filesystems are in: GUIs cannot
reliably display filenames because they’re just octets and the
textual meaning depends on the locale of the user who created
the file, as well as the phase of the moon.

Consider trying to collect TAP streams over HTTP from many
different sources. Is it safe to mix them together? Who knows?
It’s all octets! How do I display them? I guess I should just
show the user hexdumps. Hooray.

Personally I would say something like “it’s all UTF-8 by default
unless you signify otherwise with a special directive” (which we
will have to invent). A notation for roundtripping actual non-
textual octet sequences would be useful also. We could mandate
something inherently programmer-readable for this purpose.

Sorry, there is no winding out of this issue.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Ovid <[email-masked]>
Date: Tue, 20 Jan 2009 16:00:41 -0800 (PST)

> From: Michael G Schwern <[email-masked]>

> When writing the TAP spec the problem of character sets comes up.  What

I wasn't aware that there *was* a problem with character sets.  Why not UTF8?  What's wrong with that?  If you can articulate *why* this is a problem, I can better understand your solution.

Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

---

From: "David E. Wheeler" <[email-masked]>
Date: Tue, 20 Jan 2009 16:01:32 -0800

> something inherently programmer-readable for this purpose.

+1

Encodings are hard. Let's get them right rather than trying to ignore  
the problem.

Best,

David

---

From: "David E. Wheeler" <[email-masked]>
Date: Tue, 20 Jan 2009 16:02:33 -0800

> I wasn't aware that there *was* a problem with character sets.  Why  
> not UTF8?  What's wrong with that?  If you can articulate *why* this  
> is a problem, I can better understand your solution.

Some people want to use encodings other than UTF-8 in their TAP  
output. Think about a test failure where an exception thrown by the  
app includes an error message written in Big5.

Best,

David

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 21 Jan 2009 01:30:26 +0100

>Maybe I wasn't being clear, because your comment about a security hole via

So how would you write a general purpose tap2html script that can safely
support untrusted tap streams? For the HTML code you have two options,
either you declare some character encoding, or you declare none. If you
declare none, a browser will try to auto-detect it based on the bytes in
the document. As an attacker would control the bytes, he could smuggle
in a certain sequence that would make the browser guess poorly, and then
another sequence that would do something malicious.

If you do specify an encoding, you may not have accounted for unusual
sequences that would need escaping, in particular, if the tap document
was not in the encoding you specified for the HTML code, there may be
errors. Generally either case could be exploited. So it would be quite
difficult to write a general purpose automated tap2html script, more so
if people are careless and stuff data of varying encodings into a single
tap stream.

To give two simple examples, if you do not declare an encoding, some
browsers may autodetect UTF-7 which is a strange format that maps any
Unicode string into only ascii octets. There are very many ways to en-
code a string like "<script>" in it. Another example, if you declare the
us-ascii encoding, some browsers will treat bytes with the 8th bit set
as if the 8th bit wasn't set, so you have two ways to render <script> in
that encoding.

In contrast, if you'd have some declaration ala

  TAP version 14 encoding big5

You can check that all the strings actually are in Big5 and transcode
them to whatever encoding you need, or just echo it after checking and
proper escaping.

>by the parties involved in the exchange".

You are assuming that it is not desirable to have an application that
just takes random tap streams and does something interesting with them
that involves processing the strings. I am unsure how valid that is. For
example, getting meaningful diagnostics from your build system via email
would seem a worthwile functionality. But the writer of the tap2mail
script cannot know you are using big5, or utf8, or whatever, yet the
script needs to know to produce a proper email message from it.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Michael G Schwern <[email-masked]>
Date: Tue, 20 Jan 2009 16:46:01 -0800

> another sequence that would do something malicious.

I think the key difference here is I'm talking about "<script>" as bytes 60
115 99 114 105 112 116 62 which just happen to come out as "<script>" in
ASCII.  You're talking about "<script>" as being whatever bytes "<", "s",
etc... turn out to be in the given character set.  I fully agree that the
latter is madness.

There is no guessing about the TAP grammatical elements.  Things like "ok" and
"SKIP" are a fixed series of bytes, full stop.  I'm only talking about the
reasons and descriptions which would be the equivalent of HTML CDATA.

> them to whatever encoding you need, or just echo it after checking and

That's an interesting proposal for an extension.

-- 
24. Must not tell any officer that I am smarter than they are, especially
    if it's true.

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Wed, 21 Jan 2009 03:35:14 +0100

]:
> I'm only talking about the reasons and descriptions

Right, and if there is no declared encoding for those, then the
only *safe* way to display them is as a hexdump. After all it’s
just a bunch of bytes with no associated encoding. That’s not the
height of userfriendliness.

> which would be the equivalent of HTML CDATA.

There is no such thing as CDATA in HTML. It only exists in XML.
And it’s not equivalent then either: the content of CDATA
sections is textual data in the same encoding as the rest of the
document.

> That's an interesting proposal for an extension.

Haven’t I seen this somewhere before…? *headscratch*

… oh wait:

    <?xml version="1.0" encoding="utf-8"?>

:-)

(Exactly the kind of thing I was saying we need, obviously.)

Regards,

---

From: Michael G Schwern <[email-masked]>
Date: Tue, 20 Jan 2009 18:46:52 -0800

> I wasn't aware that there *was* a problem with character sets.  Why not UTF8?  What's wrong with that?  If you can articulate *why* this is a problem, I can better understand your solution.

As in the Big5 example, if you're testing a program using Big5 (or anything
that's not UTF8 compatible, Latin 1?) but TAP can only output in UTF8, now you
have the awkward problem of translating all your Big5 strings into UTF8 before
outputting them.  This adds unnecessary complexity to the simple case where
TAP producer and parser are run by the same person.  And the translation means
diagnostic information might be lost or distorted.

I think this is one of those issues which we can't enforce.  Whatever
character set we specify for TAP strings is really irrelevant as it's all just
interpretation.  If we adapt the grammar I proposed earlier, where it's just a
byte stream with certain bytes forbidden, you can still throw a Big5 string
into a TAP description and the parser will deal with it just fine.

Language like this should cover it:

"TAP strings must be in UTF8 unless the interchange parties agree otherwise". [1]

This allows any given TAP displayer to assume UTF8 unless told otherwise,
while giving enough wiggle room for non-UTF8 programs to be tested without
violating the TAP spec and without having to do a bunch of translation.  And
it doesn't change the parser one bit.  In fact, it lines up with current
implementation.  TAP::Parser::Grammar matches descriptions as "anything but a #".

[1] That "interchange parties" language being the standard issue cop-out from
ISO 8601.

-- 
I'm pale as formica, social skills stunted small. But I'm accurate
like a pica, I know the capital of Nepal. I'm the nemesis of error,
dreadful diction fears my skills, more inquisitive than Jim Lehrer,
snottier than Beverly Hills.
    -- I.L.O.P. Secret Rap  http://goats.com/archive/020830.html
_______________________________________________
tap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tap

---

From: Andy Armstrong <[email-masked]>
Date: Wed, 21 Jan 2009 12:31:41 +0000

> But how do you tell if a certain byte (or sequence of bytes) maps to  
> the "#" character. Without an encoding you can't tell. "#" is the  
> same byte in Latin1, ASCII, UTF-8 and Big5 but is not the same byte  
> (or bytes) in UTF-16, UTF-32, UCS-2 or EBCDIC.

What we've implemented is that TAP < v13 is ASCII (or some superset  
thereof) and TAP >= v13 is UTF-8.

I'm working on the assumption that, because UTF-8 is already a  
variable length encoding and has codespace for future expansion, it's  
the most future proof encoding extant. I don't think that's a  
particularly daring assumption :)

-- 
Andy Armstrong, Hexten

---

From: Steffen Schwigon <[email-masked]>
Date: Wed, 21 Jan 2009 14:13:52 +0100

---

From: Michael Peters <[email-masked]>
Date: Wed, 21 Jan 2009 09:06:05 -0500

icode.com> writes:
>> On Jan 20, 2009, at 3:24 PM, Aristotle Pagaltzis wrote:
>>
> +1 me too.
> It also nicely fits into what Andy said about the current state beeing
> v12 default ASCII, v13 default UTF8, and room for v14 with specified
> encoding, eg. like Bjoern suggested.

+1 from me as well

-- 
Michael Peters
Plus Three, LP

---

From: Michael Peters <[email-masked]>
Date: Wed, 21 Jan 2009 09:10:06 -0500

> that involves processing the strings. I am unsure how valid that is. 

Very valid. In fact, this is what Smolder already does. It takes random TAP archives and displays 
them in an interactive web application. If people are going to dump descriptions in some random 
encoding without specifying what that is, there's nothing Smolder can do but show junk.

Any solution that works in a terminal for a live TAP stream, but not for a recorded TAP stream 
(which is what a TAP archive is) is fatally flawed IMO. The recorded stream should be self contained 
enough that it can be parsed without any outside knowledge.

-- 
Michael Peters
Plus Three, LP