Subject: [idn] Fwd: I-D ACTION:draft-josefsson-idn-test-vectors-00.txt

From: Paul Hoffman / IMC <[email-masked]>
Date: Fri, 28 Mar 2003 07:55:59 -0800

There is a new draft that is of very direct interest to this mailing 
list. It would be great if people who have implemented IDNA could 
check their results against this draft. And thank you to Simon for 
doing the work on this! These kinds of documents are very useful in 
the IETF and can help IDNA progress on standards track.

--Paul Hoffman

>To: IETF-Announce: ;
>From: [email-masked]
>Reply-to: [email-masked]
>Subject: I-D ACTION:draft-josefsson-idn-test-vectors-00.txt
>Date: Fri, 28 Mar 2003 07:22:21 -0500
>Sender: [email-masked]
>
>
>
>A New Internet-Draft is available from the on-line Internet-Drafts 
>directories.
>
>
>	Title		: Nameprep and IDNA Test Vectors
>	Author(s)	: S. Josefsson
>	Filename	: draft-josefsson-idn-test-vectors-00.txt
>	Pages		: 41
>	Date		: 2003-3-27
>
>This document contains test vectors for Nameprep and IDNA
>
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-josefsson-idn-test-vectors-00.txt
>
>To remove yourself from the IETF Announcement list, send a message to
>ietf-announce-request with the word unsubscribe in the body of the message.
>
>Internet-Drafts are also available by anonymous FTP. Login with the username
>"anonymous" and a password of your e-mail address. After logging in,
>type "cd internet-drafts" and then
>	"get draft-josefsson-idn-test-vectors-00.txt".
>
>A list of Internet-Drafts directories can be found in
>http://www.ietf.org/shadow.html
>or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>
>Internet-Drafts can also be obtained by e-mail.
>
>Send a message to:
>	[email-masked].
>In the body type:
>	"FILE /internet-drafts/draft-josefsson-idn-test-vectors-00.txt".
>
>NOTE:	The mail server at ietf.org can return the document in
>	MIME-encoded form by using the "mpack" utility.  To use this
>	feature, insert the command "ENCODING mime" before the "FILE"
>	command.  To decode the response(s), you will need "munpack" or
>	a MIME-compliant mail reader.  Different MIME-compliant mail readers
>	exhibit different behavior, especially when dealing with
>	"multipart" MIME messages (i.e. documents which have been split
>	up into multiple messages), so check your local documentation on
>	how to manipulate these messages.
>
>
>Below is the data which will enable a MIME compliant mail reader
>implementation to automatically retrieve the ASCII version of the
>Internet-Draft.
>
>
>[The following attachment must be fetched by mail. Command-click the 
>URL below and send the resulting message to get the attachment.]
><mailto:[email-masked]?body=ENCODING%20mime%0D%0AFILE%20/internet-drafts/draft-josefsson-idn-test-vectors-00.txt>
>[The following attachment must be fetched by ftp.  Command-click the 
>URL below to ask your ftp client to fetch it.]
><ftp://ftp.ietf.org/internet-drafts/draft-josefsson-idn-test-vectors-00.txt>

---

From: [email-masked] (Martin v. Löwis)
Date: 28 Mar 2003 19:17:30 +0100

Paul Hoffman / IMC <[email-masked]> writes:

> check their results against this draft.

Please correct me if I'm wrong: I believe the UTF-8 strings are wrong
in a number of tests:

4.3:  U+00DF should be \xc3\x9f, not \xc3\xdf
4.9:  U+01F0 should be \xc7\xb0, not \xc7\xf0
4.44: U+00DF should be \xc3\x9f, not \xc3\xdf
4.45: Likewise.

Regards,
Martin

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 28 Mar 2003 21:06:07 +0100

> 4.44: U+00DF should be \xc3\x9f, not \xc3\xdf

You are right.  I thought I could convert simple strings by hand, but
obviously I didn't select the proper UTF-8 encoding (in some cases).
I believe you catched all errors.  Since those examples test whether
the application uses a validating UTF-8 decoder, I will keep those
examples modified to result in an UTF-8 decoding error.
Non-validating UTF-8 decoders should produce the correct Unicode code
point for those UTF-8 encodings though.

---

From: [email-masked] (Martin v. Löwis)
Date: 28 Mar 2003 21:24:03 +0100

> point for those UTF-8 encodings though.

I suggest to correct the examples; those tests really test something
else. UTF-8 is not part of IDNA, and any self-respecting UTF-8 library
should have separate test cases, eg. the ones from RFC 2279.

(and yes, I probably found all such errors, as I literally copied the
test cases into my Python IDNA test suite).

Regards,
Martin

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 28 Mar 2003 21:55:20 +0100

I want to point out that nobody should be using non-validating UTF-8
converters; not only are they not conformant to the Unicode standard, they
also represent a security risk.

Mark
(ᛗᚪᚱᚳ)
________
[email-masked]
IBM, MS 50-2/B11, 5600 Cottle Rd, SJ CA 95193
(408) 256-3148
fax: (408) 256-0799

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Fri, 28 Mar 2003 12:51:52 -0800

>should have separate test cases, eg. the ones from RFC 2279.

I agree with Martin here: tests of UTF-8 don't belong in IDNA. 
Remember that IDNA is encoding-independent.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: John C Klensin <[email-masked]>
Date: Fri, 28 Mar 2003 16:28:21 -0500

--On Friday, 28 March, 2003 21:55 +0100 Mark Davis 
<[email-masked]> wrote:

> I want to point out that nobody should be using non-validating
> UTF-8 converters; not only are they not conformant to the
> Unicode standard, they also represent a security risk.

Mark, I clearly agree with this, and with Paul's point that 
UTF-8 is not part of IDNA.   But that, it seems to me, is an 
argument for Simon's tests containing those tests, rather than 
against it: if non-validating/ non-conforming UTF-8 converters 
are out there, then testbeds that catch them would seem to be in 
everyone's interest.

    john