Subject: [idn] IDNs in email message bodies

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 23 Mar 2001 23:07:44 +0000

I just looked at the IDNs-in-MUAs talk (thanks to StarOffice).  I agree
with it regarding the mail headers, but I have doubts about performing
ACE conversions in message bodies.

If the Content-Type is text/plain, then the message body is plain
text, pure and simple.  There should be no ACE encoding or decoding
happening there.  You can include strings that happen to be (and look
like) ACE-encoded domain names, and you can include strings that happen
to be (and look like) non-ACE-encoded IDNs, and in either case you can
copy and paste them into your IDN-aware browser (or other application,
including the MUA itself) and it will work (because that application
will perform ACE encoding if necessary).

For MUAs that seek out URL-ish things in plain text message bodies and
turn them into hyperlinks, they may need to perform conversions on the
domain names for the purpose of creating the href attributes, but the
visible text of the link should not be altered.

AMC

---

From: [email-masked] (Bob Jung)
Date: Fri, 23 Mar 2001 16:58:43 -0800

I have not looked at the talk, but I'd assume that if you want to send
ACE-encoded email, then it should be treated as a MIME
"Content-Transfer-Encoding" just like QP or base64.

So, your message headers would include something like:

                Content-Type: text/plain; charset=UTF-8
   Content-Transfer-Encoding:  x-ACE

-bob

---

From: "James Seng/Personal" <[email-masked]>
Date: Sat, 24 Mar 2001 10:59:26 +0800

Erm, kind of bad idea to use x-ACE as TES for the whole body
content...We have qp and base64...lets leave it alone.

What the ppt is focusing on are the issues when IDN names appears on
RFC821 SMTP command, RFC822 headers (different depending whether it is
From, or Subject) or bodies. How can we work with ACE wrt to
encoding+TES used already in the body etc.

While this is interesting, I think if this topic develops further, we
should jump this to another mailing list, probably DRUM or other related
work on Mail.

-James Seng

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Sun, 25 Mar 2001 10:23:21 -0500

| I just looked at the IDNs-in-MUAs talk (thanks to StarOffice).  I agree
| with it regarding the mail headers, but I have doubts about performing
| ACE conversions in message bodies.

  The comments to this presentation were basically "Some good ideas, but
try again.  Too many layer violations."  

  Applications need to differentiate between IDNs in a resolution context 
from other contexts.  In the context of a message body or HTML document, 
for example, I would expect to see IDNs presented 'as is': ACE as ACE, and
native characters as native characters.  So for example users should be
able to do something like this in a web page:

  <A HREF="http://中國.com/">中國.com</A>

  The browser (or some piece of software) in this context is going to have 
to process the domain name here before it gets sent to the resolver.  At
least that's the model I've been operating under. :-)

  The WG is going to have to provide very explicit guidelines here in
the specification so that implementors know exactly what to do.  There's
going to be a lot of confusion on this.

  -bws

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 25 Mar 2001 14:33:15 -0800

> encoding+TES used already in the body etc.

Whether an IDN appears in a header or in the body is irrelevant. We can't
change the data.

First of all, users are likely to use whatever characters are available
from their charset in the message text, or they may change charsets if
they have to do additional encodings, but for the most part people will
use whatever characters (and encodings) they already have. Somebody
advertising a Norsk domain name will write and encode it with iso-8859-1
or iso-8859-16 or whatever. Similarly, a Japanese IDN will likely be
written and encoded in iso-2022-jp (or an alternative, maybe unicode, if
they need more characters). But for the most part, the domain names from
email addresses and URLs will be encoded in native characters. Users will
not be converting IDNs to ACE and then writing ACE strings into email
messages. Sorry.

Secondarily, it is not possible to rewrite message data since that
corrupts message signatures (Eudora bug, spell-checking a message after it
was signed invalidated the signature). That means IDNs can't be screwed
with, meaning they must be preserved as encoded by the sender.

If we accept both of those preconditions, then we also come to the
conclusion that there isn't a whole lot of difference we can make here,
other than to request that the client application (or the resolver) use
ACE conversion when they are presented with an IDN. If some additional
decoding is required -- such as converting an iso-2022-jp sequence into
utf8 first -- then that's something that needs to be pointed out. But we
can't really expect that users aren't going to type IDNs in their rich
format when they generate a message which also contains plain text from
some local charset.

Other areas of concern -- like passing Han IDNs when those characters
aren't available (all languages are problematic for charset=us-ascii) --
have the same problem and answer really. We can encourage implementations
(and users) to default to UTF8, and then leave it up to them to do the
proper conversion whenever a DNS lookup is issued. If they can't or won't
type the IDN in its native format then they can type it in as an IDN.
Chances are they will try to copy-and-paste, and hopefully a mailer or a
browser's input box will be smart enough to deal with it.

But what we cannot do is create a new encoding syntax and expect that it
will be used only for domain names. As pointed out above, it won't happen,
because changing data breaks messages, and because users are only going to
type in the charset that they have/know.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "William Tan" <[email-masked]>
Date: Mon, 26 Mar 2001 10:53:34 +0800

Adding a new TES will break things.
If Maynard means to 'convert' (sensitive word) the mail data (i.e. rfc 822)
then it would violate many rules.
However, I agree that ACE should be 'converted' for display, and strictly
for display only.
This is analogous from mail clients detecting mail / url addresses and
hyperlinking it.

wil.

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 26 Mar 2001 03:43:16 +0000

William Tan <[email-masked]> wrote:

> However, I agree that ACE should be 'converted' for display, and
> strictly for display only.  This is analogous from mail clients
> detecting mail / url addresses and hyperlinking it.

I disagree.  I don't think it is analogous.  Hyperlinking the text may
add underlines or color, but it does not change the text itself.

The recipient, looking at the message body, should ideally see the
same thing the sender saw when composing the message body.  If the
message body contains Chinese characters XXX.cn, then that's what
should be displayed.  But if the message body contains ASCII characters
bq--xxx.cn, then that's what should be displayed, because that's what
the sender saw and intended for the recipient to see.

AMC

---

From: "James Seng/Personal" <[email-masked]>
Date: Mon, 26 Mar 2001 12:39:05 +0800

If you remove yourself from the concept of "text" but think of it as a
"presentation" layer issues, then you will realise both of you are
speaking of the same thing. :-)

-James Seng

---

From: "William Tan" <[email-masked]>
Date: Mon, 26 Mar 2001 12:34:35 +0800

> the sender saw and intended for the recipient to see.

I agree that everything in the message body (after the headers) should not
be touched.
I envisage the following sending process:

1. User compose a message to an IDN email address.
2. MUA converts the IDN email address to ACE
3. MUA uses the ACE for RFC821 MAIL and RCPT commands
4. MUA uses the ACE in RFC822 From: and To: headers (as recommended in
idn-mua I-D)
5. MUA leaves the message body untouched.

wil.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sun, 25 Mar 2001 21:03:24 -0800

William Tan wrote:

> 4. MUA uses the ACE in RFC822 From: and To: headers (as recommended
> in idn-mua I-D)

What about Received, Message-ID, Path, and all of the other message
headers which store IDNs? HTTP proxies and a bunch of other apps are going
to have similar problems.

Some sort of policy probably needs to be defined somewhere.

Should we be answering these questions or should the question be raised in
a document and left for the respective WGs to answer?

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 26 Mar 2001 07:30:09 +0200

>a document and left for the respective WGs to answer?

That is the current plan I have as AD for the area with the most protocols.

IAB do though have a workshop on internationalization later in April, 
and I know this more meta/process issue will be discussed.

What I as editor of the IDNA document want to have out of this 
discussion though is example of wording which is needed to be added 
to explain that conversion should happen only for display -- which 
seems to be the view of this wg. This was also the outcome of a 
discussion between wg and bof chairs in the Applications Area.

I.e. if the conversion can happen closer to display or closer to 
resolver, it should happen closer to display. "It is a display issue 
only" was a statement I heard several times last week.

I also heard that it is extremely important that a user which (for 
example) receive a domainname which is in ACE, can cut&paste, reply 
(to an email) etc. And that these abilities are more important than 
correct display. This means for me that if an application detects the 
ace, and is "unsure" of how to display some of the characters -- it 
is better if the application does NOT convert from ace. This so it is 
sure that the domainname is not destroyed. Also, if the conversion is 
happening only for display, the underlying storage in memory can 
still have the non-converted ace all the time, so regardless of what 
is displayed, it is the ace which is copied and pasted etc.

I might be wrong in what I heard, but understand that just the 
existence of the discussion in the corridors makes some wording be 
needed in the IDNA document.

I am happy to receive proposed wordings which clearify what this wg want.

     Patrik

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2001 00:39:18 -0500

--On Monday, 26 March, 2001 12:34 +0800 William Tan
<[email-masked]> wrote:

> I agree that everything in the message body (after the
> headers) should not be touched.
> 4. MUA uses the ACE in RFC822 From: and To: headers (as
> recommended in idn-mua I-D)

Be careful here; this gets really hard (independent of the
"other header and imbedded information" problem that Eric points
out).  Suppose you send me a message, following the above rules.
So far, so good.  But I now want to send some third party who
uses English, a note that summarizes your note and suggests that
he contact you for further information (variations on this are
_very_ common).  So I copy and paste your address from the RFC
822 From: field, in ACE form, into the message body I'm about to
send, with 
   Content-type: text/plain; charset=us-ascii

My recipient now has your address in ACE form in a message body.
If she replies directly to you, and pastes that address back
into the headers of the outgoing message, all is still well, as
long as her MUA is smart enough to not try to apply your "step
1" conversion to something that is already in ACE form.   But
suppose that, instead, she translates my message into Arabic or
Chinese and then sends it, especially if she does so using a
non-Unicode charset (permissible, and, indeed, common in message
bodies) or, worse, has that translation performed by an
automatic translator that has not been "taught" to detect ACE
strings in running text (note that, in the general case, they
cannot be detected and discriminated from discussions of ACE
with perfect accuracy).  

Care to guess the odds on your address surviving?  

Also note that these sorts of scenarios implies that ACE strings
are going to be in the faces of users quite a lot, rather than,
as has been argued, only when the use is operating with an
old-style MUA.

    john

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2001 00:50:24 -0500

--On Monday, 26 March, 2001 07:30 +0200 Patrik Fältström
<[email-masked]> wrote:

>...
> I also heard that it is extremely important that a user which
> (for example) receive a domainname which is in ACE, can
> cut&paste, reply (to an email) etc. And that these abilities
> are more important than correct display. This means for me
> that if an application detects the ace, and is "unsure" of how
> to display some of the characters -- it is better if the
> application does NOT convert from ace. This so it is sure that
> the domainname is not destroyed. Also, if the conversion is
> happening only for display, the underlying storage in memory
> can still have the non-converted ace all the time, so
> regardless of what is displayed, it is the ace which is copied
> and pasted etc.
>...

As long as you assume that all devices connected to the Internet
will be able to render (and probably input/accept) ASCII...

     john > I might be wrong in what I heard, but understand that just the
> existence of the discussion in the corridors makes some
> wording be needed in the IDNA document.

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 26 Mar 2001 06:03:00 +0000

John C Klensin <[email-masked]> wrote:

> My recipient now has your address in ACE form in a message body...
> ...she translates my message into Arabic or Chinese and then sends
> it... ...translation performed by an automatic translator that has not
> been "taught" to detect ACE strings in running text...
>

How is this an ACE issue?  Wouldn't the problem be exactly the same for
a regular ASCII domain name?

AMC

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 26 Mar 2001 08:08:43 +0200

>will be able to render (and probably input/accept) ASCII...

Yes, for the first part of this paragraph. What I tried to write was 
that an application which is running in some operating system 
environment which today normally is a windowing system of some kind 
have to be extremely careful when throwing away information. The 
window system might not be able to handle ascii, and it might not be 
able to handle the characters which are encoded in ascii. All 
permutations can exists.

Because of this, keeping the internal storage in whatever encoding is 
used originally is "a good thing".

If this is ace, the only thing that is needed is that the internal 
storage in the application can handle ascii, and that that storage is 
not destroyed when converting to and from whatever display one use -- 
and that for example the clipboard also can handle ascii.

But, as I said, I am happy to receive whatever text is needed to 
clearify things.

Including large warning labels.

   paf

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2001 01:24:58 -0500

--On Monday, 26 March, 2001 06:03 +0000 "Adam M. Costello"
<[email-masked]> wrote:

>> My recipient now has your address in ACE form in a message
>> body... ...she translates my message into Arabic or Chinese
>> and then sends it... ...translation performed by an automatic
>> translator that has not been "taught" to detect ACE strings
>> in running text...
>> 
>>...
> How is this an ACE issue?  Wouldn't the problem be exactly the
> same for a regular ASCII domain name?

Just about, but not quite.  As long as all domain names (and
email address local parts) are [restricted-]ASCII, one does not
have the problem of figuring out whether a name one is looking
at is the intended name or an encoding of some other name.  And
that problem gets especially interesting when the encoding is of
a name in a script that matches the script (but not necessarily
the encoding) of a body part in which it might be embedded.

And there are law of least astonishment UI issues here.  If the
email address (local and domain parts) is, e.g., encoded in ACE
from Arabic, and a message body ends up being sent in Arabic
text, I would predict a good deal of user irritation if the
email address appeared in that running text in ASCII (with the
added irritations of embedding left-to-right text in a
right-to-left script stream) because it was pasted from a
"from:" field rather than being keyed in.

This also implies some other UI issues for which "display only"
(as suggested by Patrik's notes and others) is not sufficient.
If a message comes in for which the originating and target
addresses are encoded from the normal script of my operating
environment, I will probably expect them to be displayed into
that script.  But, if the character encodings of the operating
system are different from what shows up on the wire and/or if
the operating system's "clipboard" or equivalent are modularized
in certain ways, if I paste something out of the display that
has been rendered into native script, I'm probably going to get
the native script in my text, not the ACE form.  If that isn't
what is wanted,...

    john

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 26 Mar 2001 06:49:11 +0000

John C Klensin <[email-masked]> wrote:

> ...if I paste something out of the display that has been rendered into
> native script, I'm probably going to get the native script in my text,
> not the ACE form.

That is what I would expect to happen.  I would be very surprised if
I highlighted Arabic text and when I pasted it, it became ACE.  I
think users expect copying and pasting to be a simple, intuitive,
what-you-see-is-what-you-get operation.

> If the email address (local and domain parts) is, e.g., encoded in ACE
> from Arabic, and a message body ends up being sent in Arabic text,
> I would predict a good deal of user irritation if the email address
> appeared in that running text in ASCII (with the added irritations of
> embedding left-to-right text in a right-to-left script stream) because
> it was pasted from a "from:" field rather than being keyed in.

If the person doing the copy/paste was using an IDN-aware MUA with
support for Arabic, then the From: field was showing Arabic text,
and that is what got copied and pasted, not ACE, so there will be no
irritation.  With a non-IDN-aware MUA, or one that doesn't support
Arabic, the ACE will be displayed in the From: field, and so the ACE
will be copied and pasted, and the recipient will be irritated (but the
sender will be too, which may motivate an upgrade of the MUA).

AMC

---

From: "William Tan" <[email-masked]>
Date: Mon, 26 Mar 2001 15:06:25 +0800

> irritation.

If the recipient system is not IDN aware, it would break, defeating the
purpose of IDNA.  And even if it is IDN aware, it might not support the
character set being used to represent the IDN email address.

wil.

---

From: "Maynard Kang" <[email-masked]>
Date: Mon, 26 Mar 2001 08:32:48 +0100

> speaking of the same thing. :-)

Perhaps I should have explained more clearly in my presentation (no pun intended) why I believe this problem is not simply a presentation layer issue.

Prima facie it looks like a simple problem: Applications would just hand everything off to the presentation layer to handle what's being sent on the wire, and the presentation layer would handle whatever that's received on the wire and present it in a form that is acceptable to the application.

It would be just a simple presentation layer issue if IDNs in existing application protocols were used only for resolution purposes (e.g. ftp, ping). 

However, some IDNs may appear in these protocol exchanges as placeholders or arbitrary data, and this means that the application protocol itself needs to know how to deal with an IDN. Henceforth it becomes an application layer problem too.

I heard the comment during my presentation that there were several layer violations, and rightly so. It was not my main intention to present any proposed solution to this problem, but rather, to point out that if we as a working group fail to set the direction on this issue correctly, there WILL be more layer violations in the future during implementation and, as the growing threat of "testbed" deployment looms, this will have many disasterous consequences if we do not deal with this now.

maynard

---

From: "Maynard Kang" <[email-masked]>
Date: Mon, 26 Mar 2001 08:33:32 +0100

> suppose that, instead, she translates my message into Arabic or

I assume that when you say "translate to Arabic or Chinese" you mean it in a linguistic sense (as opposed to doing encoding conversion).

> Care to guess the odds on your address surviving?  

For all purposes, an ACE domain name present in a message should be treated as arbitrary data by any translation engine (whether human or machine), and hence MUST NOT translate that ACE string into Arabic and Chinese together with the rest of the message content.

Then again, having said that, I would love to see a translation engine handle an English-to-Arabic translation of "bq--badexample.com" (I think this is actually a valid race string).

But you are right in that there is no way to discern an ACE-string from other data with perfect accuracy. Perhaps we need an escaping mechanism for this.

maynard

---

From: "Maynard Kang" <[email-masked]>
Date: Mon, 26 Mar 2001 08:32:13 +0100

> the sender saw and intended for the recipient to see.

I think most will fully agree that bq--xxx.cn in bodies should be displayed as it is.
But how about bq-xxx.cn in headers, especially headers which contain an addr-spec, route or domain? That is the tricky bit.

maynard

---

From: "Edmon" <[email-masked]>
Date: Mon, 26 Mar 2001 11:55:09 -0500

> > 5. MUA leaves the message body untouched.

I dont agree with 4.
I believe that it is sufficient to use 2&3 and leave the headers to be in
existing MIME format because headers are for display only.  If a mail is
"replied" it will go through 1-3 again so there is no need to "store" the
header fields in ACE format at all!

Edmon

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 26 Mar 2001 19:00:45 +0200

>character set being used to represent the IDN email address.

Not if the clipboard contain the ACE encoding regardless of whether 
the "sending" application can handle ACE or not.

I.e. the application which put things in the clipboard buffer should 
do it in the ace encoding -- and this is why ace->local script is 
only a display issue.

    paf

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 26 Mar 2001 10:54:43 -0800

> >>  and that is what got copied and pasted, not ACE, so there will be
> >>  no irritation.
> >
> >If the recipient system is not IDN aware, it would break, defeating
> >the purpose of IDNA.  And even if it is IDN aware, it might not
> >support the character set being used to represent the IDN email
> >address.
> 

The clipboard is not likely to be stuffed with ACE strings. The most
likely event is that the charset-specific encoding will be converted to a
system-specific form.

EG, a mail client that supports ISO-2022-JP text (and therefore which also
supports URLs and email addresses written in ISO-2022-JP as the users are
most likely to do) will just copy the text string to the clipboard as UTF8
or UCS-2 or some other system-specific encoding. It is highly unlikely
that an application is going to convert the data to ACE based on the
possibility that the selected text may contain an ACE encoded string.

> do it in the ace encoding -- and this is why ace->local script is

I don't think that's a valid assumption, for several reasons.

What if the selected data contains blocks of text with an IDNs spread
across the text, is the application supposed to convert the IDNs to ACE
while leaving everything else alone?

Is an application supposed to convert IDNs in attachments? Maybe not
convert IDNs in Excel spreadsheet attachments but what about vCard?

It is well beyond any protocols scope to demand this conversion.

Applications will convert to system-specific formats and then convert to
ACE again when the IDNs need to be resolved

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 26 Mar 2001 11:37:27 -0800

> >Should we be answering these questions or should the question be
> >raised in a document and left for the respective WGs to answer?
> 

I think we need to split this issue into its conceptual entities.

First issue is user-generated IDNs: Users are going to *TYPE* email
address and URLs in their preferred charset. Applications and systems
cannot make intelligent conversions to ACE (reading mail in pine via
xterm, we can't expect that xterm is going to automatically convert the
string "www.XXXXX.jp" into ACE [especially if there is no "http://"
prefix]). So on that side we need to reinforce that names must be
converted to UTF-8 before they are fed to nameprep but how they are
handled locally is up to the local system and beyond the scope of any IETF
protocol.

Second issue is machine-generated IDNs, such as those that appear in
message headers, PTR lookups, HELO banners, PATH names, HTTP redirs, and
so forth.

The "default" action for PTR lookups is going to be raw ACE strings.

HELO, PATH, HTTP sequences and other "pushed" names can be ACE, or they
can use an RFC-2047 style encoding syntax. There are three issues:

 1) If they use a 2047 encoding, that opens a lot of questions.
    For example, we have no way of knowing what charset the
    end-viewer will be using, so it is not practical to force one.
    Although we could mandate that an ACE-specific 2047 encoding
    "must use UTF-8", if the user who is viewing the data is using
    some other local encoding, it may not be possible for them to
    mix UTF-8 with their local encoding. Network analyzers don't
    generally support multiple charsets, for example, and throwing
    UTF-8 into that may cause more problems than it resolves.

 2) Furthermore, 2047 encoding ("?=") of a domain name will break
    many applications and protocols which are written to *ONLY*
    use RFC-952 hostnames.

 3) The whole motivation for ACE is that it is compatible with
    legacy ASCII encodings so that it won't break stuff. Trying to
    change this into some kind of forward-compatibility will break
    stuff, which goes against the very goals of ACE. 

For those reasons I think we HAVE to mandate that machine-generated IDNs
must only provide ACE-encoded names whenever thir hostname is written.

This may be a problem for some applications that get their hostname from a
local system call, so the system calls will likely have to provide an ACE
value and a "rich" value.

If that occurs then it may be possible to amend the above rule with a
modifier of "except where the protocol explicitly specifies that the
identifier is eight-bit aware".

That is really ugly stuff, I agree. But given the constraints that we are
under by trying to maintain strict backwards compatibility (vis-a-vis ACE)
I don't see any other approach.

---

From: [email-masked] (Bob Jung)
Date: Mon, 26 Mar 2001 12:06:12 -0800

I should have been more clear.  I didn't mean to imply I was recommending
using x-ACE content-encoding.

I just meant to respond to the original posting that content-type (e.g.,
plain/text) does not imply an encoding and it would be "legal" to send ACE
encoded plain text messages if the content-encoding header was used.

-bob

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 26 Mar 2001 20:13:30 +0000

William Tan <[email-masked]> wrote:

> the purpose of IDNA.

If your system is IDN-unaware, then a raw (non-ACE) IDN is useless
to you, regardless of whether you see it on a business card, on a
billboard, or in an email message body.  I think that's acceptable.  At
least you can always reply to any email you receive.

> And even if it is IDN aware, it might not support the character set
> being used to represent the IDN email address.

If you receive a message encoded using a character set that your system
does not support, then the entire message body is useless, including any
email addresses contained in it.  I think that's acceptable.

Maynard Kang <[email-masked]> wrote:

> I think most will fully agree that bq--xxx.cn in bodies should be
> displayed as it is.  But how about bq-xxx.cn in headers, especially
> headers which contain an addr-spec, route or domain? That is the
> tricky bit.

I think IDN-aware MUAs should by default decode the ACE in an addr-spec,
route, or domain that appears in a structured header field.  But it
should be possible for the user to turn off this decoding, in case the
ACE is needed for some reason.

Edmon <[email-masked]> wrote:

> I dont agree with 4.  I believe that it is sufficient to use 2&3 and
> leave the headers to be in existing MIME format because headers are
> for display only.

RFC 822 does not allow non-ASCII characters in domain names in
structured header fields (like From: and To:), and MIME does not change
that rule, so the only way to use IDNs there (without changing RFC 822)
is via ACE.

This is the only way that non-IDN-aware MUAs will be able to reply to
messages sent from IDN addresses.

Patrik Fältström <[email-masked]> wrote:

> the application which put things in the clipboard buffer should do it
> in the ace encoding

I disagree for two reasons.  First, if I highlight Japanese text and it
gets transformed into ACE when I pasted it, I'm going to be surprised
and annoyed.  Second, what you propose is simply impossible in many
cases.  For example, my MUA is Mutt running inside an xterm.  Mutt
has no control over how copying and pasting happens, because it's all
handled by xterm.

AMC

---

From: Keith Moore <[email-masked]>
Date: Mon, 26 Mar 2001 16:05:06 -0500

> can use an RFC-2047 style encoding syntax.

as far as I'm concerned, RFC 2047 encoding is a non-starter for domain names
in protocol elements.  2047 was specifically designed to encode text whose
only purpose was to be presented to a user; and it therefore completely 
skirts all issues related to normalization, canonicalization, etc.

(note that even if you use 2047 to encode nameprepped Unicode, there is 
more than one way to encode such names... names encoded in such a way
cannot be compared for equivalence as if they were ascii)

any domain name that appears in a protocol element that is intended to be 
*interpreted* or *compared* (as opposed to merely *displayed*) needs to 
be ACE-encoded.

Keith

---

From: "Edmon" <[email-masked]>
Date: Mon, 26 Mar 2001 16:48:07 -0500

> messages sent from IDN addresses.

I was using the "MIME" word loosely and I appologise for that.  What I meant
was using the RFC 2047 (MIME Part 3: Message Header Extensions for Non-ASCII
Text) format.  In other words, extending MIME for header fields.

Edmon

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Mon, 26 Mar 2001 17:49:34 -0500

[I deleted paf's message so I'm replying to it here]

| Patrik Fältström <[email-masked]> wrote:
|
| > the application which put things in the clipboard buffer should do it
| > in the ace encoding
|
| I disagree for two reasons.  First, if I highlight Japanese text and it
| gets transformed into ACE when I pasted it, I'm going to be surprised
| and annoyed.  Second, what you propose is simply impossible in many
| cases.  For example, my MUA is Mutt running inside an xterm.  Mutt
| has no control over how copying and pasting happens, because it's all
| handled by xterm.

  I disagree with Patrik as well.  Doesn't IDNA basically say that
applications should do this sort of thing:

  if ( isRFC1035(domain_name)) {
	doDNSOperation(domain_name);
  } else {
	ace_domain_name = IDNify(domain_name);
	doDNSOperation(ace_domain_name);
  }

  sub IDNify (domain_name) {
	if (! isUnicode(domain_name) {
		domain_name = convertToUnicode(domain_name);
	}
	domain_name = doNameprep(domain_name);
	domain_name = convertToACE(domain_name);
  }

  Meaning that only when you may have to call the 'doDNSOperation'
mythical function do you need to worry about the IDN issues.

  In addition, user applications need to worry about IDN display
issues, but again only in the context where you may do an
operation on the name as a protocol element.

  Perhaps that's a good working definition:  IDNs as protocol
elements MUST be in nameprepped ACE, otherwise they're just
data.  In addition, user agents SHOULD display IDNs in protocol
element contexts in native characters.

---

From: "Edmon" <[email-masked]>
Date: Mon, 26 Mar 2001 18:59:52 -0500

> skirts all issues related to normalization, canonicalization, etc.

While I agree to a certain degree (within the protocol and
transportation/routing), I would like to say that in the context of this
discussion where we are disucssing the "message headers" and "content body"
of an email, the information is in fact for display purpose.  And as you
have precisely pointed out, cases should be preserved.  That is if I typed
in "[email-masked]"
for example, it should be displayed as is after it is sent to the recipient.
An ace representation would not preserve the case.

Edmon

---

From: Keith Moore <[email-masked]>
Date: Mon, 26 Mar 2001 19:17:32 -0500

> as far as I'm concerned, RFC 2047 encoding is a non-starter for domain
> > names in protocol elements.  2047 was specifically designed to encode 
> > text whose only purpose was to be presented to a user; and it therefore 
> > completely skirts all issues related to normalization, canonicalization, 
> > etc.
> 
> of an email, the information is in fact for display purpose.

no.  not only is this not "in fact", it's simply incorrect.
the addresses in the message header are intended both for display
and for use in forming replies.

also, in practice, RFC 2047 encoded-words are routinely decoded, 
translated, re-encoded, and/or corrupted by mail transports and
delivery systems.  

Keith

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 27 Mar 2001 07:18:20 +0200

>mythical function do you need to worry about the IDN issues.

No, you have to worry about IDN also when using the domainname as 
protocol parameter in the application protocol aswell, which include 
for example when using SMTP. One of the main things in IDNA is that 
the ace is good because the domainname whenever being a domainname 
(i.e. not only when you are to use DNS) is encoded in ascii.

This is one of the differences between IDNA and what Walid have in 
it's patent by the way -- at least when I read the patent -- because 
the patent talk about intercepting just the doDNSOperation and do the 
encoding only in that case.

I claim that is not sufficient, and that is why I wrote the IDNA spec 
the way I did.

The real problem is when a domainname is not really a domainname in a 
protocol sense -- which includes when it is inside the mail body.

And, I really would like to have explicit suggestions for text which 
I can add to the document which clearifies how to handle this case.

   paf

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Tue, 27 Mar 2001 11:19:40 -0500

| -----Original Message-----
| From: Patrik Faltstrom [mailto:[email-masked]]
| Sent: Tuesday, March 27, 2001 12:18 AM
| To: Brian W. Spolarich; Adam M. Costello; idn working group
| Subject: RE: [idn] IDNs in email message bodies
| 
| 
| At 17.49 -0500 01-03-26, Brian W. Spolarich wrote:
| >   Meaning that only when you may have to call the 'doDNSOperation'
| >mythical function do you need to worry about the IDN issues.
| 
| No, you have to worry about IDN also when using the domainname as 
| protocol parameter in the application protocol aswell, which include 
| for example when using SMTP. One of the main things in IDNA is that 
| the ace is good because the domainname whenever being a domainname 
| (i.e. not only when you are to use DNS) is encoded in ascii.

  I think my response was unclear.  I stated later that the 
important distinction for IDNs in that those in protocol
elements should be ACE.

| This is one of the differences between IDNA and what Walid have in 
| it's patent by the way -- at least when I read the patent -- because 
| the patent talk about intercepting just the doDNSOperation and do the 
| encoding only in that case.

  What WALID, Inc. is doing today (regardless of the IPR) isn't 
intended as the ultimate solution, but rather something that will
transition well as that solution emerges without breaking things
along the way.  No one would be foolish enough (I hope) to think
they have clarvoyance enough to tackle this problem unilaterally.

  My gut tells me that you're going to need a doDNSOperation-style
interceptor in order to do an effective transition to IDNA. 

| I claim that is not sufficient, and that is why I wrote the IDNA spec 
| the way I did.
| 
| The real problem is when a domainname is not really a domainname in a 
| protocol sense -- which includes when it is inside the mail body.
| 
| And, I really would like to have explicit suggestions for text which 
| I can add to the document which clearifies how to handle this case.

  I'll shut up then and contribute a suggestion in that vein. :-)

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 27 Mar 2001 19:49:44 +0200

>   I'll shut up then and contribute a suggestion in that vein. :-)

It certainly makes my life easier. I just have to add the text, and 
not think so much... ;-)

   paf