Subject: RE: canonical reference for xtext
From: "Vaudreuil, Greg M (Greg)" <[email-masked]>
To: "Eric A. Hall" <[email-masked]>, [email-masked]
Date: Tue, 30 Apr 2002 15:44:23 -0600
List-Id: <ietf-smtp.imc.org>

The replacements for rfc 1894 <draft-vaudreuil-1894bis-02.txt> no longer
defines xtext.  The definition is left-over from earlier drafts of rfc1894.
RFC 1894 does not itself use xtext, so the definition was rather extranious.

This would leave 1891bis as the single, formal, canonical definition of
xtext.

Greg V.

-----Original Message-----
From: Eric A. Hall [mailto:[email-masked]]
Sent: Tuesday, April 30, 2002 3:52 PM
To: [email-masked]
Subject: canonical reference for xtext

The xtext data-type is defined in multiple mail-related RFCs. Alas, these
definitions are different:

RFC 1891, section 5. Additional parameters for RCPT and MAIL commands

 | "xtext" is formally defined as follows:
 |
 |     xtext = *( xchar / hexchar )
 |
 |     xchar = any ASCII CHAR between "!" (33) and "~" (126) inclusive,
 |          except for "+" and "=".

RFC 1894, section 2.1.1. General conventions for DSN fields

 | "xtext" is formally defined as follows: 
 |
 |    xtext = *( xchar / hexchar / linear-white-space / comment )
 |
 |    xchar = any ASCII CHAR between "!" (33) and "~" (126) inclusive,
 |           except for "+", "\" and "(".

RFC 2554 cites RFC 1891 for its use of "xtext"

Not sure if there's a question here. I would gather that there isn't a
canonical definition for xtext (correct me if I'm wrong, please), and
people will use either the 1891 or 1894 definitions as needed (or make up
another one, as needed). Perhaps any update to 2821 should include one as
a suitable reference.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 30 Apr 2002 15:52:16 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: canonical reference for xtext
Sender: [email-masked]

The xtext data-type is defined in multiple mail-related RFCs. Alas, these
definitions are different:

RFC 1891, section 5. Additional parameters for RCPT and MAIL commands

 | "xtext" is formally defined as follows:
 |
 |     xtext = *( xchar / hexchar )
 |
 |     xchar = any ASCII CHAR between "!" (33) and "~" (126) inclusive,
 |          except for "+" and "=".

RFC 1894, section 2.1.1. General conventions for DSN fields

 | "xtext" is formally defined as follows: 
 |
 |    xtext = *( xchar / hexchar / linear-white-space / comment )
 |
 |    xchar = any ASCII CHAR between "!" (33) and "~" (126) inclusive,
 |           except for "+", "\" and "(".

RFC 2554 cites RFC 1891 for its use of "xtext"

Not sure if there's a question here. I would gather that there isn't a
canonical definition for xtext (correct me if I'm wrong, please), and
people will use either the 1891 or 1894 definitions as needed (or make up
another one, as needed). Perhaps any update to 2821 should include one as
a suitable reference.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked]
Subject: Re: 2821 source-routes, strict or loose? 
From: [email-masked]
Date: Wed, 24 Apr 2002 12:00:54 -0400
Sender: [email-masked]

--==_Exmh_249113208P
Content-Type: text/plain; charset=us-ascii

On Wed, 24 Apr 2002 10:41:36 CDT, "Eric A. Hall" said:
> 
> [email-masked] wrote:
> 
> > hand-specify the undocumented _USE_DEPRECATED_ROUTE_ADDR_ option when
> > building the config file)....
> 
> Do you know what happens after that? Is it host-specific (strict) or does
> the next-hop go through the routing agent (loose)?

Well, *if* you supply the undocumented option, it rewrites as follows:

@routehost:user@node  ->  user%node@routehost 

I'm a bit too fuzzy-brained at the moment to be able to understand the .cf
majic enough to guess whether @host1,@host2:user@node ends up as
user%node%host1@host2 or user%node%host2@host1.  In any case, the source
route ends up as the RHS of a %-hack... 

Now this will be strictly routed to whatever ends up on the RHS of
the '@' *in the absence of mailertables* and other re-writing that happens
after that.  Once it's beaten into a %-hack format, it's still subject to
other, site-defined, rerouting.

/Valdis

--==_Exmh_249113208P
Content-Type: application/pgp-signature

--==_Exmh_249113208P--

Date: Wed, 24 Apr 2002 10:41:36 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: 2821 source-routes, strict or loose?
Sender: [email-masked]

[email-masked] wrote:

> hand-specify the undocumented _USE_DEPRECATED_ROUTE_ADDR_ option when
> building the config file)....

Do you know what happens after that? Is it host-specific (strict) or does
the next-hop go through the routing agent (loose)?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked]
Subject: Re: 2821 source-routes, strict or loose? 
From: [email-masked]
Date: Wed, 24 Apr 2002 11:34:45 -0400
Sender: [email-masked]

--==_Exmh_77252344P
Content-Type: text/plain; charset=us-ascii

On Tue, 23 Apr 2002 14:05:29 CDT, "Eric A. Hall" <[email-masked]>  said:

> A better question here would be "what is the implementation norm?" Do most
> implementations treat source routes as strict, rejecting mail if a direct
> connection cannot be established, or do they process the next-hop of the
> source-route address through the routing table?

Sendmail 8.10.0 (released 03/01/2000) and later:
       CONFIG: Route addresses are stripped, i.e., <@a,@b,@c:user@d> is
                converted to <user@d>  

So "drop them puppies on the floor" seems to be the behavior there (unless
you hand-specify the undocumented _USE_DEPRECATED_ROUTE_ADDR_ option when
building the config file)....
-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

--==_Exmh_77252344P
Content-Type: application/pgp-signature

--==_Exmh_77252344P--

Date: Tue, 23 Apr 2002 17:41:38 -0500
From: "Eric A. Hall" <[email-masked]>
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: Re: 2821 source-routes, strict or loose?
Sender: [email-masked]

John C Klensin wrote:

> If you can read 2821 as ambiguous on this point, please drop me
> a note that identifies sections and I'll try to get them fixed
> when I get back to the revision.

It's not ambiguous but it is confusing managing the threads. They are
discussed with varying focus and emphasis in Section 3.7 Relaying, Section
4.1.1.3 Recipient (RCPT) and Appendix C, and mentioned elsewhere. If you
were so inclined, wrapping this into a single section would probably
improve readability.

The question was philosophical.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: [email-masked]
Subject: Re: 2821 source-routes, strict or loose? 
Date: Tue, 23 Apr 2002 18:31:09 -0400
Sender: [email-masked]

> I'm trying to figure out whether or not source-route addressing is strict
> or loose.

when RFC 821 was written it was strict.  since RFC 1123, it's been loose.

Date: Tue, 23 Apr 2002 18:29:11 -0400
From: John C Klensin <[email-masked]>
To: "Eric A. Hall" <[email-masked]>, [email-masked]
Subject: Re: 2821 source-routes, strict or loose?
Sender: [email-masked]

Eric,

The source routing rules were modified in RFC 1123 to,
essentially, "SHOULD NOT be produced, MAY BE ignored if
received".   The DRUMS consensus, reflected in 2821, was
intended to modify that to "SHOULD NOT be produced, SHOULD BE
ignored" except in exceptional cases.  Receiving SMTPs are
expected to recognize and accept the syntax for the obvious
backward-compatibility reasons, but that is about all they are
expected to do with it.

Neither 1123 nor 2821 say how the intermediate names in a source
route list should be resolved because, if I recall, the focus
was on discouraging the use of the things, not on fine-tuning
them.

If you can read 2821 as ambiguous on this point, please drop me
a note that identifies sections and I'll try to get them fixed
when I get back to the revision.

     john

--On Tuesday, 23 April, 2002 14:05 -0500 "Eric A. Hall"
<[email-masked]> wrote:

> 
> 
> I'm trying to figure out whether or not source-route
> addressing is strict or loose. This is mostly a philosophical
> question, since I think I already know the answer.
> 
> In the context of 821 predating DNS, the use of source-routes
> should be interpreted as strict, since a list of "hosts" is
> all that was available. However, consider that a routing agent
> could have a local routing table that redirected traffic for a
> specific mail-domain to another relay (without rewriting the
> envelope). Although 821 doesn't define static routing and
> precedence issues, the use of static routes wasn't exactly an
> unknown concept. Taken together, source-route addresses that
> were routed with static maps would be loose, even though they
> were defined within a strict context.
> 
> With 2821, this is even more jello-like, since each hop in the
> path may be resolved with MX resource recrods (defined, but
> not for use with source routes), or with any static maps
> (still undefined). The terminology in Appendix C makes it
> appear that the source-route construct is still intended to be
> used with "hosts" but since the hosts are FQDNs, they should
> be treated as mail domains, with MX being applied.
> 
> This means that with EG relay1:user@hosta
> 
>      originator --> submission
> 
>      submission router does MX/static lookups
>        [null] relay1=relay1
> 
>      submission --> relay1
> 
>      relay1 router does MX/static lookups
>        MX hosta=relay2, pref 10
> 
>      relay1 --> relay2
> 
>      relay2 router does MX/static lookups
>        STATIC hosta=hosta
> 
>      relay2 --> hosta
> 
> So it is loose in practice, even though it is implied as
> strict.
> 
> Is that off-base? I can't find anything that would either
> endorse or reject this output.
> 
> A better question here would be "what is the implementation
> norm?" Do most implementations treat source routes as strict,
> rejecting mail if a direct connection cannot be established,
> or do they process the next-hop of the source-route address
> through the routing table?
> 
> -- 
> Eric A. Hall
> http://www.ehsco.com/ Internet Core Protocols
> http://www.oreilly.com/catalog/coreprot/
> 
>  

Date: Tue, 23 Apr 2002 14:05:29 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: 2821 source-routes, strict or loose?
Sender: [email-masked]

I'm trying to figure out whether or not source-route addressing is strict
or loose. This is mostly a philosophical question, since I think I already
know the answer.

In the context of 821 predating DNS, the use of source-routes should be
interpreted as strict, since a list of "hosts" is all that was available.
However, consider that a routing agent could have a local routing table
that redirected traffic for a specific mail-domain to another relay
(without rewriting the envelope). Although 821 doesn't define static
routing and precedence issues, the use of static routes wasn't exactly an
unknown concept. Taken together, source-route addresses that were routed
with static maps would be loose, even though they were defined within a
strict context.

With 2821, this is even more jello-like, since each hop in the path may be
resolved with MX resource recrods (defined, but not for use with source
routes), or with any static maps (still undefined). The terminology in
Appendix C makes it appear that the source-route construct is still
intended to be used with "hosts" but since the hosts are FQDNs, they
should be treated as mail domains, with MX being applied.

This means that with EG relay1:user@hosta

     originator --> submission

     submission router does MX/static lookups
       [null] relay1=relay1

     submission --> relay1

     relay1 router does MX/static lookups
       MX hosta=relay2, pref 10

     relay1 --> relay2

     relay2 router does MX/static lookups
       STATIC hosta=hosta

     relay2 --> hosta

So it is loose in practice, even though it is implied as strict.

Is that off-base? I can't find anything that would either endorse or
reject this output.

A better question here would be "what is the implementation norm?" Do most
implementations treat source routes as strict, rejecting mail if a direct
connection cannot be established, or do they process the next-hop of the
source-route address through the routing table?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Mon, 15 Apr 2002 11:54:08 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: typo in 2821 -- failed end-of-data
Sender: [email-masked]

Section 4.2.5:

 | When an SMTP server returns a permanent error status (5yz) code after
 | the DATA command is completed with <CRLF>.<CRLF>, it MUST NOT make
 | any subsequent attempt to deliver that message.  The SMTP client
 | retains responsibility for delivery of that message and may either
 | return it to the user or requeue it for a subsequent attempt (see
 | section 4.5.4.1).

...

 | When an SMTP server returns a permanent error status (5yz) code after
 | the DATA command is completely with <CRLF>.<CRLF>, it MUST NOT make
 | any subsequent attempt to deliver the message.  As with temporary
 | error status codes, the SMTP client retains responsibility for the
 | message, but SHOULD not again attempt delivery to the same server
 | without user review and intervention of the message.

Both of those paragraphs refer to permanent failures (5yz), although the
first paragraph is actually rerferring to temporary failures (4yz). Some
clarification is also in order for the first sentence in the first
paragraph, as it appears to be a directoy copy-n-paste from the first
sentence in the second paraghraph.

Also the text in the second paragraph should be "SHOULD NOT" rather than
"SHOULD not". 

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Wed, 10 Apr 2002 00:35:06 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: SRV for 2476
Sender: [email-masked]

It seems that the Message Submission protocol defined in RFC 2476 is a
perfect candidate for a DNS SRV resource record, since it would allow the
clients to automatically fallback to a backup MSA if the primary MSA
wasn't available for some reason.

Should this be written up and added?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Thu, 04 Apr 2002 19:49:21 +0200
From: =?ISO-8859-1?Q?Patrik_F=E4ltstr=F6m?= <[email-masked]>
To: =?ISO-8859-1?Q?Patrik_F=E4ltstr=F6m?= <[email-masked]>, "Eric A. Hall" <[email-masked]>
cc: [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

This mail of mine doesn't make sense. I should not write email while being
on conference calls.

Sorry.

  paf

--On 2002-04-04 19.42 +0200 Patrik Fältström <[email-masked]> wrote:

> 
> --On 2002-04-04 11.21 -0600 "Eric A. Hall" <[email-masked]> wrote:
> 
>> To bring this back on-topic, I'm really beginning to believe that the
>> ultimate final solution to the SMTP problem will be a new message format,
>> where the headers are defined as UTF-8 from the start, in conformance
>> with the IAB guidelines. Rather than having headers co-exist, downgrade
>> the new message format whenever legacy systems and mailboxes are
>> encountered.
> 
> I might agree with this, but, my point with path [B] is that if/when this
> is happening it can happen independently of what encoding is used in the
> DNS (and other) protocols.
> 
> According to path [A], all protocols need to change at the same time.
> 
>   paf

Date: Thu, 04 Apr 2002 19:42:31 +0200
From: =?ISO-8859-1?Q?Patrik_F=E4ltstr=F6m?= <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

--On 2002-04-04 11.21 -0600 "Eric A. Hall" <[email-masked]> wrote:

> To bring this back on-topic, I'm really beginning to believe that the
> ultimate final solution to the SMTP problem will be a new message format,
> where the headers are defined as UTF-8 from the start, in conformance with
> the IAB guidelines. Rather than having headers co-exist, downgrade the new
> message format whenever legacy systems and mailboxes are encountered.

I might agree with this, but, my point with path [B] is that if/when this
is happening it can happen independently of what encoding is used in the
DNS (and other) protocols.

According to path [A], all protocols need to change at the same time.

  paf

Date: Thu, 04 Apr 2002 11:21:09 -0600
From: "Eric A. Hall" <[email-masked]>
To: Patrik =?iso-8859-1?Q?F=E4ltstr=F6m?= <[email-masked]>
CC: [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

Patrik Fältström wrote:

> > Why would you do that?
> 
> To get as many characters as possible into the field.

> A different solution is path [B] and if that is chosen, UTF8 is
> definitly not the most efficient encoding of Unicode in a binary
> field which have a length specifier, like a label in the DNS protocol.

It seems that you are mixing two things here. On the one hand we are
talking about a binary representation which needs a tagging. But if you
are removing the ACE prefix then you need some other tag, which could
easily be an EDNS label identifier. At that point, the length tag that
exists in the legacy STD13 label is not particularly relevant. To wit, one
of the changes I have to make to DM-IDNS is to allow UTF-8 equivalents to
have longer labels, so that we can get around ACE length issues that
result in longer UTF-8 names.

Secondarily, there has been much discussion that UTF-8 and IDNA both have
cases where one is shorter than the other. I don't think either of them
have overwhelming wins as far as length is concerned. A binary version of
IDNA would have the one-win scenario of relatively synchronized lengths,
but for some situations it would be longer than UTF-8 which invalidates
the argument, unless you can show that the binary IDNA is consistently and
provably shorter for the overwhelming majority. Even that may not be
enough, however.

As for normalization and comparison problems, all encodings suffer from
that equally (assuming a one-pass conversion, which multiple encodings
explicitly do not allow for, by definition), so I certainly don't see any
wins for using a different encoding there.

For existing ASCII-bound protocols and data-types, certainly the use of
ACE makes sense. But for the applications and protocols that are using
UTF-8 data in conformance with IAB guidelines, letting them extract and
resolve data in that form seems to make the most sense. The only way that
I could see another encoding as even being a valid consideration would be
for the IAB guidelines to change.

With all of that in mind, I can't see how introducing yet another encoding
helps anything, other than something that isn't actually a problem in the
first place.

To bring this back on-topic, I'm really beginning to believe that the
ultimate final solution to the SMTP problem will be a new message format,
where the headers are defined as UTF-8 from the start, in conformance with
the IAB guidelines. Rather than having headers co-exist, downgrade the new
message format whenever legacy systems and mailboxes are encountered.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Thu, 04 Apr 2002 18:56:07 +0200
From: =?ISO-8859-1?Q?Patrik_F=E4ltstr=F6m?= <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

--On 2002-04-04 10.31 -0600 "Eric A. Hall" <[email-masked]> wrote:

>> For binary fields (ASN.1 and DNS labels) we can use some binary form
>> (like Punycode without the last "base-32").
> 
> Why would you do that?

To get as many characters as possible into the field.

As I said, to use the most efficient encoding possible.

I see two different paths:

 A Use one encoding for a specific "thing", and the encoding
   have to be what is needed for the protocol which uses
   this "thing" and have the least features.
 B Use different encodings for the same "thing" in different
   protocols, and possibly use the same encoding within the
   same protocol for all "things".

IDNA is [A] where the encoding of Unicode is the minimal possible (only LDH
characters) but the same for domain names in all protocols. This minimizes
the leakage (note, I am not saying it goes away).

A different solution is path [B] and if that is chosen, UTF8 is definitly
not the most efficient encoding of Unicode in a binary field which have a
length specifier, like a label in the DNS protocol.

   paf

Date: Thu, 04 Apr 2002 10:31:24 -0600
From: "Eric A. Hall" <[email-masked]>
To: Patrik =?iso-8859-1?Q?F=E4ltstr=F6m?= <[email-masked]>
CC: [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

Patrik Fältström wrote:

> The best solution for this whole mess would be to for every protocol,
> every protocol parameter find the most efficient encoding of Unicode
> for that element.

Absolutely 100% agree.

> For binary fields (ASN.1 and DNS labels) we can use some binary form
> (like Punycode without the last "base-32").

Why would you do that?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Thu, 04 Apr 2002 18:23:47 +0200
From: =?ISO-8859-1?Q?Patrik_F=E4ltstr=F6m?= <[email-masked]>
To: "David Leung (Neteka Inc.)" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Sender: [email-masked]

--On 2002-04-04 10.21 -0500 "David Leung (Neteka Inc.)" <[email-masked]>
wrote:

> Encoding is never the problem, but identifying the encoding is the
> problem, so just like the encoding tagging in the other protocols, add
> encoding tagging to DNS, or any required protocols that does not have it
> nowadays.

First of all, sorry for not understanding your notation.

About the identification, that's why we have a prefix for the labels which
have ACE coded content. I.e. works with protocols (like DNS) which doesn't
have explicit tagging of charset.

So, what is it we do not agree on? :-)

   paf

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
To: =?iso-8859-1?Q?Patrik_F=E4ltstr=F6m?= <[email-masked]>
Cc: "Keith Moore" <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Thu, 4 Apr 2002 10:21:28 -0500
Sender: [email-masked]

> >> If we are saying that the ACE <=> IDN(UTF8) so why is the two headers
> >> not in sync?
> >
> > because there are things which will know about the old headers and
change
> > those, without changing the new utf8 headers.
>
> By the way, ACE != IDN(UTF8).
>
> UTF-8 and ACE are two different ways of representing a Unicode String.

First of all, look at the above CAREFULLY ACE <=> IDN(UTF8) does not implies
ACE = IDN(UTF8) or ACE != IDN(UTF8), what I mean is they are interchangable,
and can be treat the same but not equal : )

> The problem is not in the encoding (and with that I mean we can aswell be
> conservative and use something (ACE) which works all over the place).
> People seem to think that "if we use UTF-8 encoding, things are fine,
> people will get nice display, we will not have copy-paste-leakage" etc
etc.
>
> That is wrong.

I agree, the problem is not in the encoding, but if we treat ACE as an
encoding, so we should put it in other work group I guess, maybe we should
start a Encoding Unicode in Protocols WG : )
Display and "copy-paste" depends a lot on the applications if they support
it or not, so neither the use of UTF8 or ACE will solve this. But if
changing the protocol will make the DNS to work with UTF8 rather than
limiting ourselves to just LDH is a good thing, except we need to think of
ways how to make the transition smoother from ASCII world to the UTF8, I
think a lot of i18n applications face the same problem too...

> The problem is the use of Unicode (or having multiple charsets). Try
> matching two Unicode strings, and see how easy that is. If you manage to
> get your brain around that problem, think about sorting and things like
> paged-result-sets in LDAP. And if that is not enough, think about what
> "network byte order" means in a situation where the local part of an email
> address contain mixed reight-to-left and left-to-right text.
>
> If you have a simple solution for that aswell which makes people happy,
let
> me know.
>
> So, I think this discussion should be about first of all the use of
Unicode
> in an email address.
>
> Secondly, we can talk about the encoding. But, the encoding is not the
> problem.

Encoding is never the problem, but identifying the encoding is the problem,
so just like the encoding tagging in the other protocols, add encoding
tagging to DNS, or any required protocols that does not have it nowadays.

Date: Thu, 04 Apr 2002 09:05:49 +0200
From: =?ISO-8859-1?Q?Patrik_F=E4ltstr=F6m?= <[email-masked]>
To: Keith Moore <[email-masked]>, "David Leung (Neteka Inc.)" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Sender: [email-masked]

--On 2002-04-03 13.59 -0500 Keith Moore <[email-masked]> wrote:

>> If we are saying that the ACE <=> IDN(UTF8) so why is the two headers
>> not in sync?
> 
> because there are things which will know about the old headers and change
> those, without changing the new utf8 headers.

By the way, ACE != IDN(UTF8).

UTF-8 and ACE are two different ways of representing a Unicode String.

The best solution for this whole mess would be to for every protocol, every
protocol parameter find the most efficient encoding of Unicode for that
element.

That imply that for ACII protocol parameters, we use ACE.

For 8bit-text-fields, UTF-8 is what can be used.

For binary fields (ASN.1 and DNS labels) we can use some binary form (like
Punycode without the last "base-32").

That people discuss generic use of UTF-8 all over the place is better than
ACE all over the place, or some new binary form all over the place amazes
me.

The problem is not in the encoding (and with that I mean we can aswell be
conservative and use something (ACE) which works all over the place).
People seem to think that "if we use UTF-8 encoding, things are fine,
people will get nice display, we will not have copy-paste-leakage" etc etc.

That is wrong.

The problem is the use of Unicode (or having multiple charsets). Try
matching two Unicode strings, and see how easy that is. If you manage to
get your brain around that problem, think about sorting and things like
paged-result-sets in LDAP. And if that is not enough, think about what
"network byte order" means in a situation where the local part of an email
address contain mixed reight-to-left and left-to-right text.

If you have a simple solution for that aswell which makes people happy, let
me know.

So, I think this discussion should be about first of all the use of Unicode
in an email address.

Secondly, we can talk about the encoding. But, the encoding is not the
problem.

     paf

To: [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Wed, 03 Apr 2002 15:51:48 -0800
Lines: 24
Sender: [email-masked]

Edmon Chung <[email-masked]> writes:

> If I understand MIME correctly, even "alternate sections" are allowed.
> How is that permitted and not a UTF8 name?

multipart/alternative is one of the most politically controversial
portions of all of MIME and despite the (fairly justifiable, IMO) reasons
why it was put into the spec, probably set back adoption of MIME on Usenet
by about two years all by itself.  Even to this day, there's a completely
irrational but fairly widespread hatred of MIME in the NNTP community
because it's associated with or even considered equivalent to messages
containing tons of unreadable trash bloated to three times their original
size.

(People are slowly coming around, as more and more people point out that
this isn't even a problem with MIME but with some catastrophically bad
decisions made by certain implementations, but given how many problems it
caused on Usenet I can understand why people are gunshy.)

It's not really the example you want to emulate unless you have no other
choice.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: Keith Moore <[email-masked]>
To: "Edmon Chung" <[email-masked]>
cc: "Keith Moore" <[email-masked]>, "David Leung \(Neteka Inc.\)" <[email-masked]>, [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 03 Apr 2002 15:54:29 -0500
Sender: [email-masked]

> From: "Keith Moore" <[email-masked]>
> > the old headers will not "only" contain ACE - they will contain a mixture
> > of plain ASCII, addresses with ACE domains and ASCII local parts,
> > addresses with ACE local parts and ASCII domains, and pure ACE.
> > Even if only the ASCII components are changed this still changes the
> > information in the header.
>
> But technically, ACE names are ASCII names.  How are they supposed to be
> "changed".  Please enlighten me.

In this context I'm using ASCII here to mean names that don't use ACE.
(sorry, I thought this would be obvious)

to give a concrete example, many MTAs can rewrite some addresses in 
message headers - e.g. to hide internal domain names or to rewrite
obsolete domain names to correct ones.

Somebody sends a mail with the following header field:

To: [email-masked], [email-masked], sue@ACE-2, ACE-3@ACE-4

(I'm using ACE-x as a shorthand notation for a IDN or a email local-part
that's encoding using ACE)

now say that along with that To field the sender's MUA includes some other 
X-UTF-8-To field that spells everything out in raw utf-8.

the message then passes through an MTA that's configured to rewrite 
xxx.example.com into example.com.  The to field is then:

To: [email-masked], [email-masked], sue@ACE-2, ACE-3@ACE-4

but the MTA doesn't know about the X-UTF-8-To field, so it doesn't get
rewritten.  

if the recipient MUA looks at the X-UTF-8-To field, and tries to reply
to those addresses, the reply fails because the xxx.example.com addresses
aren't reachable outside of example.com's enterprise network.

the only way for the recipient's MUA to be able to use the X-UTF-8-To field 
is to compare it with the To field (decoding the ACE components) and see 
whether the two domains are consistent.

> If I understand MIME correctly, even "alternate sections" are allowed.  

mime provides a means of including alternate representations of contents
(essentially, attachments) - not headers.

> How is that permitted and not a UTF8 name? 

first, having alternate representations of contents (say documents) is
clearly useful and doesn't cause as many problems as having alternate 
representations of addresses in message headers.  

second, MIME has no provision to encode the same contents in different 
ways (say, one in base64 and one in quoted-printable) - since all MIME
implementations have to support both base64 and quoted-printable, there
would be absolutely no point to this.  similarly, since all MUAs that
supported IDNs would have to support ACE addresses for reasons of 
compatibility there's absolutely no point to having an alternate 
representation of the same string in pure utf-8.

third, the assumption behind multipart/ alternative is that the 
alternatives are actually different in some way (say one is plain 
text, the other PDF) and that there's a clear notion of which one 
is "better" indicated by the order of the alternatives within the
multipart.  with ACE vs. utf-8 neither one is "better" - they both
contain the same information.

fourth, at the time we wrote MIME there were relatively few MTAs that 
would rewrite message bodies in comparison to those that would rewrite
message headers.  (these days there are firewalls that do modify 
message bodies, some of which can modify a component of a multipart).
so the rewriting argument didn't seem relevant at the time, it might
be more relevant today.

frankly, even multipart/alternative has turned out to be more problematic
than we anticipated - for example, it's hard to represent a multi-dimensional
choice in such a way that the recipient's MUA can make the right decision.
and its principal use seems to be to include HTML in email which when 
displayed produces exactly the same information as plain text.  if when we 
were writing MIME we had known what we know today, we might not have 
bothered with multipart/alternative at all.  
 
> Seems to me like using a UTF8-only
> name makes a lot of sense, just like a multipart mail that have both a text
> version and an html version.

it only seems that way if you ignore how they're going to interact with
existing software.

> > no, dual representations are a bad idea.
>
> Then MIME content type for multiparts is also a bad idea according to you I
> suppose?

bottom line: they were made to solve a very different problem than the one 
you're talking about.  

Keith

To: Edmon Chung <[email-masked]>
Cc: [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
From: [email-masked]
Date: Wed, 03 Apr 2002 15:47:47 -0500
Sender: [email-masked]

--==_Exmh_-2003296507P
Content-Type: text/plain; charset=us-ascii

On Wed, 03 Apr 2002 15:31:19 EST, Edmon Chung said:

> Then MIME content type for multiparts is also a bad idea according to you I
> suppose?

There are those who hold that multipart/alternative *is* a bad idea, when
it's used to transport a text/plain and text/html with identical
semantic content.

--==_Exmh_-2003296507P
Content-Type: application/pgp-signature

--==_Exmh_-2003296507P--

From: "Edmon Chung" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "David Leung \(Neteka Inc.\)" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 3 Apr 2002 15:31:19 -0500
Sender: [email-masked]

Hi Keith,

----- Original Message -----
From: "Keith Moore" <[email-masked]>
> the old headers will not "only" contain ACE - they will contain a mixture
> of plain ASCII, addresses with ACE domains and ASCII local parts,
> addresses with ACE local parts and ASCII domains, and pure ACE.
> Even if only the ASCII components are changed this still changes the
information
> in the header.

But technically, ACE names are ASCII names.  How are they supposed to be
"changed".  Please enlighten me.

> the party reading the message (whether the recipient or some other agent)
> needs to know whether to believe the old headers or the utf8-only
> headers that you've proposed.  but unless the old headers match the
utf8-only
> headers, the party reading the message can't tell which headers to
believe.
> and checking both sets of headers to see that they match is more trouble
> than just decoding the ACE.

If I understand MIME correctly, even "alternate sections" are allowed.  How
is that permitted and not a UTF8 name?  Seems to me like using a UTF8-only
name makes a lot of sense, just like a multipart mail that have both a text
version and an html version.

> no, dual representations are a bad idea.

Then MIME content type for multiparts is also a bad idea according to you I
suppose?

Edmon

From: Keith Moore <[email-masked]>
To: "David Leung \(Neteka Inc.\)" <[email-masked]>
cc: "Keith Moore" <[email-masked]>, [email-masked], "Edmon Chung" <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 03 Apr 2002 14:44:13 -0500
Sender: [email-masked]

> > > If we are saying that the ACE <=> IDN(UTF8) so why is the two headers
> > > not in sync?
> >
> > because there are things which will know about the old headers and change
> > those, without changing the new utf8 headers.
> >
> If old headers only contains ACE why will those *things* change them, which
> means nowadays MTA or MUA may change [email-masked] to something else?! Is
> this an implementation problem?

the old headers will not "only" contain ACE - they will contain a mixture
of plain ASCII, addresses with ACE domains and ASCII local parts, 
addresses with ACE local parts and ASCII domains, and pure ACE.
Even if only the ASCII components are changed this still changes the information
in the header.  

> > because there are things which change the headers, always using the utf-8
> > headers is not the right thing to do - they might be obsolete.
> > about the best you can do is require each things that ues the new headers
> > to check for consistency with the old headers - if they're not consistent,
> > use the old headers.
> 
> I think you miss what I said, I said use ACE with the old headers and UTF8
> ONLY in the new headers. Which is not always using utf8 headers.

the party reading the message (whether the recipient or some other agent)
needs to know whether to believe the old headers or the utf8-only 
headers that you've proposed.  but unless the old headers match the utf8-only
headers, the party reading the message can't tell which headers to believe.
and checking both sets of headers to see that they match is more trouble
than just decoding the ACE.

> > in general, it's nearly always a bad idea to have two different sources
> > of the same kind of information and claim that they're both authoritative.
> > if you're going to have multiple sources of the same information,
> > you need some way to decide between them for the cases when they don't
> > agree.
> 
> So ACE is a bad idea, because it claim authoritative for the ASCII and IDN
> names : )

no, dual representations are a bad idea.  

Keith

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
To: "Keith Moore" <[email-masked]>
Cc: <[email-masked]>, "Edmon Chung" <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 3 Apr 2002 14:31:04 -0500
Sender: [email-masked]

> > If we are saying that the ACE <=> IDN(UTF8) so why is the two headers
not in
> > sync?
>
> because there are things which will know about the old headers and change
> those, without changing the new utf8 headers.
>
If old headers only contains ACE why will those *things* change them, which
means nowadays MTA or MUA may change [email-masked] to something else?! Is
this an implementation problem?

> > also if new MUA always use the new headers and only old existing MUA
wont be
> > able to read the new headers will use the old ACE headers, so why MUA
wont
> > know which one is more recent?
>
> because there are things which change the headers, always using the utf-8
> headers is not the right thing to do - they might be obsolete.
> about the best you can do is require each things that ues the new headers
> to check for consistency with the old headers - if they're not consistent,
> use the old headers.

I think you miss what I said, I said use ACE with the old headers and UTF8
ONLY in the new headers. Which is not always using utf8 headers.

> in general, it's nearly always a bad idea to have two different sources
> of the same kind of information and claim that they're both authoritative.
> if you're going to have multiple sources of the same information,
> you need some way to decide between them for the cases when they don't
> agree.

So ACE is a bad idea, because it claim authoritative for the ASCII and IDN
names : )

From: Keith Moore <[email-masked]>
To: "David Leung \(Neteka Inc.\)" <[email-masked]>
cc: [email-masked], "Keith Moore" <[email-masked]>, [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 03 Apr 2002 14:20:31 -0500
Sender: [email-masked]

> So that's why I never said putting the UTF8 into existing headers where
> existing MUA or MTA may % escaped it, what I said is USE EXISTING HEADERS
> WITH ACE, AND ADD NEW VENDOR SPECIFY HEADERS LIKE X-IDN-FROM, X-IDN-TO, etc
> FOR THE UTF8, SO THAT NEW MUA,MTA WILL BE ABLE TO PICK UP THOSE HEADERS AND
> USE THEM, AND OLD EXISTING MUA, MTA WILL JUST USE THE ACE IN THE OLD HEADER
> LINES. 

yes we understand your proposal.  we're trying to tell you why it doesn't work.

Keith

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
To: <[email-masked]>
Cc: "Keith Moore" <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 3 Apr 2002 14:23:08 -0500
Sender: [email-masked]

>> If we are saying that the ACE <=> IDN(UTF8) so why is the two headers not
in
>> sync?
>
>To paraphrase somebody (Was it Vernon Schreier?):
>
>Either the two fields are semantically identical, in which case it is
>superfluous to send one of them, or they are not, in which case there is
>breakage if you use the one with less semantic content.
>
>His comment was regarding text/plain and text/html, but is equally true
here.

If the headers are one of the X- headers which is vendor specific, even
existing mail will have X- headers that can only be read by certain MUA, and
simply ignore by most MTA, so what breakage are you talking about?? Unless
the existing implementation of some MTA and MUA is not correct and raise
problem when parsing the headers that has X-...

>> also if new MUA always use the new headers and only old existing MUA wont
be
>> able to read the new headers will use the old ACE headers, so why MUA
wont
>> know which one is more recent?
>
>You also have the case where you're using a new MUA, but the mail could
have
>passed through and old MTA that didn't know about the new headers, and thus
>did a conversion on the old headers but not the new ones.

How is the MTA going to do a conversion, if it is an old MTA that wont be
able to read this new headers, why and how would the MTA all of a sudden
converting a line that it cannot read, also the old header contains the ACE,
so why is the old MTA trying to change the ACE address, will existing MTA
change your [email-masked] to something else?!

>Those of us who lived throught the %-hack ages know ALL about this - you'd
>get a mail that had a replyable 'From:' because an MTA had %-hacked it
>for you, but a 'reply all' would fail miserably because the MTA had failed
>to %-hack the CC: list.
So that's why I never said putting the UTF8 into existing headers where
existing MUA or MTA may % escaped it, what I said is USE EXISTING HEADERS
WITH ACE, AND ADD NEW VENDOR SPECIFY HEADERS LIKE X-IDN-FROM, X-IDN-TO, etc
FOR THE UTF8, SO THAT NEW MUA,MTA WILL BE ABLE TO PICK UP THOSE HEADERS AND
USE THEM, AND OLD EXISTING MUA, MTA WILL JUST USE THE ACE IN THE OLD HEADER
LINES. If MUA or MTA that touches the lines X- which it doesnt even
recognize, then I think that should be considered a bad or incorrect
implementation.

From: Keith Moore <[email-masked]>
To: "Daryl Odnert" <[email-masked]>
cc: "'Keith Moore'" <[email-masked]>, [email-masked], "Jacob Palme" <[email-masked]>, [email-masked]
Subject: Re: Spam prevention gone too far? 
Date: Wed, 03 Apr 2002 14:07:49 -0500
Sender: [email-masked]

> > As for the problem of inappropriate bouncing of mail - I do think it
> > might be useful to write an RFC documenting dubious practices,
> > such as refusing mail with a null return-path, expecting the mail
> > to come from a client machine with the same name as the sender's 
> > domain, trusting blacklists, etc.
> 
> See RFC 2505 - "Anti-Spam Recommendations for SMTP MTAs".

some of the necessary information is buried in there, but not all of it.
also, times have changed a bit, and I'd make slightly different 
recommendations today than are in that document, based on recent experience.

From: Keith Moore <[email-masked]>
To: "David Leung \(Neteka Inc.\)" <[email-masked]>
cc: "Keith Moore" <[email-masked]>, [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 03 Apr 2002 13:59:25 -0500
Sender: [email-masked]

> If we are saying that the ACE <=> IDN(UTF8) so why is the two headers not in
> sync?

because there are things which will know about the old headers and change
those, without changing the new utf8 headers.

> also if new MUA always use the new headers and only old existing MUA wont be
> able to read the new headers will use the old ACE headers, so why MUA wont
> know which one is more recent?

because there are things which change the headers, always using the utf-8
headers is not the right thing to do - they might be obsolete.
about the best you can do is require each things that ues the new headers 
to check for consistency with the old headers - if they're not consistent,
use the old headers.

in general, it's nearly always a bad idea to have two different sources 
of the same kind of information and claim that they're both authoritative.
if you're going to have multiple sources of the same information,
you need some way to decide between them for the cases when they don't 
agree.

Keith

From: "Daryl Odnert" <[email-masked]>
To: "'Keith Moore'" <[email-masked]>, [email-masked]
cc: "Jacob Palme" <[email-masked]>, [email-masked]
Subject: RE: Spam prevention gone too far?
Date: Wed, 3 Apr 2002 10:39:39 -0800
Sender: [email-masked]

Keith Moore wrote:
> As for the problem of inappropriate bouncing of mail - I do think it
> might be useful to write an RFC documenting dubious practices,
> such as refusing mail with a null return-path, expecting the mail
> to come from a client machine with the same name as the sender's 
> domain, trusting blacklists, etc.

See RFC 2505 - "Anti-Spam Recommendations for SMTP MTAs".

Daryl Odnert
Tumbleweed Communications
[email-masked]

To: "David Leung (Neteka Inc.)" <[email-masked]>
Cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
From: [email-masked]
Date: Wed, 03 Apr 2002 13:42:18 -0500
Sender: [email-masked]

--==_Exmh_233976269P
Content-Type: text/plain; charset=us-ascii

On Wed, 03 Apr 2002 13:24:20 EST, "David Leung (Neteka Inc.)" said:

> If we are saying that the ACE <=> IDN(UTF8) so why is the two headers not in
> sync?

To paraphrase somebody (Was it Vernon Schreier?):

Either the two fields are semantically identical, in which case it is
superfluous to send one of them, or they are not, in which case there is
breakage if you use the one with less semantic content.

His comment was regarding text/plain and text/html, but is equally true here.

> also if new MUA always use the new headers and only old existing MUA wont be
> able to read the new headers will use the old ACE headers, so why MUA wont
> know which one is more recent?

You also have the case where you're using a new MUA, but the mail could have
passed through and old MTA that didn't know about the new headers, and thus
did a conversion on the old headers but not the new ones.

Those of us who lived throught the %-hack ages know ALL about this - you'd
get a mail that had a replyable 'From:' because an MTA had %-hacked it
for you, but a 'reply all' would fail miserably because the MTA had failed
to %-hack the CC: list.

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

--==_Exmh_233976269P
Content-Type: application/pgp-signature

--==_Exmh_233976269P--

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
To: "Keith Moore" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 3 Apr 2002 13:24:20 -0500
Sender: [email-masked]

> > > > If we add new MIME headers and ESMTP commands, MUA and MTA should be
> > > > able to have a smooth transition from 7bit encoded 8bits to UTF8.
> > >
> > > I don't think that's the right way to go - first, because SMTP MTAs
have
> > > no idea about whether the MUA can support raw utf-8 (this was the
mistake
> > > we made with 8BITMIME, let's not repeat it); second, because expecting
> > > MTAs to extensively rewrite messages is a good way to introduce
errors;
> > > third, because this pretends that a good part of the problem can be
> > > solved by changes to the MTAs, when in reality the UA issues are a far
> > > bigger hurdle.
> >
> > Let me give an example: *Enhanced* MUA that can use ESMTP will use the
new
> > ESMTP command with the new MIME headers to have UTF8... let's say the
new
> > header we called it X-IDN-FROM and X-IDN-TO,
>
> that solution path was considered for MIME back in 1990 or so.  it was
> a non-starter then, and I expect it's a non-starter now, for the same
reasons.
> it's a very bad idea to have two different versions of a message header,
> first because they won't stay in sync (these things do change somtimes in
> transit, sometimes for good reasons), and second because an MUA that
> supports both won't know which one is more recent.
>
> in short, what you're proposing is far worse than 2047 and ACE.

If we are saying that the ACE <=> IDN(UTF8) so why is the two headers not in
sync?
also if new MUA always use the new headers and only old existing MUA wont be
able to read the new headers will use the old ACE headers, so why MUA wont
know which one is more recent?

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: Jacob Palme <[email-masked]>, [email-masked]
Subject: Re: Spam prevention gone too far? 
Date: Wed, 03 Apr 2002 12:27:13 -0500
Sender: [email-masked]

> I did like the last part of this RISKS item, which said:
> 
> > I would even go so far as to say that I question the
> > legitimacy of allowing RISKS submissions from people who make it impossible
> > for people to send them E-mail responses to their submissions.
> 
> A Tom Lehrer line comes to mind:
> 
>    I know there are people in this world who do not love their fellow human
>    beings and I hate people like that!

I'm reminded of Gosper's line from HAKMEM about a guy with a
"read-only mind".   How else do you desribe someone who insists on sending 
mail to large groups of people without being able to accept a response? 

As for the problem of inappropriate bouncing of mail - I do think it
might be useful to write an RFC documenting dubious practices,
such as refusing mail with a null return-path, expecting the mail
to come from a client machine with the same name as the sender's 
domain, trusting blacklists, etc.  But traditionally we've said
that the folks running mail servers have the right to refuse mail for 
any reason, and I don't think that we can change that.    

As usual, we can try to provide useful advice.  Whether folks follow it 
is a different matter.

Keith

Date: Wed, 3 Apr 2002 12:26:41 +0200 (CEST)
From: Erik Nordmark <[email-masked]>
Reply-To: Erik Nordmark <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever!
To: "Eric A. Hall" <[email-masked]>
Cc: James Seng <[email-masked]>, Edmon Chung <[email-masked]>, [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> Dan should certainly be encouraged to resubmit his UDNS proposal, but in
> the meantime, I would like to point out that draft-hall-dm-idns-00.txt is
> the dual-mode draft that is currently in the WG pool. Perhaps your
> confusion stems from the I-D-N.net web site, which still lists UDNS as an
> active document, and which does not list DM-IDNS.

And for those that are looking at other IDNA to UTF-8 in DNS transitional
schemes it would be useful to look at the open issue regarding DNSSEC
that I pointed out about draft-hall-dm-idns-00.txt a while back.

I suspect that all such transitional schemes will have the same issue
when DNSSEC is used to provide integrity of the RRsets: can't depends
on servers synthesing records in general and can't have caching resolvers
modify the queries and/or responses.

   Erik

Date: Wed, 3 Apr 2002 10:52:20 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Jacob Palme <[email-masked]>
Cc: [email-masked]
Subject: Re: Spam prevention gone too far?
Sender: [email-masked]

Jacob Palme <[email-masked]>
> Quoted below is a message from the RISKS mailing list.
> Any comments by SMTP experts on this?

I guess we've all seen countless bounces as result of heartachingly stupid
misconfiguration. The fact that this particlar SNAFU is due to misguided
spam filtering isn't important. Everything else about the world's MTAs has
been misused now and then, why should spam filters be an exception?

--Arnt

Date: Tue, 02 Apr 2002 23:52:43 -0800 (PST)
From: [email-masked]
Subject: Re: Spam prevention gone too far?
To: Jacob Palme <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Quoted below is a message from the RISKS mailing list.
> Any comments by SMTP experts on this?

There's nothing new here. This sort of stuff has been going on for years and it
happens all the time; check out any mailing list where operational issues for
messaging software are discussed and you'll find dozens of similar accounts.

And as far as messaging standards go, there's really nothing to say. Site
administrators are people, and people sometimes do monumentally stupid things.
Unfortunately you cannot standardize intelligence or outlaw stupidity.

I did like the last part of this RISKS item, which said:

> I would even go so far as to say that I question the
> legitimacy of allowing RISKS submissions from people who make it impossible
> for people to send them E-mail responses to their submissions.

A Tom Lehrer line comes to mind:

   I know there are people in this world who do not love their fellow human
   beings and I hate people like that!

				Ned

Date: Wed, 3 Apr 2002 08:57:04 +0200
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Spam prevention gone too far?
Sender: [email-masked]

Quoted below is a message from the RISKS mailing list.
Any comments by SMTP experts on this?

--- ---

Date: Thu, 24 Jan 2002 16:23:17 -0500
From: Jonathan Kamens <[email-masked]>
Subject: Spam prevention gone too far

I recently attempted to send E-mail to the author of a RISKS submission.
Since my DSL line was down when I sent the E-mail, and since outbound SMTP
connections are blocked from the dial-up accounts provided by my ISP, I had
to send my E-mail through my ISP's mail server.  It bounced as follows (the
identify of the intended recipient has been masked):

   <RECIPIENT@RECIPIENT-HOST>:
   Connected to RECIPIENT-HOST-IP but sender was rejected.
   Remote host said: 571 <[email-masked]>... 
Return address  [email-masked]  does not match 
sending computer  mail11.speakeasy.net  -- check your 
configuration. http://www.RECIPIENT-HOST/mail/571_2.html 
for details.

If you visit the URL referenced above, you discover that this site's system
administrators have decided to block all E-mail for which the host name in
the envelope address can't be matched up obviously (using a simple string
comparison) with the host name of the mail server sending the message.  In
other words, if you have your own domain name, but you send E-mail through
your ISP's mail server, you simply can't send E-mail to this site.
Supposedly, they check their logs for such bounces "as we have time" and add
messages that should have gone through to an exception list, but who knows
when/if they'll ever get around to doing that in any particular case.
Furthermore, they provide no mechanism for contacting them by E-mail or Web
to ask to be excepted -- all they give on the Web page is a long-distance
telephone number.

Fortunately for me, or so I thought, I *do* send outbound mail through my
own mail server when my DSL line is up, and it was fixed yesterday, so I
decided to attempt to resend my message.  It bounced again, with a different
error:

   RECIPIENT@RECIPIENT-HOST
       (reason: 550 We do not accept mail from the 
spam-relay machine:  jik-0.dsl.speakeasy.net.. 
http://www.RECIPIENT-HOST/mail/571_1.html for details.)

If you visit *that* URL, you see that they're claiming that my machine is a
spam relayer.  It isn't and never has been.  I've never sent spam and I
block all third-party relaying through my machine.  I can't find an entry
for either my IP address or my subnet in any of the black-lists checked by
   <URL:http://relays2.osirusoft.com/cgi-bin/rbcheck.cgi>.

Of course, they don't bother to say *why* they think my machine is a
spam-relay machine, so who knows where they got that charming idea?  And, as
mentioned above, they don't provide any way to contact them on-line to
complain about it.  For example, many sites which enforce restrictions this
draconian provide an address which is exempt from the restrictions to which
people can complain; the spammers don't ever bother complaining, so it
really isn't particularly burdensome to do this.  Unless, of course, you
really don't care if people can't send legitimate E-mail to your site.

I understand the desire to block spam, but there are ways to do it which
don't also block legitimate E-mail, or at the very least which provide an
on-line mechanism people can use for getting themselves unblocked.  This is
just really excessive; I would even go so far as to say that I question the
legitimacy of allowing RISKS submissions from people who make it impossible
for people to send them E-mail responses to their submissions.

   jik

From: Keith Moore <[email-masked]>
To: "David Leung \(Neteka Inc.\)" <[email-masked]>
cc: "Keith Moore" <[email-masked]>, [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Tue, 02 Apr 2002 19:44:43 -0500
Sender: [email-masked]

> > > If we add new MIME headers and ESMTP commands, MUA and MTA should be
> > > able to have a smooth transition from 7bit encoded 8bits to UTF8.
> >
> > I don't think that's the right way to go - first, because SMTP MTAs have
> > no idea about whether the MUA can support raw utf-8 (this was the mistake
> > we made with 8BITMIME, let's not repeat it); second, because expecting
> > MTAs to extensively rewrite messages is a good way to introduce errors;
> > third, because this pretends that a good part of the problem can be
> > solved by changes to the MTAs, when in reality the UA issues are a far
> > bigger hurdle.
> 
> Let me give an example: *Enhanced* MUA that can use ESMTP will use the new
> ESMTP command with the new MIME headers to have UTF8... let's say the new
> header we called it X-IDN-FROM and X-IDN-TO, 

that solution path was considered for MIME back in 1990 or so.  it was
a non-starter then, and I expect it's a non-starter now, for the same reasons.
it's a very bad idea to have two different versions of a message header,
first because they won't stay in sync (these things do change somtimes in 
transit, sometimes for good reasons), and second because an MUA that 
supports both won't know which one is more recent.

in short, what you're proposing is far worse than 2047 and ACE.

Keith

From: "James Seng" <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
Cc: "Edmon Chung" <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever!
Date: Wed, 3 Apr 2002 08:43:38 +0800
Sender: [email-masked]

My apologises.

Let me correct myself: DM-IDNS or UDNS are both good starting point, one
which the UTF-8 supportor group should go aside and finish it.

-James Seng

> James Seng wrote:
>
> > It is also okay if you wish to start from UDNS, a draft which I find
> > it is reasonable altho incomplete. Talk to Oscarsson if you wish but
> > lets put it down in writing.
>
> Dan should certainly be encouraged to resubmit his UDNS proposal, but in
> the meantime, I would like to point out that draft-hall-dm-idns-00.txt is
> the dual-mode draft that is currently in the WG pool. Perhaps your
> confusion stems from the I-D-N.net web site, which still lists UDNS as an
> active document, and which does not list DM-IDNS.
>
> [For the record, message <016c01c14c76$ea176c20$dd00a8c0@jamessonyvaio>
> shows that UDNS was removed from the pool on Oct 4, 2001 while message
> <023301c172ee$62f4fc50$0701000a@jamessonyvaio> shows that DM-IDNS was
> accepted into the WG pool on Nov 22, 2001. Don't rely on the web site.]
>
> --
> Eric A. Hall                                        http://www.ehsco.com/
> Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 2 Apr 2002 22:30:43 +0200
From: Keld =?iso-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>
To: Edmon Chung <[email-masked]>
Cc: James Seng <[email-masked]>, Keld =?iso-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>, [email-masked], John Stracke <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

On Tue, Apr 02, 2002 at 02:24:43PM -0500, Edmon Chung wrote:
> James,
> 
> I think you missed the point completely.  The point, as Keld have mentioned
> is that there are people, who believes that it is beneficial to have UTF8
> option, I being one of them and Keld being one of them, and it is a fact
> that cannot be ignored and should not be ignored.  From my interpretation of
> your previous words, you also believe that it is beneficial to go beyond
> ASCII.  That, is my point and please dont try to confuse the message with
> the means!  Human language is not machine language and you have to
> understand the context before you can get the message.  If you cant
> understand this simple depth and intricacy of human interaction and
> communication, I cant help you.

Well, I have to state that I am not so sure we need an UTF-8 upgrade.
Maybe ACE is enough. Things can get complicated if eg the primary DNS
handles utf-8 while one of the secondaries do not. 
DNS is different from SMTP as all DNS clients need to *understand*
all DNS servers, each character needs to be understood. In SMTP the
MTAs are only exchanging messages between consenting adults, and
it is up to the sender and receiver to mutually understand eachother. 

Also the situation back in 1991 with SMTP was different from
todays DNS issues. Then we had a lot of email that was not
ASCII, but a myriad of other encodings, incl national 7-bit variants,
japanese, and iso-8859-?. Today we almost entirely still have
only (restricted) ascii in the DNS.

I frankly think that for interoperation, all DNS data needs to be
stored in ACE encoded ISO 10646, and on the wire it should be ACE
encoded queries and responses. It would then be up to the clients
to do the normalization and conversion from/to ACE.
That would mean no changes to existing DNS protocols!

Then you could have tools for maintenance of the zone data in
one or more charsets for convenience of the DNS zone maintainers.

A weak point of mine is that *iff* we go beyond ASCII, UTF-8 is
the only additional encoding we should consider for the wire,
and *iff* seemless interoperability can be proven with existing 
operations and some gains can be shown, I would not be against 
a new UTF-8 protocol for DNS.

Best regards
Keld

From: "James Seng" <[email-masked]>
To: "Edmon Chung" <[email-masked]>, =?iso-8859-1?Q?Keld_J=F8rn_Simonsen?= <[email-masked]>
Cc: <[email-masked]>, "John Stracke" <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever!
Date: Wed, 3 Apr 2002 08:41:44 +0800
Sender: [email-masked]

And you miss the point completely too.

IETF move forward with draft. Repeatively say "i want utf-8, i want it now"
after the first time are just noise.

Write your draft. It is make sense, then support will come.

-James Seng

> I think you missed the point completely.  The point, as Keld have
mentioned
> is that there are people, who believes that it is beneficial to have UTF8
> option, I being one of them and Keld being one of them, and it is a fact
> that cannot be ignored and should not be ignored.  From my interpretation
of
> your previous words, you also believe that it is beneficial to go beyond
> ASCII.  That, is my point and please dont try to confuse the message with
> the means!  Human language is not machine language and you have to
> understand the context before you can get the message.  If you cant
> understand this simple depth and intricacy of human interaction and
> communication, I cant help you.
>
> The exercise is to establish a meaningful support before I waste my time
> working on a draft if no one wants to talk about it.
> I have a draft of a draft inplace for UDNA, and I am updating it as we
> continue to discuss some of the elements of it on the list.  I think it
> makes more sense this way, than to throw something in here fresh.  Also I
> think it would be beneficial to have a draft on IDNX (IDN Transition).  So
> please stop distrupting the discussion of a good discussion here.
>
> Edmon
>
>
> ----- Original Message -----
> From: "James Seng" <[email-masked]>
> To: "Edmon Chung" <[email-masked]>; "Keld Jørn Simonsen" <[email-masked]>
> Cc: <[email-masked]>; "John Stracke" <[email-masked]>;
> <[email-masked]>; <[email-masked]>; <[email-masked]>
> Sent: Tuesday, April 02, 2002 1:25 PM
> Subject: Re: [idn] Re: 7 bits forever!
>
>
> > > Whole heartedly agree.  This time we have yourself and the Asian
> community
> > > standing up saying we want IDN(UTF8), hopefully, this is unquestioned
as
> > > well.
> >
> > I am aware that some chinese members in this group have stood up to ask
> for
> > TC/SC in IDN. I am also aware that Lee have asked for reordering.
> >
> > But I am not aware of any "Asian community standing up saying we want
> > IDN(UTF8)". I honestly hope I havent missed anything important.
> >
> > Or is this just another fragment of your imagation?
> >
> > It is frustrating to hear you repeating "I want UTF-8, I want UTF-8
now!"
> > again and again. Lets just say "yes, we hear you say that and lets move
> on".
> >
> > By moving on, I mean you write a draft to cover:
> > a) what is the advantage of IDN(UTF8) over the existing IDNA
> > b) how IDN(UTF8) is going to work
> > c) how you intend to get there
> >
> > -James Seng
> >
> >
> >
>

Date: Tue, 2 Apr 2002 19:10:32 +0200
From: Keld =?iso-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>
To: Edmon Chung <[email-masked]>
Cc: [email-masked], John Stracke <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

On Tue, Apr 02, 2002 at 09:56:44AM -0500, Edmon Chung wrote:
> I think the issue here is how we can plan a better transition than what we
> have done for SMTP.
> That is, if we start thinking about "8+ everywhere" instead of "ACE
> everywhere" scenario, then how we can get there from here. That is the
> question for the IDN group I believe.

Well, I think SMTP was very successful in getting to 8-bits everywhere.
Or almost everywhere. Today ESMTP is the predominant protocol for email
exchange. And this was really not in the cards when we designed the 
MIME stuff. The predominant policy at those times was  "use 7-bit
generally, and 8-bit locally", the so-called "enclave" model. 
I was amongst the most vocal proponents af a general 8-bit email
exchange protocol, the one that emerged as ESMTP. I remember being
at IETF in Santa Fé 1992-ish, where we discussed this, and I was the
only European around when it was asked if somebody, especially from
Europe, really wanted a general 8-bit exchange protocol. And I raised
my voice and insisted. So it was not really questioned anymore
and ESMTP was produced. But mostly as a courtesy to the Europeans.
Sometimes you can make a difference:-)

I think getting to >8bit is a non-starter, as per IESG policies
that promotes utf-8 as the primary encoding in protocols.
And I really do not see why we should open up to a myriad
of charsets (being the author of rfc 1345 I need to stress that:-)
We have one, we can deal with two, not many.

What we have now is a restricted ASCII for DNS, and we want to
go to a restricted utf-8. And nothing else. And we want the
backwards compatibility and operation between ascii and utf-8
so we can ensure a happy migration and coexistance.

Best regards
Keld

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
To: "Keith Moore" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Tue, 2 Apr 2002 18:18:19 -0500
Sender: [email-masked]

> > > using raw utf-8 in message headers wouldn't replace 2047. instead,
> > > it would add yet another format that MUAs are expected to parse
> > > and display. it would not reduce complexity, it would add to it.
> >
> > If we add new MIME headers and ESMTP commands, MUA and MTA should be
able to
> > have a smooth transition from 7bit encoded 8bits to UTF8.
>
> I don't think that's the right way to go - first, because SMTP MTAs have
> no idea about whether the MUA can support raw utf-8 (this was the mistake
> we made with 8BITMIME, let's not repeat it); second, because expecting
> MTAs to extensively rewrite messages is a good way to introduce errors;
> third, because this pretends that a good part of the problem can be
> solved by changes to the MTAs, when in reality the UA issues are a far
> bigger hurdle.

Let me give an example: *Enhanced* MUA that can use ESMTP will use the new
ESMTP command with the new MIME headers to have UTF8... let's say the new
header we called it X-IDN-FROM and X-IDN-TO, and also the existing FROM and
TO will have ACE, so none *Enhanced* MUA will still display the ACE while
new MUA that can read X-IDN-FROM and X-IDN-TO will be able to display the
UTF8 as is... also any MTA along the path will act the same way too, capable
server will use ESMTP with the new MIME header, and none capable/existing
MTA will still use SMTP with ACE and the existing header, there is no need
for the MTA to *extensively rewrite messages*, the *Enhanced* MUA will have
both send out if the first hop MTA is able to use the new ESMTP and MIME
headers, if not then the mail will be send with ACE, which works the same
way as regular email address now adays... I am not saying that we can solve
this by just changing the MTA, but what I am saying is that we should have
the MTA ready, so when we have MUA that can use the new ESMTP and MIME
headers, it will be a smoother transition... what we are doing all the time,
is proposing some implementation that can make smoother transition and not
directly supporting just ACE or UTF8 extremely, like the others... we want
to have both ACE and UTF8 to co-exists so that the transition will be
smooth, why forcing to have just ACE or just UTF8?!

> > If there is a requirement of change anyway, why not change it to
> > something that is more simple and well support by the Unicode
Consortium.
>
> It's simply not the case that raw utf-8 in headers is simpler to deal
> with than the current method of encoding things in ASCII.  There are
> complexities associated with each, but the UTF-8 complexities are new
> ones whereas the ASCII ones are already familiar. In some ways we'd be
> far better off if we kept structured fields in ASCII - even though that
> seems unpalatable.

I agree with this, we are familiar with the behaviour of ASCII headers, so
we should keep ASCII with old headers, but if we introduce new headers
especially the ones that start with X-, where they are vendor specified, I
dont see why any MTA or MUA will behave badly since they are not suppose to
be able to pick up those headers anyways, not unless a bad implementation
: )

From: Keith Moore <[email-masked]>
To: "David Leung \(Neteka Inc.\)" <[email-masked]>
cc: "Keith Moore" <[email-masked]>, [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Tue, 02 Apr 2002 17:02:27 -0500
Sender: [email-masked]

[recipient list trimmed; this has nothing to do with idn, and is too
narrow for IETF, and Dan's mailer complains if you cc him anyway.]

> > using raw utf-8 in message headers wouldn't replace 2047. instead,
> > it would add yet another format that MUAs are expected to parse
> > and display. it would not reduce complexity, it would add to it.
> 
> If we add new MIME headers and ESMTP commands, MUA and MTA should be able to
> have a smooth transition from 7bit encoded 8bits to UTF8. 

I don't think that's the right way to go - first, because SMTP MTAs have
no idea about whether the MUA can support raw utf-8 (this was the mistake 
we made with 8BITMIME, let's not repeat it); second, because expecting 
MTAs to extensively rewrite messages is a good way to introduce errors;
third, because this pretends that a good part of the problem can be
solved by changes to the MTAs, when in reality the UA issues are a far 
bigger hurdle.

> If there is a requirement of change anyway, why not change it to 
> something that is more simple and well support by the Unicode Consortium.

It's simply not the case that raw utf-8 in headers is simpler to deal
with than the current method of encoding things in ASCII.  There are
complexities associated with each, but the UTF-8 complexities are new
ones whereas the ASCII ones are already familiar. In some ways we'd be 
far better off if we kept structured fields in ASCII - even though that
seems unpalatable.

Keith

Date: Tue, 02 Apr 2002 13:40:43 -0600
From: "Eric A. Hall" <[email-masked]>
To: James Seng <[email-masked]>
CC: Edmon Chung <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

James Seng wrote:

> It is also okay if you wish to start from UDNS, a draft which I find
> it is reasonable altho incomplete. Talk to Oscarsson if you wish but
> lets put it down in writing.

Dan should certainly be encouraged to resubmit his UDNS proposal, but in
the meantime, I would like to point out that draft-hall-dm-idns-00.txt is
the dual-mode draft that is currently in the WG pool. Perhaps your
confusion stems from the I-D-N.net web site, which still lists UDNS as an
active document, and which does not list DM-IDNS.

[For the record, message <016c01c14c76$ea176c20$dd00a8c0@jamessonyvaio>
shows that UDNS was removed from the pool on Oct 4, 2001 while message
<023301c172ee$62f4fc50$0701000a@jamessonyvaio> shows that DM-IDNS was
accepted into the WG pool on Nov 22, 2001. Don't rely on the web site.]

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: "Edmon Chung" <[email-masked]>
To: "James Seng" <[email-masked]>, "Keld Jørn Simonsen" <[email-masked]>
Cc: <[email-masked]>, "John Stracke" <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever!
Date: Tue, 2 Apr 2002 14:24:43 -0500
Sender: [email-masked]

James,

I think you missed the point completely.  The point, as Keld have mentioned
is that there are people, who believes that it is beneficial to have UTF8
option, I being one of them and Keld being one of them, and it is a fact
that cannot be ignored and should not be ignored.  From my interpretation of
your previous words, you also believe that it is beneficial to go beyond
ASCII.  That, is my point and please dont try to confuse the message with
the means!  Human language is not machine language and you have to
understand the context before you can get the message.  If you cant
understand this simple depth and intricacy of human interaction and
communication, I cant help you.

The exercise is to establish a meaningful support before I waste my time
working on a draft if no one wants to talk about it.
I have a draft of a draft inplace for UDNA, and I am updating it as we
continue to discuss some of the elements of it on the list.  I think it
makes more sense this way, than to throw something in here fresh.  Also I
think it would be beneficial to have a draft on IDNX (IDN Transition).  So
please stop distrupting the discussion of a good discussion here.

Edmon

----- Original Message -----
From: "James Seng" <[email-masked]>
To: "Edmon Chung" <[email-masked]>; "Keld Jørn Simonsen" <[email-masked]>
Cc: <[email-masked]>; "John Stracke" <[email-masked]>;
<[email-masked]>; <[email-masked]>; <[email-masked]>
Sent: Tuesday, April 02, 2002 1:25 PM
Subject: Re: [idn] Re: 7 bits forever!

> > Whole heartedly agree.  This time we have yourself and the Asian
community
> > standing up saying we want IDN(UTF8), hopefully, this is unquestioned as
> > well.
>
> I am aware that some chinese members in this group have stood up to ask
for
> TC/SC in IDN. I am also aware that Lee have asked for reordering.
>
> But I am not aware of any "Asian community standing up saying we want
> IDN(UTF8)". I honestly hope I havent missed anything important.
>
> Or is this just another fragment of your imagation?
>
> It is frustrating to hear you repeating "I want UTF-8, I want UTF-8 now!"
> again and again. Lets just say "yes, we hear you say that and lets move
on".
>
> By moving on, I mean you write a draft to cover:
> a) what is the advantage of IDN(UTF8) over the existing IDNA
> b) how IDN(UTF8) is going to work
> c) how you intend to get there
>
> -James Seng
>
>
>

From: "James Seng" <[email-masked]>
To: "Edmon Chung" <[email-masked]>, =?iso-8859-1?Q?Keld_J=F8rn_Simonsen?= <[email-masked]>
Cc: <[email-masked]>, "John Stracke" <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever!
Date: Wed, 3 Apr 2002 02:25:46 +0800
Sender: [email-masked]

> Whole heartedly agree.  This time we have yourself and the Asian community
> standing up saying we want IDN(UTF8), hopefully, this is unquestioned as
> well.

I am aware that some chinese members in this group have stood up to ask for
TC/SC in IDN. I am also aware that Lee have asked for reordering.

But I am not aware of any "Asian community standing up saying we want
IDN(UTF8)". I honestly hope I havent missed anything important.

Or is this just another fragment of your imagation?

It is frustrating to hear you repeating "I want UTF-8, I want UTF-8 now!"
again and again. Lets just say "yes, we hear you say that and lets move on".

By moving on, I mean you write a draft to cover:
a) what is the advantage of IDN(UTF8) over the existing IDNA
b) how IDN(UTF8) is going to work
c) how you intend to get there

-James Seng

From: "James Seng" <[email-masked]>
To: "Edmon Chung" <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Wed, 3 Apr 2002 02:07:12 +0800
Sender: [email-masked]

> Its so great to at least have some response.  ;-)
> I have said that there will be holes in my initial proposal, but this is a
> good start.  Lets work on ironing out the pathway.

It is nice to have a response. It would be better if you have a draft to
explain what you doing. It is also okay if you wish to start from UDNS, a
draft which I find it is reasonable altho incomplete. Talk to Oscarsson if
you wish but lets put it down in writing.

-James Seng

From: "Edmon Chung" <[email-masked]>
To: "Keld Jørn Simonsen" <[email-masked]>
Cc: <[email-masked]>, "John Stracke" <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever!
Date: Tue, 2 Apr 2002 12:43:22 -0500
Sender: [email-masked]

Hi Keld,

----- Original Message -----
From: "Keld Jørn Simonsen" <[email-masked]>
> I was amongst the most vocal proponents af a general 8-bit email
> exchange protocol, the one that emerged as ESMTP. I remember being
> at IETF in Santa Fé 1992-ish, where we discussed this, and I was the
> only European around when it was asked if somebody, especially from
> Europe, really wanted a general 8-bit exchange protocol. And I raised
> my voice and insisted. So it was not really questioned anymore
> and ESMTP was produced. But mostly as a courtesy to the Europeans.
> Sometimes you can make a difference:-)

Whole heartedly agree.  This time we have yourself and the Asian community
standing up saying we want IDN(UTF8), hopefully, this is unquestioned as
well.

> I think getting to >8bit is a non-starter, as per IESG policies
> that promotes utf-8 as the primary encoding in protocols.
> And I really do not see why we should open up to a myriad
> of charsets (being the author of rfc 1345 I need to stress that:-)
> We have one, we can deal with two, not many.
>
> What we have now is a restricted ASCII for DNS, and we want to
> go to a restricted utf-8. And nothing else. And we want the
> backwards compatibility and operation between ascii and utf-8
> so we can ensure a happy migration and coexistance.
>
> Best regards
> Keld
>

Makes good sense.

Edmon

From: "Edmon Chung" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever! 
Date: Tue, 2 Apr 2002 12:36:36 -0500
Sender: [email-masked]

Its so great to at least have some response.  ;-)
I have said that there will be holes in my initial proposal, but this is a
good start.  Lets work on ironing out the pathway.

> On Tue, 02 Apr 2002 10:29:00 EST, Edmon Chung said:
> > 1. IDNA clients
> >     - the client does ACE and display it properly for the user
> >     - servers (including DNS and mail and http, etc.) does NOT upgrade
for
> > ACE.  Even administration side is kept as is, allowing leakage
> >     - IDNA clients MUST also be able to decode IDN(UTF8)/EDNS responses
>
> This makes the rather rash assumption that the clients will be upgraded
> to support a *future* feature.  You're also assuming that there will be
> a way to test/debug IDN/EDNS responses when there are none actually being
> seen "in the wild".

The assumption is no different than assuming clients will upgrade to ACE.
If we can assume that clients will be willing to upgrade to IDN then, there
is no reason to think that implementations would not include both ACE and
IDN(UTF8)/EDNS if IDN so specifies.

> > 2. IDN(UTF8)/EDNS DNS Server upgrade
> >     - DNS servers will now accept both ACE (as usual) and IDN(UTF8)/EDNS
> > requests
>
> Are there any hidden gotchas here?  Are there any changes needed to BIND
> to allow it to process a UTF8 zone file?  Are there any "bad news"
> characters
> that might cause parsing problems?

This is an application implementation issue, NOT a protocol issue.  If BIND
doesnt implement it well, there will be other software vendors that will
make it work as "specified".

> >     - ACE requests will yield Both an ACE response + an IDN(UTF8)/EDNS
> > response
>
> Here there be dragons.  Do a 'dig aol.com mx', and ask yourself what
> happens to performance if that doesn't fit into one DNS packet anymore,
> and as a result a resolver has to drop back to TCP (instead of one packet
> each way, you're now up to a 3-packet handshake, the data, and the FIN
> packets) - and note that the TTL on the AOL.COM SOA is 3600....

I dont understand.  If you do a dig with any non-upgraded clients, then you
will be only digging for English domains.  If you are using an upgraded
client then you will be able to parse the packet.  Also, for most cases I
would imagine it will still fit in one packet.

> What happens to an ACE-based client that receives an EDNS response that
it's
> not able to parse?

That, as specified in 1. is not possible.  ALL ACE implementations MUST also
implement IDN/EDNS.  That is why the discussion should start NOW! Finally
someone realizes my point on why we need to discuss this now and not after
we have done the ACE part :-)

> >     - IDNA clients continue to send in ACE but will start to see UDNA
> > responses
>
> This is a can of worms - how does the server know for sure that the client
> is upgraded and is able to parse a UDNA response?  Remember - the client
> is probably *NOT* asking directly.  If I've upgraded example.com's DNS
> to be UDNA ready, how do I know it's safe to send a UDNA response to a
query
> from foo-bar.com (since it's quite possible that the *USER* has upgraded,
> but the sysadmins have NOT upgraded the DNS that's recursing the query for
> the user....

I see your point.  What if ACE requests will receive only ACE results, while
UDNA requests will get both the ACE and the UTF result.  This way, we know
for sure that the inquirer knows UDNA if they initiated it first.

> > 3. IDN(UTF8)/EDNS for clients & everywhere
> >     - IDNA clients begin to obsolete, new versions of application
software
> > comes with IDN(UTF8)/EDNS built in.
>
> Which will be slower than you might think..
>

Not much slower than what you should expect if you go for "ACE everywhere"

> >     - In case DNS servers are not upgraded yet, clients will continue to
> use
> > the IDNA client, thereby also creating an incentive for server side to
> > upgrade
>
> OK.. How does my client tell that a DNS server in Zimbabwe is or is not
> upgraded?  Remember - DNS is *distributed*, and just because YOUR DNS
> server is upgraded doesn't mean that the one actually providing an
> authoritative answer has been upgraded...

First of all, the EDNS migration path will be expected.  Then, if the
Zimbabwe DNS operator wishes to offer IDNs, they would upgrade to IDN/EDNS.
If they do not provide IDN resolution service, then IDNs wont work anyway.

> >     - Other applications/servers upgrade to IDN(UTF8)/EDNS to accept
UTF8
> > domains, administrators might continue to see ACE leakage if the client
> > still uses IDNA to send requests, but UDNA requests will be managed in
> UTF8
>
> It's this "might continue to see ACE leakage" that's a major problem...

Clients that do not upgrade will eventually be obsoleted.  At least we know
that.  Also, if the admin is so desparate to know what the ACE is, s/he
could always use other conversion tools to view it.  But as the core app,
ACE need not be done.  That's the beauty of ACE, isnt it?

> > 4. IDN(UTF8)/EDNS
> >     - Slowly but certainly, we will move towards full UTF8 because
> > administrators will likely want to deal with UTF8 than ACE, especially
for
> > say those admins in China who are dealing with IDNs frequently.  Added
to
> > the urge from the user community to upgrade the server as they upgrade
> their
> > clients to IDN(UTF8)/EDNS.
>
> Just remember the pain felt by *both* 50%s when 50% of the users had
> MIME-aware MUAs, and 50% didnt....

Precisely why we should plan ahead better this time.

Edmon

From: Keith Moore <[email-masked]>
To: "Edmon Chung" <[email-masked]>
cc: [email-masked], "John Stracke" <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Tue, 02 Apr 2002 11:16:09 -0500
Sender: [email-masked]

> That is, if we start thinking about "8+ everywhere" instead of "ACE
> everywhere" scenario, then how we can get there from here. That is the
> question for the IDN group I believe.

I cannot imagine that the IDN group is competent to decide how email
(or any other specific application) should migrate.  

Keith

To: Edmon Chung <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
From: [email-masked]
Date: Tue, 02 Apr 2002 11:09:34 -0500
Sender: [email-masked]

--==_Exmh_569342384P
Content-Type: text/plain; charset=us-ascii

(trimming main IETF list off the cc:)

On Tue, 02 Apr 2002 10:29:00 EST, Edmon Chung said:
> 1. IDNA clients
>     - the client does ACE and display it properly for the user
>     - servers (including DNS and mail and http, etc.) does NOT upgrade for
> ACE.  Even administration side is kept as is, allowing leakage
>     - IDNA clients MUST also be able to decode IDN(UTF8)/EDNS responses

This makes the rather rash assumption that the clients will be upgraded
to support a *future* feature.  You're also assuming that there will be
a way to test/debug IDN/EDNS responses when there are none actually being
seen "in the wild".

Re-work the model, assuming that (a) 50% of the people won't upgrade until
the new feature is *usable*, (b) 10% of the clients *wont* upgrade, and
(c) there will be at least 2 or 3 show-stopper bugs found in the EDNS
handling when it starts being widely used.

You'll never get widespread deployment of a feature that is both not usable
and untested (because there's no use of it).

> 2. IDN(UTF8)/EDNS DNS Server upgrade
>     - DNS servers will now accept both ACE (as usual) and IDN(UTF8)/EDNS
> requests

Are there any hidden gotchas here?  Are there any changes needed to BIND
to allow it to process a UTF8 zone file?  Are there any "bad news" characters
that might cause parsing problems?

Again - what happens if not everybody upgrades?

>     - ACE requests will yield Both an ACE response + an IDN(UTF8)/EDNS
> response

Here there be dragons.  Do a 'dig aol.com mx', and ask yourself what
happens to performance if that doesn't fit into one DNS packet anymore,
and as a result a resolver has to drop back to TCP (instead of one packet
each way, you're now up to a 3-packet handshake, the data, and the FIN
packets) - and note that the TTL on the AOL.COM SOA is 3600....

What happens to an ACE-based client that receives an EDNS response that it's
not able to parse?

>     - IDNA clients continue to send in ACE but will start to see UDNA
> responses

This is a can of worms - how does the server know for sure that the client
is upgraded and is able to parse a UDNA response?  Remember - the client
is probably *NOT* asking directly.  If I've upgraded example.com's DNS
to be UDNA ready, how do I know it's safe to send a UDNA response to a query
from foo-bar.com (since it's quite possible that the *USER* has upgraded,
but the sysadmins have NOT upgraded the DNS that's recursing the query for
the user....

> 3. IDN(UTF8)/EDNS for clients & everywhere
>     - IDNA clients begin to obsolete, new versions of application software
> comes with IDN(UTF8)/EDNS built in.

Which will be slower than you might think..

>     - In case DNS servers are not upgraded yet, clients will continue to use
> the IDNA client, thereby also creating an incentive for server side to
> upgrade

OK.. How does my client tell that a DNS server in Zimbabwe is or is not
upgraded?  Remember - DNS is *distributed*, and just because YOUR DNS
server is upgraded doesn't mean that the one actually providing an
authoritative answer has been upgraded...

>     - Other applications/servers upgrade to IDN(UTF8)/EDNS to accept UTF8
> domains, administrators might continue to see ACE leakage if the client
> still uses IDNA to send requests, but UDNA requests will be managed in UTF8

It's this "might continue to see ACE leakage" that's a major problem...

> 4. IDN(UTF8)/EDNS
>     - Slowly but certainly, we will move towards full UTF8 because
> administrators will likely want to deal with UTF8 than ACE, especially for
> say those admins in China who are dealing with IDNs frequently.  Added to
> the urge from the user community to upgrade the server as they upgrade their
> clients to IDN(UTF8)/EDNS.

Just remember the pain felt by *both* 50%s when 50% of the users had
MIME-aware MUAs, and 50% didnt....

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

--==_Exmh_569342384P
Content-Type: application/pgp-signature

--==_Exmh_569342384P--

To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
From: "John Stracke" <[email-masked]>
Date: Tue, 2 Apr 2002 09:54:32 -0500
Sender: [email-masked]

D. J. Bernstein wrote:

>It is not reasonable to demand that this
>7-bit garbage be allowed to remain on the network forever, at the
>expense of all future implementors and users.

OK, you're obviously in a better position than I am to appreciate the cost 
to implementors.  But users? Suppose you could streamline qmail by 
dropping support for talking to 7-bit servers.  How do you convince a 
postmaster to install it? The bandwidth cost of having to use 7-bit 
encoding is real, but small; the cost of losing connectivity to sites is 
potentially high.  Worse, it's almost impossible to put an upper bound on 
the cost of lost connectivity.  For a company, it could mean losing 
messages (and sales) from important customers; businesses have failed over 
such things.

For commercial implementors, it's a no-brainer: dropping 7-bit support 
will lose customers, so don't do it.  For open-source implementors such as 
yourself, naturally, things are less clear-cut.

/==============================================================\
|John Stracke                    |Principal Engineer           |
[email-masked]   |Incentive Systems, Inc.      |
|http://www.incentivesystems.com |My opinions are my own.      |
|==============================================================|
|"This [OLE] isn't an API, this is a shaggy dog story in C!" --|
|Unknown                                                       |
\==============================================================/

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Tue, 02 Apr 2002 08:53:56 -0500
Sender: [email-masked]

> It is reasonable to demand time to upgrade, so that there are no
> interoperability problems. It is not reasonable to demand that this
> 7-bit garbage be allowed to remain on the network forever, at the
> expense of all future implementors and users.

forever is a long time.  and yet, even if all use of quoted-printable
were to cease today, there would still be a need for new MUAs to be
able to read old messages stored on servers for many more years, perhaps
decades.

using raw utf-8 in message headers wouldn't replace 2047. instead,
it would add yet another format that MUAs are expected to parse
and display. it would not reduce complexity, it would add to it.

Keith

Date: Tue, 2 Apr 2002 10:19:45 +0200
From: Keld =?iso-8859-1?Q?J=F8rn?= Simonsen <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

On Tue, Apr 02, 2002 at 08:08:40AM -0000, D. J. Bernstein wrote:
> 
> By the way, in case anyone thinks that my sarcastic ``7 bits forever!''
> is an exaggeration, here's an old quote from Mark Crispin: ``7-bit MTA's
> will never go away. Ever. Mail goes on many paths besides IP, some of
> which are forever 7-bits. You will always have to solve the problem of
> getting your 8-bit mail into 7-bits for some 7-bit MTA.''
> 
> It is reasonable to demand time to upgrade, so that there are no
> interoperability problems. It is not reasonable to demand that this
> 7-bit garbage be allowed to remain on the network forever, at the
> expense of all future implementors and users.

Why change? It works as designed. If we change this, we risk
fragmenting the net. Yes, it costs some more bits when 7-bit
downgrading is used, but that is not a general cost for
normal email use. And it costs some maintenance and some
space in MTAs, but that is not much in total.

keld

Date: Tue, 02 Apr 2002 00:18:42 -0800
To: "D. J. Bernstein" <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: [idn] Re: 7 bits forever!
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

At 08:08 AM 4/2/2002 +0000, D. J. Bernstein wrote:
>By the way, in case anyone thinks that my sarcastic ``7 bits forever!''
>is an exaggeration, here's an old quote from Mark Crispin: ``7-bit MTA's
>will never go away. Ever. Mail goes on many paths besides IP, some of
>which are forever 7-bits. You will always have to solve the problem of

the word sarcasm is about tone, not truth.

in fact, there is quite a bit of experience that shows just how long 
established systems and services do stay around.

the difference between 20 years and forever is not worth measuring.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

Date: 2 Apr 2002 08:08:40 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

By the way, in case anyone thinks that my sarcastic ``7 bits forever!''
is an exaggeration, here's an old quote from Mark Crispin: ``7-bit MTA's
will never go away. Ever. Mail goes on many paths besides IP, some of
which are forever 7-bits. You will always have to solve the problem of
getting your 8-bit mail into 7-bits for some 7-bit MTA.''

It is reasonable to demand time to upgrade, so that there are no
interoperability problems. It is not reasonable to demand that this
7-bit garbage be allowed to remain on the network forever, at the
expense of all future implementors and users.

> > Sorry to bother you with the facts, but the 8BITMIME requirement that
> > you're mindlessly imitating is flagrantly violated in practice by many
> > MTAs, including most versions of sendmail.
> qmail announces 8BITMIME support without actually implementing it, too.

Of course. Announcing 8BITMIME in the SMTP server benefits the users.
Deliberately ignoring the 8BITMIME requirements in the SMTP client also
benefits the users.

I thought it was clear that I was criticizing 8BITMIME, not sendmail's
violation of it. Essentially everything that sendmail has done wrong
with 8-bit support has complied with the relevant IETF specifications,
and essentially everything that sendmail has done right with 8-bit
support has violated the relevant IETF specifications.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

From: Keith Moore <[email-masked]>
To: Florian Weimer <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Tue, 02 Apr 2002 02:30:05 -0500
Sender: [email-masked]

> I agree.  8BITMIME has largely failed (strictly speaking, announcing
> 8BITMIME requires body rewrite features in mail relay software).  We
> should not introduce similar functionality, as an RFC for that will
> be ignored, implemented incorrectly, or not implemented due to the
> involved complexity.

It's a different world now.  At the time 8BITMIME was defined few MUAs 
supported MIME, but many could display 8bit text in message bodies
as long as it wasn't MIME encoded, and sender and receiver happened 
to use the same character set.  And given the demographics of the 
Internet at that time, there was at least a fair chance that both
sender and receiver used iso-8859-1 (though many others were in use).

At the present time most MUAs support MIME encoding.  Essentially all
MUAs that support utf-8 also support MIME; thus they are able to decode 
2047 headers and use the proper character encoding for display. OTOH 
unlabeled 8bit text in message headers is not treated consistently
across MUAs, and there are many more character sets in wide use than when
8BITMIME was defined.  Though it is generally possible to distinguish
between several of the popular encodings even without tagging, it's 
probably the case that header text encoded in 2047 is more likely to 
work with existing MUAs than unencoded utf-8.

The real problem with 8BITMIME was that it essentially forced the
MTA to make a guess about whether the recipient's MUA supported MIME.
I agree that we don't want to do that again.  MTAs should be encouraged 
to handle relayed messages as transparently as possible.

However, the benefit in doing so is mostly for the long term.  It still 
won't permit us to use unencoded utf-8 in message headers in the near term. 

Keith

To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
From: Florian Weimer <[email-masked]>
Date: Tue, 02 Apr 2002 08:48:14 +0200
Lines: 20
Sender: [email-masked]

"D. J. Bernstein" <[email-masked]> writes:

> Dan Kohn writes:
>> a UTF8HEADERS ESMTP extension
>
> No! 8-bit support is not an option. 8-bit support is not something to be
> negotiated. 8-bit support is _required_.

I agree.  8BITMIME has largely failed (strictly speaking, announcing
8BITMIME requires body rewrite features in mail relay software).  We
should not introduce similar functionality, as an RFC for that will
be ignored, implemented incorrectly, or not implemented due to the
involved complexity.

> Sorry to bother you with the facts, but the 8BITMIME requirement that
> you're mindlessly imitating is flagrantly violated in practice by many
> MTAs, including most versions of sendmail.

qmail announces 8BITMIME support without actually implementing it,
too.  (This defect only affects mail relays, though.)

From: Keith Moore <[email-masked]>
To: "Dan Kohn" <[email-masked]>
cc: [email-masked], "Einar Stefferud" <[email-masked]>, [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever! 
Date: Tue, 02 Apr 2002 00:20:17 -0500
Sender: [email-masked]

> I believe that Einar could be most easily satisfied with something along
> the lines of a UTF8HEADERS ESMTP extension, which would specify both
> that 8 bit character are permitted in the header and that those
> characters MUST be interpreted as UTF-8.  

there are lots of problems with this idea.  for instance, there's no 
way for SMTP to know whether the recipient's MUA can deal with 
utf-8 even in 2047 encoded form, to say nothing about unencoded form.   
and it's not as simple as saying just use utf-8 in the message header -
there's also the question of which characters are legal and which aren't,
which ones are separator characters, which ones are white space, etc.

then consider that, except for the abliity to have non-ascii email
addresses, there's no functional advantage to having raw utf-8 in headers.
there might be a slight win to having a more efficient encoder, but 
MUAs would still have to implement both utf-8 and 2047 for many more years.

let's get the details of non-ascii email addresses worked out before
we start spewing utf-8 in random places in the message header.

Keith

Date: 2 Apr 2002 05:14:22 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: [idn] Re: 7 bits forever!
Sender: [email-masked]

Dan Kohn writes:
> a UTF8HEADERS ESMTP extension

No! 8-bit support is not an option. 8-bit support is not something to be
negotiated. 8-bit support is _required_. Do you want implementors in
twenty years to be dealing with 7-bit MTAs?

> The challenge is that any mail relay implementing this
> extension MUST be able to downconvert headers to 7-bit RFC 2047 [3]
> compliant encoded words if the next hop does not support UTF8HEADERS.

Sorry to bother you with the facts, but the 8BITMIME requirement that
you're mindlessly imitating is flagrantly violated in practice by many
MTAs, including most versions of sendmail. (All versions from 6.57 in
1993 until 8.7 in 1995; at least a few years after that if the popular 8
flag is set; I haven't checked the most recent versions.)

Furthermore, converting an 8-bit email address to 7 bits would destroy
interoperability. This isn't just a Q-P display failure; this is mail
delivered to the wrong address. Reliably undoing the conversion means
upgrading every mail program on the Internet, rather than just the 7-bit
programs that are actually at fault.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

content-class: urn:content-classes:message
Subject: RE: [idn] Re: 7 bits forever!
Date: Mon, 1 Apr 2002 17:31:28 -0800
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>
Cc: "Einar Stefferud" <[email-masked]>, <[email-masked]>, <[email-masked]>
Sender: [email-masked]

I believe that Einar could be most easily satisfied with something along
the lines of a UTF8HEADERS ESMTP extension, which would specify both
that 8 bit character are permitted in the header and that those
characters MUST be interpreted as UTF-8.  It would be expected that this
extension would normally be used in conjunction with 8BITMIME [1] or
BINARYMIME [2].  The challenge is that any mail relay implementing this
extension MUST be able to downconvert headers to 7-bit RFC 2047 [3]
compliant encoded words if the next hop does not support UTF8HEADERS.

This would seem to be a middle ground between Dan's desire for
just-send-8 and other's demand that existing RFC 2821 [4] compliant
relays not be broken by adding support for UTF-8 headers.

This should be an easy RFC to write up.  If there is interest from
implementers, I would be happy to do so.  In any event, I would suggest
taking follow-up to ietf-smtp [5].

[1] http://www.ietf.org/rfc/rfc1652.txt
[2] http://www.ietf.org/rfc/rfc3030.txt
[3] http://www.ietf.org/rfc/rfc2047.txt
[4] http://www.ietf.org/rfc/rfc2821.txt
[5] http://www.imc.org/ietf-smtp/

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>
Essays announced on <mailto:[email-masked]> 

-----Original Message-----
From: Einar Stefferud [mailto:[email-masked]] 
Sent: Monday, April 01, 2002 11:47
To: [email-masked]; [email-masked]
Subject: Re: [idn] Re: 7 bits forever!

Hi John -- My thesis is that the problem lies now in the SMTP
standards, where the current "Send 8-bits" problem has always
resided.  The MIME working group could not solve that problem, and so
did not, but it made mail work since its deployment, which was the
objective in the MIME WG.  inside the RFC822 Body there is only one
way to handle 8-bit data, and that is to tag it, and bag it.  Even if
you can transmit 8-bit data without encoding it in the body of an
SMTP message, it needs to be tagged and delimited in the body.

MIME Quoted-Printable was designed to be a work-around-damage
solution at the time, and was not design to remove the need for
handling 8-bit data in SMTP.

So, my only point is that it appears that some work needs to be done
in the SMTP standards, and beating up on the MIME WG does not compute
given this observation.

That is all...  Just trying to shift the focus of this thread to
where someone might think of a way to solve the current problem.

In the meantime, Thankfully, Mail still works, which was the MIME
objective for Quoted-Printable.

Cheers;-)....\Stef

At 12:41 PM -0500 4/1/02, John C Klensin wrote:
>Stef,
>
>Either I misunderstand your message, or we have a bit of a
>disconnect.
>
>The SMTP extension "8BITMIME" was specified fairly early in the
>game.  My impression is that it is fairly widely deployed and
>used, possibly to the extent that most non-ASCII traffic is
>moving that way; someone on this list (or the SMTP extensions
>one) may have some numbers.  Where it is deployed, MIME is still
>required to label ("tag") the data -- one can still not tell one
>sort of 8bit character from another, and there is, I think,
>still much more traffic on the network in ISO 8859-N variants,
>or IS 2022-based systems, or local character sets than there is
>UTF-8 encoding of Unicode -- but quoted-printable (or base64)
>are not, unless the mail store or mail retrieval implementation
>cannot deal with 8bit materials.
>
>       john
>
>
>--On Monday, 01 April, 2002 08:59 -0800 Einar Stefferud
><[email-masked]> wrote:
>
>  > Not to pick on Dan's particular message.
>  > It was just the last one to provide me with a handy response
>  > point.
>  >
>  > Looking back, the original 1990-1991 argument was between
>  > "Just send 8 bits" with no tagging of the content, vs "tagging
>  > and bagging" to deal with the fact that 8-bit at that time
>  > would break too many systems, and the problem of needing to
>  > tag the types of text characters that were contained in RFC822
>  > message bodies.
>  >
>  > But, as there were two problems to solve (8 bit SMTP ) and
>  > (tagging RFC822 content), the decision was made (after a very
>  > long and very heated argument) to split that problem according
>  > to the boundaries of SMTP and RFC822.
>  >
>  > In fact, the two problems were and still are totally distinct.
>  >
>  > The RFC822 solution group produced MIME with Quoted-Printable
>  > as its charter specified, to assure interworkability as soon
>  > as possible.
>  >
>  > The 8-bit folks did not do anything then or since to require
>  > 8-bit carriage.
>  >
>  > So, it is time for a working group to solve this problem.
>  >
>  > It is not going to help anyone to argue about the past, as in
>  > fact, it was very rational at the time, and one side of the
>  > problem was ignored as soon as quoted printable relieved the
>  > immediate pressure for an SMTP solution.
>  >
>  > So, now is a good time for some constructive discussion of
>  > ways to solve the charset problems, hopefully with some useful
>  > charset tagging tools for the non-RFC-822 parts of the SMTP
>  > envelope.  (But I do not mean to specify either the work or
>  > the results at this point.)
>  >
>  > It is indeed unfortunate the the DRUMS WG did not find a way
>  > to solve the 8-bit SMTP problem, but now is the time to get to
>  > it.  There should not be too much 7-bit SMTP code to fix by
>  > now.
>  >
>  > Cheers...\Stef
>  >
>  >
>  >
>  >
>  > At 8:35 AM +0000 4/1/02, D. J. Bernstein wrote:
>  >> Claus Faerber writes:
>  >>  > D. J. Bernstein <[email-masked]> schrieb/wrote:
>  >>  > > I'm not saying that Quoted-Printable had no short-term
>  >>  > > benefits for
>  > its
>  >>  > > short-term costs. I'm saying that, viewed from our
>  >>  > > long-term
>  > perspective
>  >>  > > eleven years later, the failure to require 8-bit
>  >>  > > transparency was an amazingly stupid decision.
>  >>  > From our present perspective, that's true. Back then, it
>  >>  > might have been the best solution.
>  >>
>  >> No. The failure to require 8-bit transparency was
>  >> inexcusable. Every approach that failed to require 8-bit
>  >> transparency could have been dramatically improved. Consider,
>  >> for example, these three approaches:
>  >>
>  >>    (1) Quoted-Printable;
>  >>    (2) Quoted-Printable plus ``you must handle unencoded
>  >>    8-bit data too''; (3) pure 8-bit without this 7-bit
>  >>    garbage.
>  >>
>  >> Whether or not you understand that #3 would have been better
>  >> than #2, surely you understand that #2 would have been vastly
>  >> better than #1.
>  >>
>  >>  > Further, remember that the first MIME standards date back
>  >>  > to 1992. Back then, Unicode was brand-new and UTF-8 only
>  >>  > came with the 2.0 version in 1996. Without UTF-8, you just
>  >>  > could not even think about using Unicode in message
>  >>  > headers; and without Unicode, you could not solve the
>  >>  > charset-labelling problem.
>  >>
>  >> Get your facts straight. First, UTF-8 was introduced years
>  >> before 1996, although under another name. Second, even
>  >> without knowing about UTF-8, people _were_ thinking about
>  >> Unicode in headers, and proposed several workable approaches.
>  >> Third, even without Unicode, there were several solutions to
>  >> the character-set labelling problem.
>  >>
>  >> Anyway, none of this is relevant to IETF's acceptance of
>  >> 7-bit garbage in 1991, and none of it is relevant to IETF's
>  >> acceptance of 7-bit garbage today.
>  >>
>  >>  > The movement towards UTF-8 everywhere is quite new.
>  >>
>  >> Again, get your facts straight. The ``movement'' started with
>  >> X-Open, Rob Pike, and Ken Thompson a decade ago. RFC 2277,
>  >> requiring UTF-8 support for all text on the Internet, is four
>  >> years old.
>  >>
>  >> ---D. J. Bernstein, Associate Professor, Department of
>  >> Mathematics, Statistics, and Computer Science, University of
>  >> Illinois at Chicago
>  >
>  >
>  >
>  >
>  >
>  >

-
This message was passed through [email-masked],
which
is a sublist of [email-masked]. Not all messages are passed.
Decisions on what to pass are made solely by Raffaele D'Albenzio.