Subject: Re: authenticating the source of mail
From: Dave Crocker <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: "Eric A. Hall" <[email-masked]>, Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Date: Mon, 20 May 2002 10:26:46 -0700
List-Id: <ietf-822.imc.org>

At 01:19 PM 5/20/2002 -0400, Keith Moore wrote:
> > Let's go through this slowly.
>
>I've got a better idea.

this thread has gotten quite long.

can someone explain what it is accomplishing?

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail 
Date: Mon, 20 May 2002 13:19:42 -0400
Sender: [email-masked]

> > I've seen several [] users who routinely post from multiple accounts
> > from different ISPs, using a variety of SMTP servers, but using the
> > same From address.
> 
> Let's go through this slowly.

I've got a better idea.  let's not go through this at all, because it's
simply a non-starter.  it penalizes the wrong people, and it doesn't 
accomplish anything useful.

Keith

Date: Mon, 20 May 2002 11:17:52 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> I've seen several [] users who routinely post from multiple accounts
> from different ISPs, using a variety of SMTP servers, but using the
> same From address.

Let's go through this slowly.

First of all, the domain in the reverse-path is what will get checked in
these cases. If the users or admins of that mail domain don't want to
limit the hosts that mail can be sent from, they have the obvious choice
of not using MS RRs. In this situation, nothing will change. The users can
still send mail from wherever they want, and spammers and trojans can
continue to use that mail domain in the reverse-path of the junk they
generate.

Orgs that want to limit the use of their mail domain to certain specific
hosts and/or networks can essentially volunteer an ingress ACL for remote
mail routers to use. This may be because company policy requires that all
mail go through a content-analyzer/modifier before it goes public, or it
may be because the org doesn't want anybody misrepresenting themselves as
an employee of that organization, or it may be because hotmail and aol
email is only supposed to come from some servers, or whatever.

There are already a bunch of ways that mobile users in these networks can
be supported. There are the MS flag types, submission servers, a subdomain
which doesn't run MS, dynamic DNS updates (via a login script or mail
client tweak, if necessary), webmail access to their home server, and
several other options. Heck, they can use their mobile email for the
reverse-path, with the From and/or Reply-To header fields pointing back to
the org's mail domain if nothing else.

In my case, I can use any and all of those and will be able to do this
without any difficulty, even though I frequently use remote systems to
monitor and send mail. As such, any mail from ehsco.com can be rejected if
it doesn't come from the hosts I list as authorized by me to send mail for
my domain. After a while, I would hope that successfully matching an MS RR
would act as a positive indicator against weighting filters, perhaps to
the point of being used as a whitelist-bypass (right now, we have ~40:1
ratio of negative triggers to positive triggers, and here's another
positive trigger).

So the real question remains to be ~what other methods can we use so that
more mail domains can take advantage of this, thereby improving the
chances of its use? What methods will be palatable to you? If you have no
interest in this, you do not have to opt-in, but that will not greatly
diminish the benefits to the orgs who can take advantage of it.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: Paul Smith <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail 
Date: Mon, 20 May 2002 10:41:19 -0400
Sender: [email-masked]

> >I don't think *you* understand it very well either - certainly
> >not well enough to be trying to explaining it to others.
> 
> Well, I understand that the proposed 'MS' records would 'protect' the
> domain name from abuse rather than the mail server, and that is the
> misapprehension that I was trying to correct.

whatever.  using MS records in this way was *not* my idea.

Keith

Date: Mon, 20 May 2002 15:37:52 +0100
To: Keith Moore <[email-masked]>
From: Paul Smith <[email-masked]>
Subject: Re: authenticating the source of mail
Cc: [email-masked]
Sender: [email-masked]

At 08:40 20/05/2002 -0400, you wrote:
> > I don't think you understand Keith's idea very well.

Sorry, I meant Eric's idea..

>I don't think *you* understand it very well either - certainly
>not well enough to be trying to explaining it to others.

Well, I understand that the proposed 'MS' records would 'protect' the 
domain name from abuse rather than the mail server, and that is the 
misapprehension that I was trying to correct.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

From: Keith Moore <[email-masked]>
To: Paul Smith <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail 
Date: Mon, 20 May 2002 08:40:52 -0400
Sender: [email-masked]

> I don't think you understand Keith's idea very well.

I don't think *you* understand it very well either - certainly
not well enough to be trying to explaining it to others.

Keith

From: Keith Moore <[email-masked]>
To: Paul Smith <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail 
Date: Mon, 20 May 2002 08:39:19 -0400
Sender: [email-masked]

> > > > 1. it's a non-starter to force senders to submit to/relay through any
> > > >    particular host or hosts.  people can and do send mail from
> > > >    anywhere, and this is essential functionality.
> > >
> > > Providing a list of authorized senders is a sender-side choice. As such,
> > > this is an elective system and has to be recognized as one.
> >
> >right.  but it's an elective system that has significant barriers to
> >actually being used.
> 
> Why? The only reasonable argument I've seen against it is your 'wanting to
> send mail through any server' argument. For 99% or more users, this won't
> be an issue.

I've seen several unsophisticated users who routinely post from multiple
accounts from different ISPs, using a variety of SMTP servers, but using
the same From address.  I've seen several slighlty more sophisticated users
who maintain multiple mailboxes and post from multiple From addresses
via the same ISP and SMTP server.  

Even if these people represent less than 1% of the population, it's still
valuable functionality.  Perhaps more to the point, it's also a set of people 
whose mail cannot be disregarded.

Keith

Date: Mon, 20 May 2002 10:05:07 +0100
To: Keith Moore <[email-masked]>
From: Paul Smith <[email-masked]>
Subject: Re: authenticating the source of mail
Cc: [email-masked]
Sender: [email-masked]

At 14:59 17/05/2002 -0400, Keith Moore wrote:

> > > 1. it's a non-starter to force senders to submit to/relay through any
> > >    particular host or hosts.  people can and do send mail from
> > >    anywhere, and this is essential functionality.
> >
> > Providing a list of authorized senders is a sender-side choice. As such,
> > this is an elective system and has to be recognized as one.
>
>right.  but it's an elective system that has significant barriers to
>actually being used.

Why? The only reasonable argument I've seen against it is your 'wanting to 
send mail through any server' argument. For 99% or more users, this won't 
be an issue.

> > BUT, providing this kind of information is as beneficial to the sender as
> > it is to the recipient.
>
>right, but it only works for senders who always use the same relays.

As the vast majority of people do.

> > So rather than looking at this in terms of ~"can I reject mail from
> > senders without MS RRs?" (the answer to that is "no"), the appropriate way
> > to look at this is ~"can the owner of a domain use this to prevent
> > forgeries from their domain" and "can the recipients use this as a
> > supplemental method for killing spam" (the answer to both of those
> > questions is "yes").
>
>the answer to the first of the questions is "no, not in general".
>the answer to the second question is "yes, but it won't make much
>of a difference unless most of the sources of spam use it"

It would make a difference if the *faked* sources of spam use it. Eg if 
Hotmail use it, then spammers will no longer be able to use @hotmail.com 
addresses on their spam, because they're not relaying through Hotmail's 
servers.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Mon, 20 May 2002 09:57:53 +0100
To: Keith Moore <[email-masked]>
From: Paul Smith <[email-masked]>
Subject: Re: authenticating the source of mail
Cc: [email-masked]
Sender: [email-masked]

At 17:55 17/05/2002 -0400, Keith Moore wrote:
> > Let's apply some analysis here. Would this fix be any worse than the
> > policy changes you instituted as a result of getting listed in ORBS?
> > In retrospect, was that change so dramatic that you would reverse it,
> > given the chance to do so?
>
>ORBS just tried to close down open relays; they didn't try to force
>users to submit mail from specific hosts.

Do you think spam is a big problem? If so, do you think it's necessary to 
do something about it?
Do you think it's a big enough problem to warrant some inconvenience to a 
few legitimate users in order to confer a benefit to all email users 
(except spammers) everywhere?

Can you think of a solution to the problem which would work better for 
everyone, and be less inconvenient to you?

If we run with the idea of some form of email sender authentication, then 
there are possible solutions to your particular problem - eg we could 
specify that if the MS record points to '0.0.0.0' then any host is allowed 
to send mail with that domain as the sender address (and you run the risk 
of spammers faking your address on their spam, but you seem to be happy 
with that idea)

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Mon, 20 May 2002 09:48:37 +0100
To: Keith Moore <[email-masked]>
From: Paul Smith <[email-masked]>
Subject: Re: authenticating the source of mail
Cc: [email-masked]
Sender: [email-masked]

At 17:34 17/05/2002 -0400, Keith Moore wrote:
> > > also for spam filtering/prevention/remediation it's not terribly
> > > useful to authenticate the sender's email address.  what you really
> > > want is to
> >
> > this sentence drops off in mid-stream
>
>yeah, I decided it was too much trouble to explain it.

Hmm. If you know the answer to the spam problem, can you share it with the 
rest of us?

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Mon, 20 May 2002 09:35:06 +0100
To: Arnt Gulbrandsen <[email-masked]>
From: Paul Smith <[email-masked]>
Subject: Re: authenticating the source of mail
Cc: [email-masked]
Sender: [email-masked]

At 13:35 19/05/2002 +0200, Arnt Gulbrandsen wrote:
>"Eric A. Hall" <[email-masked]>
> > Any comments on this approach?
>
>It sounds like the first users of this would block some legitimate mail
>and no spam.
>
>Some legitimate mail, because all mail-blocking schemes do. No spam,
>because neither spammers like Postmastergeneral nor clueless sites like
>those Korean schools will add MS RRs in their zone files.

I don't think you understand Keith's idea very well.

What would happen in these situations is that the spammers will still be 
able to send spam through their own mail servers, but they won't be able to 
use [email-masked]' or [email-masked]' style email addresses. If they did, then 
the receiving mail server would do a lookup for the MS/MX records to see if 
that mail server is allowed to send mail from hotmail.com or yahoo.com 
domains. Since it wouldn't be, your mail server would 
remove/quarantine/filter/etc those messages.

If the spammers send spam through their own mail servers using imaginary 
domains a mail server could theoretically detect that. You could use normal 
mail filtering to filter mail that spammers send through their own mail 
servers using their own domain names.

I think it's quite a good idea, and I wish people would run with the idea 
to see how far it goes, rather than just being pessimistic about it.

There does need to be some way of authenticating the source of email to 
start reducing spam which, to be honest, has the possibility of killing off 
email. We have customers who have reduced their use of email because of all 
the spam they're getting.

This proposed solution seems like quite a low cost way of implementing it, 
within existing infrastructure.

No one would force an MTA to check these records.

>This, MS probably wouldn't succeed: Noone would be the first to use it,
>and if noone uses MS, noone will add it to their zone files (except those
>few admins who added LOC records).

Actually, I reckon quite a few people would be 'the first' to implement the 
software to check for MS records. Content filtering companies, firewall 
companies etc would be glad to have some way of doing this type of 
authentication.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Sun, 19 May 2002 11:25:02 -0500
From: "Eric A. Hall" <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
CC: Andrew Church <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> One day my IP connectivity to the main office is broken, but I can
> talk to most of the world. No problemo, I just comment out the
> "smarthost" stuff in my server's sendmail.cf and see that my
> outgoing mail works again.
>
> Next day the main office is back on the net, but but I forget to
> reenable the smarthost setting, and noone discovers it until the main
> office, confident that everyone's using the central smarthost, adds an
> MS RR and some of my mail starts bouncing.

Would the domain-wide PTR mapping work for this branch office setup? Your
mail appears to be coming from a fixed network associated with your
$cityname branch office, so you could theoretically just point to that
subdomain and be done with it.

There are certainly going to be exception cases, which cannot be designed
for. The best choice for those networks is not to use MS RRs at all. If
the mail network doesn't use a set of authorized relay servers, then there
is no reason to list anything in MS. But in your example, it seems that
the domain-wide PTR would work. Would it? If not, would something else
work as well?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: 19 May 2002 11:25:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: text/html an abomination
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Dave Crocker)  wrote on 06.05.02 in <[email-masked]>:

> At 11:43 PM 5/5/2002 -0400, Keith Moore wrote:
> >that wouldn't bother me a bit if such messages didn't have so much
> >gratuitous markup.  in the vast majority of html messages I see, the html
> >contains no more
> >information than could have been conveyed in plain text, but has so much
> >cruft in it that it essentially cannot be read as text.
>
> When people first get access to mechanisms for formatting, fonts, etc.,
> they tend to overuse it.  Over time, they learn to be more judicious.

Over how much time? At least 99% of the HTML formatting in mail that I see  
*today* is not only injudicious, but flat out gruesome.

> Formatting that is done well aids the communication, rather than detracting
> from it, and adds another dimension to the communication.
>
> Let's not damn the capability just because some people have not yet learned
> to use it.

"Some people" == "Practically everyone who does use it". I've seen maybe  
one or two exceptions. (And those were strictly not necessary, either.)

> The idea that email content should be required to have less visual control
> than a web page or a document simply doesn't make any sense.

Makes sense to me, given what people actually do with it.

MfG Kai

Date: Sun, 19 May 2002 15:21:53 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Andrew Church <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Andrew Church <[email-masked]>
> >Some legitimate mail, because all mail-blocking schemes do.
> 
>      Can you give me an example?

Why? Mistakes happen, we all know that.

>      My primary goal with this RR is not to block spam per se, but to allow
> domain owners to prevent mail with forged headers (usually, but not
> necessarily, spam) containing their domain name from being sent.  In other
> words, I want to keep Joe Spammer from using openrelay.example.com to send
> mail that appears to be from achurch.org.  Thus, all I need is support in
> SMTP clients; if clueless sites don't want to add MS RRs, their loss (but
> not mine).

Here's a scenario where that breaks, modelled on something that happened
to me last month.

    I work at the small $city_name office of a company with many locations
    throughout the world. One day my IP connectivity to the main office is
    broken, but I can talk to most of the world. No problemo, I just
    comment out the "smarthost" stuff in my server's sendmail.cf and see
    that my outgoing mail works again.

    Next day the main office is back on the net, but but I forget to
    reenable the smarthost setting, and noone discovers it until the main
    office, confident that everyone's using the central smarthost, adds an
    MS RR and some of my mail starts bouncing.

Or a variation:

    I work at the ... and I see that my mail apparently works again.
    Because on that day, nothing in the mail queue went to an MS-testing
    site and I wasn't aware that the main office had added an MS RR.

    I then forget about it. Months later I discover that an important
    customer has been MS-testing and silently discarding mail.

--Arnt

Date: Sun, 19 May 2002 13:35:20 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

"Eric A. Hall" <[email-masked]>
> Any comments on this approach?

It sounds like the first users of this would block some legitimate mail
and no spam.

Some legitimate mail, because all mail-blocking schemes do. No spam,
because neither spammers like Postmastergeneral nor clueless sites like
those Korean schools will add MS RRs in their zone files.

This, MS probably wouldn't succeed: Noone would be the first to use it,
and if noone uses MS, noone will add it to their zone files (except those
few admins who added LOC records).

--Arnt

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: authenticating the source of mail 
Date: Fri, 17 May 2002 19:54:38 -0400
Sender: [email-masked]

> > we didn't have an open relay.
> 
> I won't argue this, but <[email-masked]> indicates
> otherwise, and the nature of the ORBS probes essentially required a
> message to be relayed. If you say that these facts are not representative
> of the truth, so be it.

ORBS did require a single message to be relayed.  But if you actually
tried using the server to send spam, you'd find that it would only 
relay to a small number of non-local recipients per day - few enough
that it wasn't worth the trouble.  It was a stopgap measure that was 
intended to keep us from being an aid to spammers and yet to bide time 
until we could get users to authenticate their submissions.  And it served
its intended purpose.  Unfortunately, it didn't keep ORBS from blacklisting
us, and it didn't keep naive/stupid sysadmins from using the ORBS to
prevnet us from sending mail to them, and it didn't keep technically
less astute people who were higher in the local food chain from insisting
that we screw users who were using us as a submit server and prevent 
all nonlocal mail relaying.

So perhaps this isn't "force" but it's damn close.

Keith

Date: Fri, 17 May 2002 16:48:25 -0700 (PDT)
From: [email-masked]
Subject: Re: authenticating the source of mail
To: "Eric A. Hall" <[email-masked]>
Cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> > we didn't have an open relay.

> I won't argue this, but <[email-masked]> indicates
> otherwise, and the nature of the ORBS probes essentially required a
> message to be relayed. If you say that these facts are not representative
> of the truth, so be it.

The ORBS probes are widely known to be bogus. And you continue to demonstrate
an almost total lack of knowledge of the actual operational issues of email.

				Ned

Date: Fri, 17 May 2002 18:39:51 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> we didn't have an open relay.

I won't argue this, but <[email-masked]> indicates
otherwise, and the nature of the ORBS probes essentially required a
message to be relayed. If you say that these facts are not representative
of the truth, so be it.

> that's what I meant by naivete - a solution that forces everyone to
> incur costs and give up useful functionality without accomplishing
> anything useful.

It's voluntary and forces nothing. There is no NECESSARY loss of
functionality (multiple workarounds of varying levels are available, and
are only required if you willingly choose to participate in the elective
and voluntar system). And it has the useful property that organizations
can identify the authorized servers for the mail domains under their
control, allowing recipients to reject blatant forgeries.

Wow, every freaking element of your comment is completely wrong.

I'll write up the I-D and you can piss on that. There is nothing more to
be gained from this round.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: authenticating the source of mail 
Date: Fri, 17 May 2002 19:08:04 -0400
Sender: [email-masked]

> You closed your open relay. Naive or no, it certainly worked for you.

we didn't have an open relay.  the ORBS people forced us to take away
valuable functionality from our users, with no purpose whatsoever.

meanwhile, the spammers keep spamming. ORBS hasn't even made a dent.

that's what I meant by naivete - a solution that forces everyone to
incur costs and give up useful functionality without accomplishing
anything useful.

Keith

Date: Fri, 17 May 2002 17:48:51 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> I'll grant that the ORBS approach suffers from the same kind of
> naivete.

You closed your open relay. Naive or no, it certainly worked for you.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: authenticating the source of mail 
Date: Fri, 17 May 2002 18:42:10 -0400
Sender: [email-masked]

> > > Let's apply some analysis here. Would this fix be any worse than the
> > > policy changes you instituted as a result of getting listed in ORBS?
> > > In retrospect, was that change so dramatic that you would reverse it,
> > > given the chance to do so?
> >
> > ORBS just tried to close down open relays; they didn't try to force
> > users to submit mail from specific hosts.
> 
> First of all, you didn't answer the question.

that's because your question misses the point. 
 
> Secondly, your non-answer is somewhat misleading, since ORBS et al are
> very much intent on preventing submissions from specific hosts, which is
> in the same general category as this approach (secondary difference is
> that this one is voluntary).

I'll grant that the ORBS approach suffers from the same kind of naivete.

Keith

Date: Fri, 17 May 2002 17:10:46 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> > Let's apply some analysis here. Would this fix be any worse than the
> > policy changes you instituted as a result of getting listed in ORBS?
> > In retrospect, was that change so dramatic that you would reverse it,
> > given the chance to do so?
> 
> ORBS just tried to close down open relays; they didn't try to force
> users to submit mail from specific hosts.

First of all, you didn't answer the question.

Secondly, your non-answer is somewhat misleading, since ORBS et al are
very much intent on preventing submissions from specific hosts, which is
in the same general category as this approach (secondary difference is
that this one is voluntary).

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: authenticating the source of mail 
Date: Fri, 17 May 2002 17:55:22 -0400
Sender: [email-masked]

> > > So setup cs.utk.edu with a dynamic DNS agent that accepts MS updates
> > > for wherever you happen to be.
> >
> > NO, damnit.  Just because people submit mail through arbitrary servers
> > does not mean they do so from a nomadic host.
> 
> So basically you don't like this because it requires ongoing maintenance
> which conflicts with your overly liberal network policy. Swell.

There are far more things in heaven and email, Eric, than are dreamt of in 
your philsophy.
 
> Let's apply some analysis here. Would this fix be any worse than the
> policy changes you instituted as a result of getting listed in ORBS? 
> In retrospect, was that change so dramatic that you would reverse it, 
> given the chance to do so?

ORBS just tried to close down open relays; they didn't try to force
users to submit mail from specific hosts.

Keith

Date: Fri, 17 May 2002 16:50:42 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> > So setup cs.utk.edu with a dynamic DNS agent that accepts MS updates
> > for wherever you happen to be.
> 
> NO, damnit.  Just because people submit mail through arbitrary servers
> does not mean they do so from a nomadic host.

So basically you don't like this because it requires ongoing maintenance
which conflicts with your overly liberal network policy. Swell.

Let's apply some analysis here. Would this fix be any worse than the
policy changes you instituted as a result of getting listed in ORBS? In
retrospect, was that change so dramatic that you would reverse it, given
the chance to do so?

Can you think of another fix that would be easy for you?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: authenticating the source of mail 
Date: Fri, 17 May 2002 17:34:13 -0400
Sender: [email-masked]

> Keith Moore wrote:
> 
> > you're missing the point.  I might send mail from *anywhere*, from
> > any host or ISP I happen to be using at the moment.
> 
> So setup cs.utk.edu with a dynamic DNS agent that accepts MS updates for
> wherever you happen to be. 

NO, damnit.  Just because people submit mail through arbitrary servers
does not mean they do so from a nomadic host.

> > also for spam filtering/prevention/remediation it's not terribly
> > useful to authenticate the sender's email address.  what you really
> > want is to
> 
> this sentence drops off in mid-stream

yeah, I decided it was too much trouble to explain it.  

Keith

Date: Fri, 17 May 2002 14:47:53 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> you're missing the point.  I might send mail from *anywhere*, from
> any host or ISP I happen to be using at the moment.

So setup cs.utk.edu with a dynamic DNS agent that accepts MS updates for
wherever you happen to be. If this is too much data, setup a mail domain
for moore.cs.utk.edu and let the rest of us get on with filtering the
mail. I mean, for 99% of the mail domains, there will be little or no
extra effort. The remaining 1% of you can bend a little, maybe?

> also for spam filtering/prevention/remediation it's not terribly
> useful to authenticate the sender's email address.  what you really
> want is to

this sentence drops off in mid-stream

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: authenticating the source of mail 
Date: Fri, 17 May 2002 15:28:42 -0400
Sender: [email-masked]

> Keith Moore wrote:
> 
> > > providing this kind of information is as beneficial to the sender
> > > as it is to the recipient.
> > 
> > right, but it only works for senders who always use the same relays.
> 
> I tried to account for that with the domain flag. An organization can
> reference any number of individual hosts, or it can reference an entire
> domain of hosts as valid senders if it so chooses:

you're missing the point.  I might send mail from *anywhere*, from
any host or ISP I happen to be using at the moment.  the cs.utk.edu
domain alone represents such a diverse and well-travelled set of people
that it would not be feasible to list the hosts (all over the world)
which might accept their mail submissions and relay them.

also for spam filtering/prevention/remediation it's not terribly useful
to authenticate the sender's email address.  what you really want is to 

Keith

Date: Fri, 17 May 2002 14:11:15 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> > providing this kind of information is as beneficial to the sender
> > as it is to the recipient.
> 
> right, but it only works for senders who always use the same relays.

I tried to account for that with the domain flag. An organization can
reference any number of individual hosts, or it can reference an entire
domain of hosts as valid senders if it so chooses:

| Any and all hosts in the mail.example.com domain are authorized relay
| servers for example.com:
|
|  [Answer Section]
|   example.com.           MS   1   mail.example.com.
|
|  [Additional-Data Section]
|   <empty>
|
| [the PTR for the current mail session should be examined, and if it
| resides within the specified domain, then it is authorized]

In your case, that flag could be applied to cs.utk.edu. Since your host
(astro.cs.utk.edu) is within that domain, it would be accepted as an
authorized sender.

If the host is in another domain, the remote domain can als be referenced,
either as a host-specific entry or a domain-wide entry.

What situation are you thinking about which would not work with this?

> > the appropriate way to look at this is ~"can the owner of a domain
> > use this to prevent forgeries from their domain" and "can the
> > recipients use this as a supplemental method for killing spam"
> > (the answer to both of those questions is "yes").
> 
> the answer to the first of the questions is "no, not in general".

Why not? Even though this would be an elective system, it would be
effective towards this purpose when it was used. Any reduction is some
reduction.

> the answer to the second question is "yes, but it won't make much
> of a difference unless most of the sources of spam use it"

Again, the principle benefit from validation is to make the remaining
evidence more actionable.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: authenticating the source of mail 
Date: Fri, 17 May 2002 14:59:45 -0400
Sender: [email-masked]

> > 1. it's a non-starter to force senders to submit to/relay through any
> >    particular host or hosts.  people can and do send mail from
> >    anywhere, and this is essential functionality.
> 
> Providing a list of authorized senders is a sender-side choice. As such,
> this is an elective system and has to be recognized as one. 

right.  but it's an elective system that has significant barriers to
actually being used.

> BUT, providing this kind of information is as beneficial to the sender as
> it is to the recipient. 

right, but it only works for senders who always use the same relays.

> So rather than looking at this in terms of ~"can I reject mail from
> senders without MS RRs?" (the answer to that is "no"), the appropriate way
> to look at this is ~"can the owner of a domain use this to prevent
> forgeries from their domain" and "can the recipients use this as a
> supplemental method for killing spam" (the answer to both of those
> questions is "yes").

the answer to the first of the questions is "no, not in general".
the answer to the second question is "yes, but it won't make much
of a difference unless most of the sources of spam use it"

> > 2. my guess is that IESG isn't likely to approve a new use of, or
> >    extensions to, the MS record.
> 
> Maybe you could be bothered to provide some kind of support for this.

it's just my guess from having been on IESG.  

Keith

Date: Fri, 17 May 2002 13:43:37 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:
> 
> >  Any comments on this approach?

> 1. it's a non-starter to force senders to submit to/relay through any
>    particular host or hosts.  people can and do send mail from
>    anywhere, and this is essential functionality.

Providing a list of authorized senders is a sender-side choice. As such,
this is an elective system and has to be recognized as one. There will
still be hosts and domains that do not use the elective system[1], so this
cannot be relied upon as an exclusive test.

BUT, providing this kind of information is as beneficial to the sender as
it is to the recipient. EG, if hotmail used this, then people like me
would stop blocking those domains, and would only block mail from hosts
which hotmail did not list as authorized for their domain.

So rather than looking at this in terms of ~"can I reject mail from
senders without MS RRs?" (the answer to that is "no"), the appropriate way
to look at this is ~"can the owner of a domain use this to prevent
forgeries from their domain" and "can the recipients use this as a
supplemental method for killing spam" (the answer to both of those
questions is "yes").

> 2. my guess is that IESG isn't likely to approve a new use of, or
>    extensions to, the MS record.

Maybe you could be bothered to provide some kind of support for this.

[1] as stated earlier, any such system will only work if its usage is
mandatory from the beginning

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail 
Date: Fri, 17 May 2002 14:25:43 -0400
Sender: [email-masked]

>  Any comments on this approach?

yes.

1. it's a non-starter to force senders to submit to/relay through any 
   particular host or hosts.  people can and do send mail from anywhere, 
   and this is essential functionality.

2. my guess is that IESG isn't likely to approve a new use of, or 
   extensions to, the MS record.  

Keith

Date: Fri, 17 May 2002 12:36:49 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

More thinking on this, probably the best approach here is to define a
flags sub-field in the MS RR, which allows for domain indirection.

Two flags which could be used right away:

   bit 1 off=value refers to a host
          on=value refers to a domain

   bit 2 off=use A/AAAA RRs associated with value
          on=use MX RRs associated with value

Combining these flags would allow for various functions, with a minimal
amount of overhead. Some examples below.

Use the IP addresses associated with outbound.example.com as the
authorized relay servers for example.com:

 [Answer Section]
  example.com.           MS   0   outbound.example.com.

 [Additional-Data Section]
  outbound.example.com.  A    10.0.0.3

Use the MX RRs associated with mail.example.com as the authorized relay
servers for example.com:

 [Answer Section]
  example.com.           MS   3   mail.example.com.

 [Additional-Data Section]
  mail.example.com.      MX   5   mx-1.example.com.
  mail.example.com.      MX   5   mx-2.example.com.
  mx-1.example.com.      A    10.0.0.1
  mx-2.example.com.      A    10.0.0.2

[note that the MX RRs would not be provided in the Additional-Data section
if a query for ALL resulted in them being provided in the Answer section
of the response]

Any and all hosts in the mail.example.com domain are authorized relay
servers for example.com:

 [Answer Section]
  example.com.           MS   1   mail.example.com.

 [Additional-Data Section]
  <empty>

[the PTR for the current mail session should be examined, and if it
resides within the specified domain, then it is authorized]

These could also be cumulative. EG, use the IP addresses associated with
outbound.example.com PLUS the MX RRs associated with the mail.example.com
domain as the authorized relay servers for example.com:

 [Answer Section]
  example.com.           MS   0   outbound.example.com.
                         MS   3   mail.example.com.

 [Additional-Data Section]
  outbound.example.com.  A    10.0.0.3
  mail.example.com.      MX   5   mx-1.example.com.
  mail.example.com.      MX   5   mx-2.example.com.
  mx-1.example.com.      A    10.0.0.1
  mx-2.example.com.      A    10.0.0.2

The benefit of this approach is that it allows for sender-based
customization of the interpretation (hosts vs MX lists), but minimizes the
risk of overloads. The above needs more smoothing out but I think it is
workable.

Any other flags which would be useful? Any comments on this approach?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Fri, 17 May 2002 09:03:01 -0700 (PDT)
From: [email-masked]
Subject: Re: E-mail major cause of bad security says security expert
To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked], Jacob Palme <[email-masked]>, IETF mailing list on MIME and e-mail <[email-masked]>
Sender: [email-masked]

> [email-masked] wrote:

> > > >"All of the security and privacy issues on the Web now relate to
> > > > e-mail," said Adam Shostack, director of technology at
> > > > Zero-Knowledge Systems, a Montreal-based privacy and security
> > > > company. "The shame about this behavior is that it's going on
> > > > surreptitiously and people are not given an obvious way to opt
> > > > out."  [...]

> There are multiple ways to parse that statement. One of them is:

>    mailprobs = mailprobs + webprobs

That's the way the statement was intended to be parsed and it is how I parsed
it. (If you look up the original article in Risks 22.03, as I did before I
first responded, you'll agree.)

I guess I should have also pointed out that Jacob's read of this as "E-mail
major cause of bad security" is incorrect. That mail systems are the root cause
of security problems is neither said nor implied by the article.

> which is certainly true for HTML-over-mail,

On the contrary, it is still hyperbole. As I said before, the web has a vast
array of security issues that don't relate to email at all. Attacks against
Web servers. Keeping track of the search terms a given user directly enters
into a search engine. And so on.

> esp with mailers that don't treat HTML mail specially.

Of course a mail user agent that simply passes HTML off to a web browser is
vulnerable to the same subset of web privacy and security issues that relate to
HTML viewing. However, a mail user agent can constrain its handling of HTML in
ways that would be completely unacceptable for a web browser to do. And while
broader understanding of and agreement on what constraints on HTML make sense
in the context of email, some clients do a fair job of handling this already.

> > My only comment is that such a clearly incorrect assessment of the
> > situation isn't worthy of a response. There are plenty of Web security
> > and privacy issues that have nothing whatsoever to do with email.

> Do they apply to HTML mail?

Only if you have an extremely limited and unrealistic view of the web.

				Ned

Date: Fri, 17 May 2002 12:19:33 -0400
From: Adam Shostack <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: IETF mailing list on MIME and e-mail <[email-masked]>
Subject: Re: E-mail major cause of bad security says security expert
Sender: [email-masked]

On Fri, May 17, 2002 at 04:47:04PM +0200, Arnt Gulbrandsen wrote:
> The following quotation arrived at the ietf-822 mailing list (cc'd), where
> it has caused a bit of discussion.
> 
> > "All of the security and privacy issues on the Web now relate to
> > e-mail," said Adam Shostack, director of technology at
> > Zero-Knowledge Systems, a Montreal-based privacy and security
> > company. "The shame about this behavior is that it's going on
> > surreptitiously and people are not given an obvious way to opt
> > out."  [...]
> 
> Is the quotation accurate? If so, would you mind elaborating?

It seems reasonably accurate.  The original article, by the way, is at
http://news.com.com/2100-1023-875992.html, and may provide some
context for what that "all" was referring to, which was tracking
technologies.

What I was talking about with the reporter was the use of web tracking
technologies, such as web bugs, counters, and cookies, and the privacy
impact that those things have.  Those issues started out in the
browsers and with the rise of html mail, now affect people's email as
well.  Similarly, with email being displayed by the same engines that
render web pages, I've seen Javascript spam with pop-unders.

What are you folks working on, and how did this come up?

Adam

Date: Fri, 17 May 2002 16:47:04 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Adam Shostack <[email-masked]>
Cc: IETF mailing list on MIME and e-mail <[email-masked]>
Subject: Re: E-mail major cause of bad security says security expert
Sender: [email-masked]

The following quotation arrived at the ietf-822 mailing list (cc'd), where
it has caused a bit of discussion.

> "All of the security and privacy issues on the Web now relate to
> e-mail," said Adam Shostack, director of technology at
> Zero-Knowledge Systems, a Montreal-based privacy and security
> company. "The shame about this behavior is that it's going on
> surreptitiously and people are not given an obvious way to opt
> out."  [...]

Is the quotation accurate? If so, would you mind elaborating?

--Arnt

Date: Fri, 17 May 2002 09:08:46 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: Jacob Palme <[email-masked]>, IETF mailing list on MIME and e-mail <[email-masked]>
Subject: Re: E-mail major cause of bad security says security expert
Sender: [email-masked]

[email-masked] wrote:

> > >"All of the security and privacy issues on the Web now relate to
> > > e-mail," said Adam Shostack, director of technology at
> > > Zero-Knowledge Systems, a Montreal-based privacy and security
> > > company. "The shame about this behavior is that it's going on
> > > surreptitiously and people are not given an obvious way to opt
> > > out."  [...]

There are multiple ways to parse that statement. One of them is:

   mailprobs = mailprobs + webprobs

which is certainly true for HTML-over-mail, esp with mailers that don't
treat HTML mail specially.

> My only comment is that such a clearly incorrect assessment of the
> situation isn't worthy of a response. There are plenty of Web security
> and privacy issues that have nothing whatsoever to do with email.

Do they apply to HTML mail?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

To: [email-masked]
Subject: Re: E-mail major cause of bad security says security expert
Sensitivity: 
From: Nick Shelness<[email-masked]>
Date: Fri, 17 May 2002 11:40:21 +0100
Sender: [email-masked]

Am I the only one to have noticed that the speaker represents "Zero-Knowledge Systems". Zero knowledge seems to sum him up.

Nick

Nick Shelness
Independent Technology Consultant

From: [email-masked] (Juergen Helbing)
To: [email-masked]
Subject: Re: E-mail major cause of bad security says security expert
Date: Fri, 17 May 2002 07:14:34 GMT
Sender: [email-masked]

On Thu, 16 May 2002 20:08:20 +0200, you wrote:

>>"All of the security and privacy issues on the Web now relate to e-mail,"
>>said Adam Shostack, director of technology at Zero-Knowledge Systems, a
>>Montreal-based privacy and security company. "The shame about this behavior
>>is that it's going on surreptitiously and people are not given an obvious
>>way to opt out."  [...]
>
>Any comments on this, which I found in the RISKS mailing
>list. Bad standards, or bad implementations?

Such a guy is "director of technology" ?   ;-)

Email is designed to transport messages - not to start programs.
That's all.

-- 
Juergen

Date: Thu, 16 May 2002 11:45:50 -0700 (PDT)
From: [email-masked]
Subject: Re: E-mail major cause of bad security says security expert
To: Jacob Palme <[email-masked]>
Cc: IETF mailing list on MIME and e-mail <[email-masked]>
Sender: [email-masked]

> At 16:51 -0700 02-04-15, Monty Solomon <[email-masked]> wrote:
> >"All of the security and privacy issues on the Web now relate to e-mail,"
> >said Adam Shostack, director of technology at Zero-Knowledge Systems, a
> >Montreal-based privacy and security company. "The shame about this behavior
> >is that it's going on surreptitiously and people are not given an obvious
> >way to opt out."  [...]

> Any comments on this, which I found in the RISKS mailing
> list. Bad standards, or bad implementations?

My only comment is that such a clearly incorrect assessment of the situation
isn't worthy of a response. There are plenty of Web security and privacy issues
that have nothing whatsoever to do with email.

				Ned

Date: Thu, 16 May 2002 20:08:20 +0200
To: IETF mailing list on MIME and e-mail  <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: E-mail major cause of bad security says security expert
Sender: [email-masked]

At 16:51 -0700 02-04-15, Monty Solomon <[email-masked]> wrote:
>"All of the security and privacy issues on the Web now relate to e-mail,"
>said Adam Shostack, director of technology at Zero-Knowledge Systems, a
>Montreal-based privacy and security company. "The shame about this behavior
>is that it's going on surreptitiously and people are not given an obvious
>way to opt out."  [...]

Any comments on this, which I found in the RISKS mailing
list. Bad standards, or bad implementations?
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Wed, 15 May 2002 11:34:05 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Paul Smith wrote:

> The difficult bits, at the moment, are: (1) knowing that the message
> has come from a server authorized to transfer mail on behalf of
> hotmail.com, and (2) if it's gone through multi-hop servers, checking
> that it's not been faked in that part of the process.
> 
> Problem (1) could be solved by having an SMTP extension where the MX
> servers for a domain can be asked if a particular IP address is
> allowed to send mail on behalf of that domain.

> Problem (2) is more tricky, but it would only be needed where mail
> from the sender's domain doesn't come directly to you, but goes
> through other (untrusted) relays in between.

I've been noodling on this and I think that a DNS approach will work,
although there are some costs associated with this approach that need to
be recognized first.

First off, using an MX RR for this is inappropriate, since MX defines
inbound relays, but many organizations use different servers for their
inbound and outbound relay functions. Some other RR has to be defined for
this to work with those organizations. However, it may be feasible to
specify that an MX RRset MAY be used if this other RR is not provided,
since this would include the 99% common case scenario where the inbound
and outbound relays are identical.

As for which RR to use, draft-church-dns-mail-sender-01.txt already
describes an MS (Mail Sender) RR for this purpose, and it would be just as
usable as anything else.

The principle problem with the use of these kinds of RRs is that they add
more RRs to the owner domain name. This contributes to a problem where the
DNS answer set overflows the response message, especially when queries are
issued for ALL RRs associated with the domain name (as is the wont with
Sendmail). For example, if the domain name already has an SOA, A, and
multiple NS and MX RRs defined, then it is already somewhat likely that
the 512 octet limit on the DNS message will be exceeded (look at
hotmail.com, with 14 MX RRs it overflows responses for just MX, nevermind
queries for ALL).

Another related problem here -- and the real concern for overflows -- is
that new RRs cannot use compression, since downstream caches have no
knowledge of the RRdata structure. They don't know if the compression
pointer is actually a compression pointer or if it is a numeric value or
what, so the result is that no new RRs can have compression. This is
almost guaranteed to trigger overflow even in relatively small sets.

Although from a hypertechnical perspective this is not a problem (the DNS
protocol defines the proper behavior), in practical terms this is an issue
to worry about, since DNS overflow requires at a minimum that the client
reissue the query over TCP, but not all DNS systems perform this fallback
processing. Furthermore, TCP DNS queries incur higher penalties on busy
servers, and they introduce a minimum of four round-trip times (counting
the original UDP failure) for every lookup. Adding a bunch of MS RRs will
absolutely result in more zones overflowing DNS answer sets and this will
result in more incomplete MX answer sets when fallback processing doesn't
take place, and performance problems when TCP fallback processing does
take place.

Another problem with this approach is that it becomes somewhat difficult
to make this service function in a dynamic fashion. EG, if I am a mobile
user and I want to list my ISP-dujour as a valid sender for my mail
domain, then I would have to add an MS RR to the domain name, update the
zone, wait for the changes to propagate to the replication slaves and then
hope that the destination sites don't have my earlier MS RRset in their
caches.

For all of these reasons, linking an MS RR to the mail domain name is not
the best design, and it should be avoided. Unfortunately, it is probably
the least worst choice available given the current SMTP model.

An alternative here would be to define a service name for the lookups, and
then using SRV RRs to avoid the owner name overflow problem. EG, come up
with something like _tcp._smtpc.example.com, with the SMTPC service name
providing links to the outbound relay hosts. However, this violates
several different design elements. First of all, the SRV RR defines
servers not clients. Second, there is no fixed port number for clients, so
defining an SMTPC service would be an exception condition. Furthermore,
there are still problems with dynamic updates and caching. In the end,
this approach trades one problem of overflows with other different
problems, and doesn't really get us anywhere.

The other alternative is for the owner domain name associated with the
relay server itself to list domains it relays for. This won't actually do
anything to solve the problem, since spammers can just list yahoo.com as
one of their domains and then we are right back to where we started.

So I think that using an MS RR is the most direct solution to the problem,
although it does introduce additional concerns. However, as was pointed
out earlier, it may also be feasible to allow the existing MX RR to be
reused as supplemental data if no MS RRs are defined. In that particular
scenario, the threat of overflow would be reduced, and for the vast
majority of the cases, this would provide immediate beneifts since most
organizations use the same hosts for inbound and outbound relays (we
wouldn't have to wait for people to add MS RRs to their zones to start
reaping the rewards).

In short:

  1) Take the domain element from the reverse-path and do an
     MS lookup

  2) If no MS RRs were returned, do an MX lookup

  3) Locate/extract the A RRs associated with the MS/MX RR data

  4) compare the IP addresses of those hosts against the current
     connection tuple

  5) accept if matched, issue 551 (or something) if not

This will work without any additional effort in the common cases, EG a
direct connection from a sender's submission server with an MX RR, or
indirect through sender-side relay with MS/MX RRs.

This will not work if the messaging clients have their own full-service
MTAs. In that condition, it may be that a special wildcard interpretation
can be defined for use, such as:

    example.com  MS  outbound.example.com    ; outbound relay
    example.com  MS  *.eng.example.com       ; Engineering dept
    example.com  MS  *.example.net           ; ISP network

That would allow all of the hosts in the eng.example.com domain to use
their own MTAs, but without having to list each host. This approach could
also work for roaming users who use different mail servers every day, but
which are always using the same ISP. Note that the above approach won't
work as described but something similar to it can probably be fleshed out
and made usable.

Another problem condition is with delivery servers which are isolated
behind an internal relay server. If the delivery servers generate these
lookups, then the answer set will never match with the IP address in use
(the address of the in-house relay server). Those servers would have to
disable this feature, or would have to add the relay to an internal list
of known-good relays.

The remaining killer problem here is with aliasing, as Philip pointed out.
If mail is sent to [email-masked] and is then sent to [email-masked]
-- but the original return-path is retained -- then the sending server
won't match with the MS/MX RRs for the original sender's mail domain. I
think that this can also be handled via whitelist exceptions, as with
internal relays above. If you know that you will frequently receive mail
from a particular alias, then you can add that domain to your exception
list and get all of that mail without any additional work. Plus, if the
incoming servers for the alias domain implement this system, then the mail
will be filtered at that junction, so this won't mandate that you get spam
from those domains.

All of the other conditions should work as normal. Mailing lists that
rewrite the reverse-path will function. Resent-* forwards will work since
the reverse-path will point to the resender. I don't see any other issues
or problem cases.

Thoughts?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 14 May 2002 09:28:12 -0500
From: "Eric A. Hall" <[email-masked]>
To: Philip Hazel <[email-masked]>
CC: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Philip Hazel wrote:

> Don't forget about forwarding. Case (2) arises when a user arranges
> for her "home" MTA to forward mail somewhere else, either permanently
> or temporarily.

Another weird scenario is redirects, via Resent-* forwarding

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 14 May 2002 09:03:50 +0100 (BST)
From: Philip Hazel <[email-masked]>
To: Paul Smith <[email-masked]>
cc: [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

On Mon, 13 May 2002, Paul Smith wrote:

> In that case, it depends whether the 'home' MTA would change the MAIL FROM
> in the envelope to be someone at the home MTA's domain or leave it as the
> original MAIL FROM address. (It can often be a bad idea to leave it as the
> original MAIL FROM address because bounces can be very confusing then...)

It can also be a bad idea to change the MAIL FROM, because bounces then
may not get to anybody.

But I don't want to argue about which is the better way to do it. The
point is that some MTAs do not change the MAIL FROM. So this case must
be dealt with.

> If the envelope is changed to be someone at the home MTA's domain, then
> this type of checking won't be any different from a mail sent directly from
> the home MTA.

True, but that's not really the point.

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Mon, 13 May 2002 10:27:23 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: authenticating the source of mail
Sender: [email-masked]

At 10:08 13/05/2002 +0100, Philip Hazel wrote:
>On Sat, 11 May 2002, Paul Smith wrote:
>
> > The difficult bits, at the moment, are: (1) knowing that the message has
> > come from a server authorized to transfer mail on behalf of hotmail.com,
> > and (2) if it's gone through multi-hop servers, checking that it's not been
> > faked in that part of the process.
>
>Don't forget about forwarding. Case (2) arises when a user arranges
>for her "home" MTA to forward mail somewhere else, either permanently or
>temporarily.

In that case, it depends whether the 'home' MTA would change the MAIL FROM 
in the envelope to be someone at the home MTA's domain or leave it as the 
original MAIL FROM address. (It can often be a bad idea to leave it as the 
original MAIL FROM address because bounces can be very confusing then...)

If the envelope is changed to be someone at the home MTA's domain, then 
this type of checking won't be any different from a mail sent directly from 
the home MTA.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Mon, 13 May 2002 10:08:07 +0100 (BST)
From: Philip Hazel <[email-masked]>
To: Paul Smith <[email-masked]>
cc: [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

On Sat, 11 May 2002, Paul Smith wrote:

> The difficult bits, at the moment, are: (1) knowing that the message has
> come from a server authorized to transfer mail on behalf of hotmail.com,
> and (2) if it's gone through multi-hop servers, checking that it's not been
> faked in that part of the process.

Don't forget about forwarding. Case (2) arises when a user arranges
for her "home" MTA to forward mail somewhere else, either permanently or
temporarily.

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Sat, 11 May 2002 23:29:58 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: authenticating the source of mail
Sender: [email-masked]

At 16:22 11/05/2002 -0500, Eric A. Hall wrote:
>which I'm not sure about). There is certainly some scale involved here but
>the core problem of identity within a domain has already been solved by
>SMTP AUTH. If I could refuse mail from [email-masked] unless it came
>from a server authorized to transfer mail on behalf of hotmail.com -- and
>if hotmail.com enforced authentication (which they do, via the login page)
>-- then I can be pretty sure that [email-masked] really sent the mail.

That's an interesting point.

The difficult bits, at the moment, are: (1) knowing that the message has 
come from a server authorized to transfer mail on behalf of hotmail.com, 
and (2) if it's gone through multi-hop servers, checking that it's not been 
faked in that part of the process.

Problem (1) could be solved by having an SMTP extension where the MX 
servers for a domain can be asked if a particular IP address is allowed to 
send mail on behalf of that domain.

This would solve the problem where the sender (eg Hotmail) sends directly 
to your own SMTP server. Your server could then do an MX lookup on the MAIL 
FROM address, and send a query back to that server to ask if the sending 
mail server is allowed to send mail from that address.

Problem (2) is more tricky, but it would only be needed where mail from the 
sender's domain doesn't come directly to you, but goes through other 
(untrusted) relays in between. The only solution I can come up with, off 
the top of my head, is a 'brute force' one - each SMTP server could 
remember ALL the message-IDs of messages it has sent in the past, say, 5 
days, then when you receive a message, you could do an MX lookup and ask 
the source server 'did you send message-ID xxxx'. I'm not really happy with 
this solution, but I don't really know if there is a problem here in the 
first place. (How often do people receive mail which comes to them via an 
untrusted relay not run by the sending domain, or the sending domain's ISP)

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Sat, 11 May 2002 16:22:43 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> Being able to pin down the source to a "small enough" organization
> or set of people might be enough, and achieving that might be much
> easier than trying to authenticate every single possible individual
> sender.

I don't think the size of the organization matters (if that is your point,
which I'm not sure about). There is certainly some scale involved here but
the core problem of identity within a domain has already been solved by
SMTP AUTH. If I could refuse mail from [email-masked] unless it came
from a server authorized to transfer mail on behalf of hotmail.com -- and
if hotmail.com enforced authentication (which they do, via the login page)
-- then I can be pretty sure that [email-masked] really sent the mail.

I don't know many people at cs.utk.edu so the likelihood of somebody from
that domain (assuming that they had privs to send mail through that
server) going to the trouble of forging mail from [email-masked] is
pretty unlikely, even if SMTP AUTH isn't used locally.

The problem here is with multi-hops and relays. Some sort of recursive
signatures will probably be required.

As for viruses, not much we can do to stop idiots from opening
attachments. Worms are somewhat constrained by this kind of system though,
since they will have to be sent from the local user, riding on top of the
local server and its authentication mechanisms.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail 
Date: Sat, 11 May 2002 17:01:38 -0400
Sender: [email-masked]

> Historically, the approach to "secure" mail has been to provide mechs that
> let a receiver validate the identity of the sender, but I am really
> wondering if this isn't inappropriate in the majority of the cases.
> 
> In the paper-based world, we "trust" the delivery system, and do not
> normally attempt to validate the supposed sender of the materials. 

That's only because snail-mailed bombs and anthrax have been relatively 
rare.

I *wish* we could make the same statement about email viruses.

However, I do think that some form of verifable source authentication,
if it were widely-enough deployed, might be useful in deterring viruses 
even if it doesn't always identify the exact person who sent the message.
Being able to pin down the source to a "small enough" organization 
or set of people might be enough, and achieving that might be much easier
than trying to authenticate every single possible individual sender. 

Keith

Date: Sat, 11 May 2002 15:41:07 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Historically, the approach to "secure" mail has been to provide mechs that
let a receiver validate the identity of the sender, but I am really
wondering if this isn't inappropriate in the majority of the cases.

In the paper-based world, we "trust" the delivery system, and do not
normally attempt to validate the supposed sender of the materials. We
don't demand that people physically bring contracts to us so that we can
verify that they are the authors. We don't demand thumbprints for
comparison to a notarized set of fingerprints on file. Instead, we "trust"
the letterhead and the postmarks (or the fax caller-ID), and then use
blind assumptions for the rest of it. Only in special cases do we require
validation of sender identity.

Similarly, with protocols like HTTP, we trust the transfer protocol, and
do not demand that each HTML document be independently signed. We assume
that the documents have not been forged and rely on the delivery system to
provide the minimum comfort level.

So with mail, for 99% of the cases, we could probably provide "secure
mail" by adding integrity to the delivery system. We can still use
certificates and signatures for the remaining cases where we need to
"prove" identity, but for the vast majority of cases, just knowing that
"this host is authorized to send mail for this domain, so I will trust
that this sender is legitimate" will probably be good enough. It would
certainly be good enough to crush anonymous spam, and would also be
sufficient for the overwhelming day-to-day communications.

Also, the SMTP part of the transfers is the only time when the parties are
guaranteed to be connected, so providing the necessary credentials at this
phase is somewhat feasible, whereas user certs and the like may not be
workable due to asynchronous connectivity problems.

The questions then, is it possible to provide this level of security
within the protocol, such that it is easy to implement on a wide basis,
but it is secure enough not to be easily forged? Forget the users, is
there a practical way to ensure that public SMTP agents can always be
positively identifed?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

To: [email-masked]
Subject: Re: text/html an abomination
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sat, 11 May 2002 12:27:16 -0700
Lines: 16
Sender: [email-masked]

Florian Weimer <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>>     This message is in HTML.  Formatting HTML for viewing can be slow
>>     and may let the HTML code do things that you do not expect, so the
>>     raw HTML code of this message is displayed below.  If you trust
>>     the sender of this HTML message and want to see the formatted view
>>     of this message, click on this box to render it.

> The "trusted sender won't send malicious HTML" assumption is
> fundamentally flawed.

Agreed; mea culpa.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: text/html an abomination
From: Florian Weimer <[email-masked]>
Mail-Followup-To: Russ Allbery <[email-masked]>, [email-masked]
Date: Sat, 11 May 2002 11:02:12 +0200
Lines: 10
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:

>     This message is in HTML.  Formatting HTML for viewing can be slow
>     and may let the HTML code do things that you do not expect, so the
>     raw HTML code of this message is displayed below.  If you trust
>     the sender of this HTML message and want to see the formatted view
>     of this message, click on this box to render it.

The "trusted sender won't send malicious HTML" assumption is
fundamentally flawed.

To: [email-masked]
Xref: clerew local.mime:1503
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 10 May 2002 11:31:06 GMT
Lines: 79
Sender: [email-masked]

In <[email-masked]> Marc Mutz <[email-masked]> writes:

>On Wednesday 08 May 2002 13:34, Charles Lindsey wrote:
>>
>> Yes, I think on further reflection that you would not insist on too much
>> (or even any) canonicalization of the RFC 2047 stuff, but you would
>> require it to be decoded to an octet stream at the far end, and the local
>> part would be considered to match if the octet streams matched.
><snip>

>But that breaks existing software that needs to compare local-parts for 
>equality (think mailing list handling software). This breaking can only be 
>avoided if the encoding has the property to always yield the same output 
>octet-sequence for the same input Unicode sequence (modulo Unicode n11n and 
>c13n).

Yes, I think you have to accept that agents that expect to process I18N
local-parts would have to have RFC 2047 decoding built into them as part
of their comparison process if this particular method is to be used.

><snip (1)>
>>
>> 2. The result of applying IDNA to some valid local-part is another
>> local-part. Who is to tell whether that was not the intended local-part in
>> the first place. So you would first need to restrict RFC 2822 local-parts
>> in some way.

>But this is a problem that arises for every encoding that is representable in 
>current local-part syntax. Rfc2047 also yields just another valid 
>local-part[1] (altough less likely to confilct with existing local-part due 
>to the many "broken" mailers that would display it decoded).

Yes, although the RFC 2047 syntax is bizarre enough that if you see
anything of the form =?...?...?...?= you can pretty safely assume it was
not intended to be taken literally (and lots of software around will
immediately decode it whether it should or not).

So that leads us to the inevitable conclusion that we are FORCED to change
the syntax of local-part in RFC 2822 before we can make progress on this.
Two possibities:

1. Restrict it by excluding some characters (which are then available to
indicate that some encoding has taken place). The simplest restriction
would be to declare
	local-part = token / quoted-string
which frees up '?' and '='. So either you then use RFC 2047 encoding, or
you use IDNA with perhaps '??' in the Nameprep. So our friend jürgen
becomes ??--jrgen-kva .

2. Allow some extra characters, which would only be allowed for encoded
local-parts. Trouble is they would have to be chosen from the 'specials',
and there are no obvious candidates.

So I think it has to be (1), which then leaves the choice of RFC 2047 or
IDNA open for further discussion (though the problem of uppercase letters
in IDNA would still need resolution).

So we are then back to the issue of whether it is safe to allow all and
sundry applications to indulge in Unicode normalization and lowercasing.
People seem determined to argue that we need this. If that is so, then I
still think it better to let the operating system do it. That would mean
having a relocateable library containing the Unicode tables permanently
built into or supplied with each operating system. There might be some
future in that if the Unicode people could publish it (complete with a
unicode.h file to establish its structure). Do we know whether they have
plans to do that?  I do know that they publish various tables, but only in
txt form AIUI.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Subject: Re: All these lonely accents, where do they all come from?
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 09 May 2002 13:03:37 -0700
Lines: 20
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Keith Moore <[email-masked]> writes:

>> which is why you can't expect all Unicode to be normalized, because it
>> may have been translated from another charset.

> Rubbish. Read what I wrote.

> I was talking of code C -> *normalized* Unicode -> code C.

> So the Unicode in question was normalized by definition. And the
> transformation from C to normalized unicode is NOT invertible (in
> general).

Which, as Keith said, is why people don't do that transformation and
instead transform code C into unnormalized Unicode so that they can later
translate back into code C if they want to.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: Marc Mutz <[email-masked]>
Organization: KDE
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Thu, 9 May 2002 19:08:09 +0200
Content-Description: clearsigned data
Sender: [email-masked]

On Wednesday 08 May 2002 13:34, Charles Lindsey wrote:
> In <[email-masked]> Marc Mutz <[email-masked]> writes:
<snip>
> >A base64-encoded UTF-8 encoded-word.
>
> Yes, I think on further reflection that you would not insist on too much
> (or even any) canonicalization of the RFC 2047 stuff, but you would
> require it to be decoded to an octet stream at the far end, and the local
> part would be considered to match if the octet streams matched.
<snip>

But that breaks existing software that needs to compare local-parts for 
equality (think mailing list handling software). This breaking can only be 
avoided if the encoding has the property to always yield the same output 
octet-sequence for the same input Unicode sequence (modulo Unicode n11n and 
c13n).

> But, for local-parts, the only other suggestion is to use IDNA. That
> suffers from two problems, which nobody has responed to yet:
>
<snip (1)>
>
> 2. The result of applying IDNA to some valid local-part is another
> local-part. Who is to tell whether that was not the intended local-part in
> the first place. So you would first need to restrict RFC 2822 local-parts
> in some way.

But this is a problem that arises for every encoding that is representable in 
current local-part syntax. Rfc2047 also yields just another valid 
local-part[1] (altough less likely to confilct with existing local-part due 
to the many "broken" mailers that would display it decoded).

Marc

[1] If you follow rfc2047's suggestion to encode any specials, too, so the 
encoded-word looks like an atom.

- -- 
Marc Mutz <[email-masked]>

Date: Thu, 9 May 2002 11:25:01 -0500
To: "Charles Lindsey" <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: All these lonely accents, where do they all come from?
Cc: [email-masked]
Sender: [email-masked]

On 5/9/02 at 8:45 AM +0000, Charles Lindsey wrote:

>If Apple's Unicode converter does not normalize, then it is severely 
>broken and should be fixed.

Of course it *can* normalize. The key is that it doesn't *always* 
normalize, because there are loads of reasonable things you can do 
with non-normalized text: You may want the conversion to be 
reversible, you may want to include separators because you are using 
the text in a word processor that needs those separators, etc. You 
absolutely cannot expect that all Unicode text floating around is 
normalized.

pr
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

To: [email-masked]
Xref: clerew local.mime:1497
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: All these lonely accents, where do they all come from?
Date: Thu, 9 May 2002 08:45:06 GMT
Lines: 29
Sender: [email-masked]

In <a05111808b8fe678cc604@[216.43.25.67]> Pete Resnick <[email-masked]> writes:

>3. I take the text of the old message and run it through Apple's 
>Unicode conversion library to convert it to Unicode. I (not knowing 
>any better) use the default conversion parameters, which results in 
>text that is not normalized, but rather easily reversible into the 
>source character set.

If Apple's Unicode converter does not normalize, then it is severely
broken and should be fixed.

The two places where normalization MUST take place, in order to ensure
that non-normalized code is never seen by normal applications, are
keyboards and converters from non-Unicode notations.

In addition to those, it probably SHOULD take place in a few really
critical places, such as in Nameprep prior to DNS lookup (but that should
be built in inside, or very close to the input to, the DNS system).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1498
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: All these lonely accents, where do they all come from?
Date: Thu, 9 May 2002 08:49:36 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> But in fact translations from some arbitrary charset into normalized
>> Unicode are not invertible.

>which is why you can't expect all Unicode to be normalized, because it
>may have been translated from another charset.

Rubbish. Read what I wrote.

I was talking of code C -> *normalized* Unicode -> code C.

So the Unicode in question was normalized by definition. And the
transformation from C to normalized unicode is NOT invertible (in
general).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: All these lonely accents, where do they all come from? 
Date: Thu, 09 May 2002 08:59:04 -0400
Sender: [email-masked]

> No, that is NOT the required property.
>
> You start with text A in some horrible legacy charset C (which knows nothing
> of normalization and embodies horrendously redundant notations). You
> convert it into Unicode U(A) and normalize it N(U(A)) (because converters,
> like keyboards, are one of the small set of programns that MUST
> normalize). 

no, the converter is not one of those things that MUST normalize, because
you might want the possibility of translating back to the original charset
after manipulating the text in Unicode.

Keith

To: [email-masked]
Xref: clerew local.mime:1496
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: All these lonely accents, where do they all come from?
Date: Thu, 9 May 2002 08:38:45 GMT
Lines: 37
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> > the reason that Unicode has multiple representations for characters in
>> > the first place - because they wanted to support invertable
>> > translation to and from legacy character sets without information loss
>> 
>> Nonsense. One can't convert UTF-8 to ISO 8859-1, for example, without
>> information loss.

>I suppose I should have been more precise, but I thought it would be
>obvious. the criteria was to be able to translate from a legacy charset
>to Unicode and back to the original charset without lossage. 

No, that is NOT the required property.

You start with text A in some horrible legacy charset C (which knows nothing
of normalization and embodies horrendously redundant notations). You
convert it into Unicode U(A) and normalize it N(U(A)) (because converters,
like keyboards, are one of the small set of programns that MUST
normalize). Then you convert it back into the legacy charset C(N(U(A))).

You cannot expect
	A = C(N(U(A)))
(though you might just be lucky, and it would likely stabilize after one
or more further cycles round the loop).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: All these lonely accents, where do they all come from? 
Date: Wed, 08 May 2002 22:55:24 -0400
Sender: [email-masked]

> But in fact translations from some arbitrary charset into normalized
> Unicode are not invertible.

which is why you can't expect all Unicode to be normalized, because it
may have been translated from another charset.

To: [email-masked]
Xref: clerew local.mime:1481
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Wed, 8 May 2002 11:34:28 GMT
Lines: 45
Sender: [email-masked]

In <[email-masked]> Marc Mutz <[email-masked]> writes:

>On Monday 06 May 2002 18:03, Charles Lindsey wrote:
><snip>
>> So perhaps we should still be looking at other possibilities for local
>> part, such as RFC 2047. Yes that needs care to ensure it is done
>> canonically, but that is just a matter of insisting that the whole
>> local-part be treated as exactly one encoded-word.

>A base64-encoded UTF-8 encoded-word.

Yes, I think on further reflection that you would not insist on too much
(or even any) canonicalization of the RFC 2047 stuff, but you would
require it to be decoded to an octet stream at the far end, and the local
part would be considered to match if the octet streams matched. I did
something similar for signing headers with RFC 2047 encodings in
draft-lindsey-usefor-signed-01.txt.

You might also require texts in other charsets to be translated to utf-8
before comparing the octet streams. It would also work as well to compare
octet streams in UTF-16 or UCS-16 form.

But, for local-parts, the only other suggestion is to use IDNA. That
suffers from two problems, which nobody has responed to yet:

1. Punycode, as it stands, converts uppercase to lowercase. That is not
necessary (lowercasing is really a part of Name-prep, and local-parts would use a
different *-prep), so the Punycode spec could be changed.

2. The result of applying IDNA to some valid local-part is another
local-part. Who is to tell whether that was not the intended local-part in
the first place. So you would first need to restrict RFC 2822 local-parts
in some way.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1480
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Wed, 8 May 2002 11:21:27 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>it's a nice idea, but keyboard drivers and file formats are far removed 
>from anything over which we have significant influence.    that, and 
>it's not sufficient - some text is generated by computer programs,
>and we can't force those programmers to always generate normalized text
>either.

If the spec says that incoming text to some application is required to be
normalized, and the result of submitting non-normalized text to it is that
mail fails to get delivered, or is delivered to the wrong place, then the
programmers in question will soon learn to mend their ways.

You need both sticks and carrotes in this business. At the moment, we
appear to be choking on a surfeit of carrots.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1482
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: All these lonely accents, where do they all come from?
Date: Wed, 8 May 2002 11:36:29 GMT
Lines: 22
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>well for starters, how about the reason that Unicode has multiple 
>representations for characters in the first place - because they wanted 
>to support invertable translation to and from legacy character sets without 
>information loss, and also because they combined several character sets 
>which had some non-null intersections but there were cases where it made 
>sense to assign different codepoints.  

But in fact translations from some arbitrary charset into normalized
Unicode are not invertible.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Wed, 08 May 2002 17:03:13 -0400
Sender: [email-masked]

> >> Surely, the lesson to be learned is that Codecs are more trouble than they
> >> are worth, and that sooner or later we need to have a Final Solution to
> >> the problem.
> 
> >surely the lesson to be learned is that those who don't understand history
> >are doomed to repeat past mistakes.  "Final Solutions" are neither.
> 
> And Codecs are one of those past mistakes. 

wrong.  what history shows us is that we can't come up with a single
universal solution for all cases, and that transitions to completely
new systems are very disruptive, so adapting is often a useful tool.
in fact utf-8 is a good example of what you are calling a codec -
it tries to make unicode fit into a box designed for 8-bit ASCII.

> UTF-8 may not be a "final solution", but it is a much cleaner solution
> than anything else on offer, and it has considerable extensibility built
> into it, and a strong and authoritative body (the Unicode Consortium) to
> keep it on track. I think is will hold the field at least as long as ASCII
> did.

utf-8 is a good piece of design.  I won't hazard a guess as to how
long it will last, but it seems presumptious and naive to expect that
it's a good enough solution to the universal language problem that
it's going to be usable without significant change for more than a few
years.  

when someone comes up with something "better" - which is to say,
better at working with the then-current set of conditions -
are you going to insist that everyone move to that new charset,
or are you going to insist that they stay with utf-8 for the 
sake of compatibility?

Keith

To: [email-masked]
Xref: clerew local.mime:1479
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
Date: Wed, 8 May 2002 11:18:03 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> Surely, the lesson to be learned is that Codecs are more trouble than they
>> are worth, and that sooner or later we need to have a Final Solution to
>> the problem. 

>surely the lesson to be learned is that those who don't understand history 
>are doomed to repeat past mistakes.  "Final Solutions" are neither.

And Codecs are one of those past mistakes. That was my point, and yet you
still seem to envisage repeating that mistake.

UTF-8 may not be a "final solution", but it is a much cleaner solution
than anything else on offer, and it has considerable extensibility built
into it, and a strong and authoritative body (the Unicode Consortium) to
keep it on track. I think is will hold the field at least as long as ASCII
did.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 8 May 2002 09:25:51 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: DSN redesign
Sender: [email-masked]

Paul Smith writes:
> I think the "guaranteed" 'some response' of DSN is good.

Pay attention. The guarantee, like the analogous 8BITMIME guarantee, is
a myth: it was shredded by sendmail's longstanding disobedience. Even
when your ``some response'' actually appears, it is normally a useless
``I dunno'' response. Furthermore, the attempt to make this guarantee
forced a link-level design that, being rather difficult to implement,
interfered with the goal of producing ``Yes'' responses.

  [ Notice-Requested-Upon-Delivery-To ]
> What about BCCs?

The mechanism works exactly the same way.

If you're asking about the fact that truly blind deliveries require
separate envelopes: yes, that's true, and obviously the sender has to
split the return receipts accordingly.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 8 May 2002 09:10:58 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: All these lonely accents, where do they all come from?
Sender: [email-masked]

Demanding that every text-handling program deal with normalization is
analogous to demanding that every text-handling program deal with 8859-1
and KOI-R and so on. In contrast, in a better-engineered system, those
details are isolated inside a small number of programs.

Paul Smith writes:
> Can you *guarantee* that non-normalised text won't ever get into emails?

That's analogous to asking ``Can you _guarantee_ that the text won't be
in 8859-1 or KOI-R?'' You're completely missing the point.

The question is about the costs of requiring normalized UTF-8
everywhere, as opposed to allowing unnormalized UTF-8. The extra
constraint has an obvious benefit for typical text-handling programs.

Keith is claiming, without a shred of justification, that the cost for
text-input programs outweighs the benefit for the much larger set of
text-handling programs. So far we haven't seen even one example where
producing normalized UTF-8 would be difficult.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 8 May 2002 08:49:02 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: All these lonely accents, where do they all come from?
Sender: [email-masked]

Pete Resnick writes:
> Apple's Unicode conversion library

Details, please. What's an example where the library produces
unnormalized text? Why does it do so? Why shouldn't Apple fix it?

Of course, in the long term, conversions to Unicode won't be an issue,
because everything will be in Unicode in the first place.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Wed, 8 May 2002 09:22:19 +0100
To: [email-masked]
From: Ian Bell <[email-masked]>
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
Sender: [email-masked]

On Tue, 7 May 2002, Charles Lindsey <[email-masked]> wrote:
>
>In <[email-masked]> Marc Mutz <[email-masked]> writes:
>
>
>>On Monday 06 May 2002 10:05, Simon Josefsson wrote:
>><snip>
>>> You can use MIME parts for this as well, mark the parts as inline and make
>>> sure you don't add newlines during encoding.  The processing rules
>>> describes fairly well how to decode it back together.
>><snip>
>
>>Euhm? Where did you read this? The only rfc I know of that defines
>>presentation of mails is rfc2183, defining the C-D header (plus others that
>>add new C-D values). RFC2183 basically interprets "inline" as "non-iconic" or
>>"expanded". In HTML speak this means <div>, not <span>.
>
>Well Turnpike can do this trick, and they swear blind that the behaviour
>in question is mandated by the MIME standards.

The MIME specs are clear that the CRLF preceding a boundary belongs to 
the boundary, therefore it is possible to meet in the wild consecutive 
inline text entities where the first part does _not_ have a final CRLF.

The MIME specs _don't_ mandate how such material should be displayed 
(and we have never said that they do). Turnpike does not generate such 
messages.

However, if Turnpike comes across such a message, and it is capable of 
displaying the character sets of both parts, it will not introduce extra 
vertical white space that is not in the message source. That is, 
Turnpike would combine the text parts with no extra CRLF. Such an 
interpretation for display seems (to me) "natural" given the MIME specs, 
but _not_ mandated.

-- 
Ian Bell                                           T U R N P I K E

Date: Wed, 8 May 2002 10:23:14 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: text/html an abomination
Sender: [email-masked]

Russ Allbery <[email-masked]>
>     This message is in HTML.  Formatting HTML for viewing can be slow
>     and may let the HTML code do things that you do not expect, so the
>     raw HTML code of this message is displayed below.  If you trust
>     the sender of this HTML message and want to see the formatted view
>     of this message, click on this box to render it.

Trusting people based on "From:" is an idea whose time has gone.

1. Virii are often sent by people you know.

2. Various aquaintances have recently received spam from "me". The "from"
   and "to" addresses in that spam seem to be harvested from the same web
   pages. Successfully too: The spam went straight through a whitelist
   filter.

--Arnt

Date: Wed, 08 May 2002 06:40:55 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: All these lonely accents, where do they all come from?
Sender: [email-masked]

At 04:31 08/05/2002 +0000, D. J. Bernstein wrote:
> > then there are existing editors and keyboard drivers that don't generate
> > normalized text.
>
>Name them, Keith.

Prove that there AREN'T any..

> > > Then explain why you think normalization should be put into every
> > > program that _handles_ text, instead of the much smaller set of programs
> > > (notably, keyboard interfaces) that _produce_ text.
> > I didn't say that.
>
>Then why are you demanding that thousands of Internet programs acquire
>Unicode normalization features?

Can you *guarantee* that non-normalised text won't ever get into emails? If 
so, then we'll all come to you for compensation when it does :-)  If not, 
then email software may need the ability to normalise it.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Wed, 8 May 2002 00:42:07 -0500
To: "D. J. Bernstein" <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: All these lonely accents, where do they all come from?
Cc: [email-masked]
Sender: [email-masked]

On 5/8/02 at 12:48 AM +0000, D. J. Bernstein wrote:

>Please explain exactly where you think unnormalized UTF-8 text 
>enters the world's computer systems.

1. My e-mail client receives a piece of e-mail in a legacy character set.

2. The user replies to that message.

3. I take the text of the old message and run it through Apple's 
Unicode conversion library to convert it to Unicode. I (not knowing 
any better) use the default conversion parameters, which results in 
text that is not normalized, but rather easily reversible into the 
source character set.

4. I take that text and what the user has typed in their reply, UTF-8 
encode it, and send it.

pr
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Wed, 08 May 2002 06:34:01 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: DSN redesign
Sender: [email-masked]

At 00:37 08/05/2002 +0000, D. J. Bernstein wrote:
>Eric A. Hall writes:
> > Secondarily, some of the recipient-specific
> > fields can't be implemented as shared header fields
>
>Have one field for each recipient, naming the recipient:
>
>    Notice-Requested-Upon-Delivery-To: [email-masked]

What about BCCs? Also, I think the "guaranteed" 'some response' of DSN is good.

OTOH, it wouldn't be impossible to extend DSN (EDSN?) to make it 'wrap' a 
DSN request into the header the first time it sends back a 'Relayed' 
notification. Then when a EDSN-aware MTA receives a wrapped DSN header it 
re-creates the DSN envelope information

So, you have something like:

MUA     (DSN aware)     - adds DSN envelope info
MTA1    (DSN aware)     - receives, and passes on DSN info
MTA2    (EDSN aware)    - receives DSN info, sees that MTA3 doesn't know 
DSN so creates a 'DSN-Info:' header with the DSN information encoded into 
it (not difficult to come up with a schema for doing this). Also sends back 
a 'relayed' notification with text indicating that 'DSN may be resumed at a 
later hop'
MTA3    (not DSN aware) - receives message with DSN-Info header. Since it's 
not DSN aware it just ignores the DSN-Info header and passes it on
MTA4    (EDSN aware)    - receives message with DSN-Info header. Recreates 
the DSN info in the envelope and removes DSN-Info header. (Optionally sends 
back a notification saying that this has happened)
MDA     (DSN aware)     - receives message with DSN info in header and acts 
on it, sending back success notifications.

Comments?

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: All these lonely accents, where do they all come from? 
Date: Wed, 08 May 2002 01:20:46 -0400
Sender: [email-masked]

> > the reason that Unicode has multiple representations for characters in
> > the first place - because they wanted to support invertable
> > translation to and from legacy character sets without information loss
> 
> Nonsense. One can't convert UTF-8 to ISO 8859-1, for example, without
> information loss.

I suppose I should have been more precise, but I thought it would be
obvious. the criteria was to be able to translate from a legacy charset
to Unicode and back to the original charset without lossage.  

> If you're trying to point to some actual feature of Unicode, give a
> complete quote and a precise reference---and then explain how this
> justifies your claims about unnormalized text entering the system.

I've got the Unicode standard on the shelf here, but frankly Dan, 
you're not worth the trouble of my opening the book.   you can find it 
yourself if you're that interested.

if you want to try to convince everybody that it's impossible for
non-normalized Unicode to leak into email, or that everyone (not just
email) should generate Unicode in the way that you think is best,
and that everyone else is likely to follow your advice (because your
strategy depends on precisely that) feel free.  since you won't get 
anything close to consensus anyway, there's not much point in my 
discussing this with you further.

Keith

Date: 8 May 2002 04:31:44 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: All these lonely accents, where do they all come from?
Sender: [email-masked]

Keith 7-bits-forever Moore writes:
> the reason that Unicode has multiple representations for characters in
> the first place - because they wanted to support invertable
> translation to and from legacy character sets without information loss

Nonsense. One can't convert UTF-8 to ISO 8859-1, for example, without
information loss.

If you're trying to point to some actual feature of Unicode, give a
complete quote and a precise reference---and then explain how this
justifies your claims about unnormalized text entering the system.

> then there are existing editors and keyboard drivers that don't generate 
> normalized text.

Name them, Keith.

> for all I know, it may not even be desirable to do so  -
> there may be some languages for which this would be seen as a defect.

Do you understand what normalization is, Keith?

> > Then explain why you think normalization should be put into every
> > program that _handles_ text, instead of the much smaller set of programs
> > (notably, keyboard interfaces) that _produce_ text.
> I didn't say that.

Then why are you demanding that thousands of Internet programs acquire
Unicode normalization features?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 07 May 2002 20:38:45 -0700 (PDT)
From: [email-masked]
Subject: Re: text/html an abomination
To: Dave Crocker <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> At 11:43 PM 5/5/2002 -0400, Keith Moore wrote:
> >that wouldn't bother me a bit if such messages didn't have so much gratuitous
> >markup.  in the vast majority of html messages I see, the html contains no
> >more
> >information than could have been conveyed in plain text, but has so much
> >cruft in it that it essentially cannot be read as text.

> When people first get access to mechanisms for formatting, fonts, etc.,
> they tend to overuse it.  Over time, they learn to be more judicious.

This is quite true but I don't believe it is relevant. The problem with HTML
isn't users using formatting in it excessively. The problem is that HTML
producing agents produce entirely gratuitous junk in their HTML output
even when there's no fancy formatting present at all.

> Formatting that is done well aids the communication, rather than detracting
> from it, and adds another dimension to the communication.

> Let's not damn the capability just because some people have not yet learned
> to use it.

Users are one thing, software is another. The inability to write software
that outputs minimalist markup is well past the point of being amusing.

> The idea that email content should be required to have less visual control
> than a web page or a document simply doesn't make any sense.

Web page designers regularly have to clean up the lousy HTML generated by
various utilities. But of course this doesn't happen with email...

				Ned

To: [email-masked]
Subject: Re: text/html an abomination
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 07 May 2002 19:43:53 -0700
Lines: 26
Sender: [email-masked]

Simon Josefsson <[email-masked]> writes:

> This looks rather sloppy to me.  It seems to say "Our HTML parser is
> insecure, but if you as a user is ready to take on the blame if any
> problems appear, just click here".  Users in general can't make security
> decisions, and they should not be able to sign the security of their
> system away by simply "clicking here".

I dunno, I think it falls into the category of not lying to the user.  :)
How many people really think that their HTML parser is completely secure?
Have you audited the code?  *wry grin*

If the code is too long for me to read and assure myself that it has no
security vulnerabilities, it probably has them, and therefore caution in
exposing it to any untrusted data is worthwhile.  I think that the warning
could be clearer about that, perhaps, but I'd still say something along
the lines of:

    This message is in HTML.  Formatting HTML for viewing can be slow
    and may let the HTML code do things that you do not expect, so the
    raw HTML code of this message is displayed below.  If you trust
    the sender of this HTML message and want to see the formatted view
    of this message, click on this box to render it.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: All these lonely accents, where do they all come from? 
Date: Tue, 07 May 2002 22:43:02 -0400
Sender: [email-masked]

> Text doesn't appear out of thin air, Keith.

thank you for that brilliant insight!
 
> Please explain exactly where you think unnormalized UTF-8 text enters
> the world's computer systems.

well for starters, how about the reason that Unicode has multiple 
representations for characters in the first place - because they wanted 
to support invertable translation to and from legacy character sets without 
information loss, and also because they combined several character sets 
which had some non-null intersections but there were cases where it made 
sense to assign different codepoints.  

then there are existing editors and keyboard drivers that don't generate 
normalized text.  for all I know, it may not even be desirable to do so  -
there may be some languages for which this would be seen as a defect.
I'm not going to second-guess the designers of charsets for other languages 
(or Unicode for that matter) and say that all text in those charsets should 
always be normalized.

as for why those legacy charsets supported combining characters - you'll 
have to ask the designers of those charsets. 

> Then explain why you think normalization should be put into every
> program that _handles_ text, instead of the much smaller set of programs
> (notably, keyboard interfaces) that _produce_ text.

I didn't say that.  I just said that putting the normalizations in the
set of programs that produce text was beyond our ability to influence,
and that it wouldn't be sufficient anyway.

Keith 

p.s. as far as I know this one is a new topic for us.  
want to write up a web page on it?

To: [email-masked]
Xref: clerew local.mime:1434
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: local-part conventions and enhancements
Date: Tue, 7 May 2002 17:06:04 GMT
Lines: 29
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>the theory is that there is benefit in extending this model for other 
>parametric local-part situations.  the clear requirement, however, is 
>backward compatibility so that the more popular free-form local-part continues.

Well if you want to make it easier to extend the existing headers with new
functionalities, then may I draw your attention to what Usefor is doing.
We have decreed that ALL headers may include MIME-style parameters (i.e.
may be followed by "; foo=value"), where parameters are to be ignored if
not having a recognized meaning.

Actually, we have made little use of this feature for now. It is intended
to allow unheard of extensions in the future. So it is a MUST accept (and
ignore for now) but SHOULD NOT generate (so as not to confuse present
agents). And we took care not to require it in the critical mail headers
like From and Reply-To (where there are slight parsing difficulties
anyway). An investment for the long term future.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1433
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
Date: Tue, 7 May 2002 16:59:20 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Marc Mutz <[email-masked]> writes:

>On Monday 06 May 2002 10:05, Simon Josefsson wrote:
><snip>
>> You can use MIME parts for this as well, mark the parts as inline and make
>> sure you don't add newlines during encoding.  The processing rules
>> describes fairly well how to decode it back together.
><snip>

>Euhm? Where did you read this? The only rfc I know of that defines 
>presentation of mails is rfc2183, defining the C-D header (plus others that 
>add new C-D values). RFC2183 basically interprets "inline" as "non-iconic" or 
>"expanded". In HTML speak this means <div>, not <span>.

Well Turnpike can do this trick, and they swear blind that the behaviour
in question is mandated by the MIME standards.

That view does not seem to be universally shared, however :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: proposal for a truce
Date: Tue, 07 May 2002 21:53:13 -0400
Sender: [email-masked]

> Handling DSN as an SMTP extension was a serious error. DSN was designed
> to achieve something users don't care about, namely guaranteeing an ``I
> dunno'' response when no other response was forthcoming, at the expense
> of what the users actually wanted, namely return receipts.

You know, we really shouldn't have these debates over and over on
email.  It's tiring and wastes time for both of us and for other 
people on the list who have heard it all before.

I propose a truce: Why don't you write up a web page for each of 
the things that I did that you want to gripe about.  For each of
those (okay, some subset of those) I'll either write up a rebuttal 
or write a page that explains things from my point-of-view.

I'll link to your page on that topic from my page on the same topic, 
and you link to mine from yours.  The links themselves should be
value-neutral (e.g. "Dan's/Keith's opinion on this topic") and
plainly visible within the document, but of course the rest of 
the content would be up to the author.

Then when the subject comes up in future discussions, one or both of
us will just send a pointer to his URL unless there's something new
to say.  If there's something new to say, we'll just post the new thing
with a minimum of re-elaboration and then add *that* to the web page. 

Of course, it should be symmetrical: the same arrangement would apply 
to things that you've done that I want to gripe about.  

You probably have some of those pages written already.
You can let me know by private email which ones those are.

The main thing we'd each be agreeing to do is:

a) link to the other's page if/when there is one
b) not re-iterate the same argument in public email unless there's 
   something new to say

I propose we try it for six months and see how well it works.

Deal?

Keith

Date: 8 May 2002 00:48:36 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: All these lonely accents, where do they all come from?
Sender: [email-masked]

Keith Moore writes:
> some text is generated by computer programs,
> and we can't force those programmers to always generate normalized text

Text doesn't appear out of thin air, Keith.

Please explain exactly where you think unnormalized UTF-8 text enters
the world's computer systems.

Then explain why you think normalization should be put into every
program that _handles_ text, instead of the much smaller set of programs
(notably, keyboard interfaces) that _produce_ text.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 8 May 2002 00:37:50 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: DSN redesign
Sender: [email-masked]

Eric A. Hall writes:
> Secondarily, some of the recipient-specific
> fields can't be implemented as shared header fields

Have one field for each recipient, naming the recipient:

   Notice-Requested-Upon-Delivery-To: [email-masked]

Notices to the envelope sender, of course. This is not rocket science.

Handling DSN as an SMTP extension was a serious error. DSN was designed
to achieve something users don't care about, namely guaranteeing an ``I
dunno'' response when no other response was forthcoming, at the expense
of what the users actually wanted, namely return receipts.

By the way, the guarantee was destroyed by the ``noreceipts'' option in
sendmail 8.6.10 through 8.9.3: sendmail accepted DSN requests and then
threw them away.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 07 May 2002 17:16:07 -0700 (PDT)
From: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
To: Keith Moore <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Sender: [email-masked]

> > Surely, the lesson to be learned is that Codecs are more trouble than they
> > are worth, and that sooner or later we need to have a Final Solution to
> > the problem.

> surely the lesson to be learned is that those who don't understand history
> are doomed to repeat past mistakes.  "Final Solutions" are neither.

Actually, assuming there are "final solutions" is one of the things history
tells is almost always a major, hideous blunder.

				Ned

Date: 7 May 2002 23:43:26 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Keith Moore vs. UTF-8: the battle begins
Sender: [email-masked]

Keith 7-bits-forever Moore writes:
> We're a long way from discovering a Universal Format for everything
> (and no, UTF-8 is NOT it

Don't hold back, Keith. Tell us exactly what's wrong with UTF-8.

> backward compatibility

What exactly is the problem, Keith? You're unhappy that UTF-8 software
won't read your EBCDIC files? Or that your 36-bit computers need new
software to handle UTF-8 data?

> only applicable to a limited kind of text)

What are you talking about, Keith?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 07 May 2002 14:58:37 -0700
To: Keith Moore <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: authenticating the source of mail 
Cc: Keith Moore <[email-masked]>, "Eric A. Hall" <[email-masked]>, [email-masked]
Sender: [email-masked]

At 05:01 PM 5/7/2002 -0400, Keith Moore wrote:
> > ISPs have their own incentives.  They care about complaints from their
> > customers.  They do not care about much more.
>
>ISPs care about profit, which equates to cost and market share.
>Figure out a way to significantly reduce cost of dealing with spam
>and/or to increase market share and ISPs will be interested.
>I think I have a handle on the former.

my experience with ISPs -- based on a subscript-based volume email startup 
that I work with -- is that spam is only interesting to them in terms of 
customer complaints.  Some will complain about bandwidth consumption but 
they do not seem to work very hard at 'fixing' that problem directly.  (I 
just heard a second-hand hotmail statistic that says 95% of the email they 
receive is spam.  Hard to imagine such a volume.)

> > It is mail originators and recipients who are the "participants" that must
> > press for the mechanisms.
>
>ISPs participate also.  Their costs of dealing with both spam and viruses
>are significant.

Their cost is relatively minor, unless it threatens their customer base.

>   A number of ISPs are now using filters or interception
>proxies to control mail flow to give them a hook to monitor and/or filter
>traffic for spam and viruses.

my point is that they do this more for marketing advantage than for 
operations survival.

i don't view the marketing basis as bad, but merely something that 
clarifies the motives and incentives.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Tue, 07 May 2002 17:57:54 -0400
Sender: [email-masked]

> Indeed. Each time we introduce a new Codec, we introduce new
> interoperability problems and disruption (usually because it gets
> implemented wrong, or not at all). Looks like IDNA will gives us yet
> another Codec, and we also need one for the local-part (though hopefully
> we can reuse an existing one there).

one interesting aspect of IDNA - if you don't implement that codec
right then *none* of your applications work.  you'll figure this out
pretty quickly.
 
> But where will this end? What more Codecs will be needed in future?

I don't see the end to them anytime soon.  We're a long way from discovering
a Universal Format for everything (and no, UTF-8 is NOT it - its flaws are 
already quite obvious, it too suffers from the backward compatibility problem,
 and it's only applicable to a limited kind of text)

> Surely, the lesson to be learned is that Codecs are more trouble than they
> are worth, and that sooner or later we need to have a Final Solution to
> the problem. 

surely the lesson to be learned is that those who don't understand history 
are doomed to repeat past mistakes.  "Final Solutions" are neither.

Keith

From: Keith Moore <[email-masked]>
To: Marc Mutz <[email-masked]>
cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Tue, 07 May 2002 17:37:56 -0400
Sender: [email-masked]

> So essentially, you'd have to:
> - - encode to (shortest possible) UTF-8
> - - apply base64 encoding
> - - prepend "=?utf-8?b?"
> - - append "?="

*canonicalized/normalized*, UTF-8, not shortest possible UTF-8.  
and then you'd have a different encoding for the local-part
than for the domain (which will surely be IDNA).

and it still would break in the face of agents that decode
and/or translate encoded-words.

it's just not going to happen this way.

Keith

To: [email-masked]
Xref: clerew local.mime:1432
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
Date: Tue, 7 May 2002 16:57:42 GMT
Lines: 29
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>sure, but the evidence also shows that new formats and protocols introduce
>interoperability problems and disruption.

Indeed. Each time we introduce a new Codec, we introduce new
interoperability problems and disruption (usually because it gets
implemented wrong, or not at all). Looks like IDNA will gives us yet
another Codec, and we also need one for the local-part (though hopefully
we can reuse an existing one there).

But where will this end? What more Codecs will be needed in future?

Surely, the lesson to be learned is that Codecs are more trouble than they
are worth, and that sooner or later we need to have a Final Solution to
the problem. UTF-8 is such a Final Solution. It may not solve the
immediate problem (which is that we have to deal with IDNA NOW), but it
will sure pay off in the long term.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
cc: Keith Moore <[email-masked]>, "Eric A. Hall" <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail 
Date: Tue, 07 May 2002 17:01:55 -0400
Sender: [email-masked]

> mandatory simply won't work.  there is no global authority agency to
> enforce mandatory anything.  it's participant incentives or it's nothing.

that's my assessment also. 
 
> >for instance, if major ISPs processed authenticated mail faster than
> >non-authenticated mail,
> 
> ISPs have their own incentives.  They care about complaints from their
> customers.  They do not care about much more.  

ISPs care about profit, which equates to cost and market share.
Figure out a way to significantly reduce cost of dealing with spam
and/or to increase market share and ISPs will be interested.
I think I have a handle on the former.  

> It is mail originators and recipients who are the "participants" that must
> press for the mechanisms.

ISPs participate also.  Their costs of dealing with both spam and viruses
are significant.  A number of ISPs are now using filters or interception
proxies to control mail flow to give them a hook to monitor and/or filter
traffic for spam and viruses.  (perhaps also for government surveillance, 
though I'm not sure about this)   I don't like interception proxies, but
this tells me that some ISPs are willing to invest some in infrastructure 
to discourage such abuse.
 
> >the other trick is building an infrastructure that allows authenticated
> >mail to be verified and traced to the source, and one which facilitates
> >quick reporting and suppression of sources of abusive mail.
> 
> as long as it has incremental deployment and use, and incremental value,
> then maybe it is worth considering.

agreed that these are necessary conditions.

Keith 

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: authenticating the source of mail 
Date: Tue, 07 May 2002 16:55:13 -0400
Sender: [email-masked]

> > for instance, if major ISPs processed authenticated mail faster than
> > non-authenticated mail, this would provide some incentive for sites
> > to authenticate.
> 
> Spam doesn't go away, it just gets delayed. Spammers don't care if the
> spam is delayed. 

yes, but virus writers do care, because it affects how fast their
viruses propagate, and how likely they are to be stopped.

> Meanwhile, the people that do care are pissed that they
> have to go through extra effort to get past the asshole postmaster.

you can probably make the amount of delay volume-sensitive so that 
occasional messages are not sigificantly delayed.  and the major ISPs 
might sign on just to reduce their costs.

> > similarly, if businesses refused to accept any non-text mail that
> > wasn't authenticated (bouncing such messages or issuing MDNs saying
> > an attachment was deleted) then sites would have a reason to
> > authenticate.
> 
> This is getting closer, but I don't see businesses wanting to even imply
> that mail from potential customers is not going to be accepted. 

no but they might impose controls in practice.

> What if a customer's mail client send multipart/alternative by default?

it depends on what the alternatives contain.
 
> These items are useful for increasing the visibility of signed mail, which
> is certainly important with an elective system. 

perhaps, but that was not exactly my intent.  the point is that while mail 
cannot be trusted in general, it's deserving of somewhat more trust if
you can reliably trace the mail to a source, especially if you can verify
that that source has agreed to accept responsibility for not sending abusive 
mail (viruses and/or spam).

> However, I don't really
> know how practical any of these measures are (see above), 

the devil is in the details.  small changes can significantly affect the
practicality.

> Also, consider things like network appliances that send activity alerts
> without signatures, so there will always be a high percentage of mail
> which is unlikely to be signed for decades to come. 

the signature doesn't have to be end-to-end, as long as you can trace
it to a party that accepts responsibility for it.

> Not harping, but that is one of the benefits of a new model. The default
> scenario is that mail is trustworthy, so stuff from outside the network is
> automatically presumed to be untrustworthy. 

you still have the burden of getting it deployed.  is it easier to
deploy a completely new mail system along with a security infrastructure
that lets you verify the source of a message, or just to deploy a security 
infrastructure that works with the existing mail system?

either way, it makes more sense to thinking about what problem to solve
than it does to start with a solution and look for problems.

of course, this isn't even close to a fully-fleshed-out proposal.
but perhaps other people would also like to think about solving the 
virus and spam problem.  we've already tried clamping down on open 
relays, IP and domain blacklists, spam and virus filters, and they're 
not working terribly well.  we need something else.

Keith

Date: Tue, 07 May 2002 13:37:13 -0700
To: Keith Moore <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: authenticating the source of mail
Cc: "Eric A. Hall" <[email-masked]>, Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

At 03:38 PM 5/7/2002 -0400, Keith Moore wrote:
> > In other words, I don't think that forgery problems can be solved until
> > the solutions for them are mandatory.

encryption-based authentication is resistant to forgery.  anything that 
requires nothing more than administrative conformance on a global scale is 
simply doomed; forgery isn't just easy, it's trivial.

>I'm not sure whether it's necessary that the solutions be mandatory,
>or whether it's sufficient that there be some incentives to using them.

mandatory simply won't work.  there is no global authority agency to 
enforce mandatory anything.  it's participant incentives or it's nothing.

>for instance, if major ISPs processed authenticated mail faster than
>non-authenticated mail,

ISPs have their own incentives.  They care about complaints from their 
customers.  They do not care about much more.  And as noted, spammers do 
not care about delays.

It is mail originators and recipients who are the "participants" that must 
press for the mechanisms.

>  this would provide some incentive for sites
>to authenticate.

serious businesses are easy.  it is the miscreants who are the problem.

>the other trick is building an infrastructure that allows authenticated
>mail to be verified and traced to the source, and one which facilitates
>quick reporting and suppression of sources of abusive mail.

as long as it has incremental deployment and use, and incremental value, 
then maybe it is worth considering.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

Date: Tue, 07 May 2002 15:21:20 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: authenticating the source of mail
Sender: [email-masked]

Keith Moore wrote:

> for instance, if major ISPs processed authenticated mail faster than
> non-authenticated mail, this would provide some incentive for sites
> to authenticate.

Spam doesn't go away, it just gets delayed. Spammers don't care if the
spam is delayed. Meanwhile, the people that do care are pissed that they
have to go through extra effort to get past the asshole postmaster.

> similarly, if businesses refused to accept any non-text mail that
> wasn't authenticated (bouncing such messages or issuing MDNs saying
> an attachment was deleted) then sites would have a reason to
> authenticate.

This is getting closer, but I don't see businesses wanting to even imply
that mail from potential customers is not going to be accepted. What if a
customer's mail client send multipart/alternative by default?

These items are useful for increasing the visibility of signed mail, which
is certainly important with an elective system. If you can get signed mail
to a usage level where people think that unsigned mail is weird, then it
becomes feasible to start enforcing signatures. However, I don't really
know how practical any of these measures are (see above), so I'm not sure
that they will actually encourage an increase in the use of signed mail.

Also, consider things like network appliances that send activity alerts
without signatures, so there will always be a high percentage of mail
which is unlikely to be signed for decades to come. This means that
enforcing signatures within the elective infrastructure will always be
undoable as a baseline filter, and that the necessary perception levels
will not be reached.

Not harping, but that is one of the benefits of a new model. The default
scenario is that mail is trustworthy, so stuff from outside the network is
automatically presumed to be untrustworthy. Even the SMTP client in your
disk array is no longer believable. Unsigned mail is weird from the very
beginning.

> the other trick is building an infrastructure that allows
> authenticated mail to be verified and traced to the source, and one
> which facilitates quick reporting and suppression of sources of
> abusive mail.

Yeah, the suppression part is the principle merit of verified recieved
headers. If the evidence cannot be easily forged, then it makes the
remaining information believable as evidence, and more easily actionable.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: authenticating the source of mail
Date: Tue, 07 May 2002 15:38:01 -0400
Sender: [email-masked]

> In other words, I don't think that forgery problems can be solved until
> the solutions for them are mandatory.

I'm not sure whether it's necessary that the solutions be mandatory,
or whether it's sufficient that there be some incentives to using them.

for instance, if major ISPs processed authenticated mail faster than 
non-authenticated mail, this would provide some incentive for sites
to authenticate.  similarly, if businesses refused to accept any 
non-text mail that wasn't authenticated (bouncing such messages
or issuing MDNs saying an attachment was deleted) then sites would
have a reason to authenticate.

the other trick is building an infrastructure that allows authenticated 
mail to be verified and traced to the source, and one which facilitates 
quick reporting and suppression of sources of abusive mail.

there would be LOTS of details to work out, and I don't pretend to 
even skim the surface here.

Keith

From: Marc Mutz <[email-masked]>
Organization: KDE
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Tue, 7 May 2002 21:12:16 +0200
Content-Description: clearsigned data
Sender: [email-masked]

On Monday 06 May 2002 18:03, Charles Lindsey wrote:
<snip>
> So perhaps we should still be looking at other possibilities for local
> part, such as RFC 2047. Yes that needs care to ensure it is done
> canonically, but that is just a matter of insisting that the whole
> local-part be treated as exactly one encoded-word.

A base64-encoded UTF-8 encoded-word.

Else, you will get ambiguity arising from different implementations encoding 
different octets and in different charsets. E.g., some might encode latin1 
space as =20, others as _.

So essentially, you'd have to:
- - encode to (shortest possible) UTF-8
- - apply base64 encoding
- - prepend "=?utf-8?b?"
- - append "?="

OTOH, encoding to "shortest possible UTF-8" surely involves some kind of 
normalization of the Unicode representation of the local-part. So we're back 
where the large tables come into the game...

But at least a heap of current mailers would _display_ the stuff without 
modifiications...

Marc

- -- 
Marc Mutz <[email-masked]>

To: Marc Mutz <[email-masked]>
Cc: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: text/html an abomination
From: Simon Josefsson <[email-masked]>
Date: Tue, 07 May 2002 21:06:27 +0200
Lines: 21
Sender: [email-masked]

Marc Mutz <[email-masked]> writes:

> On Tuesday 07 May 2002 04:20, Russ Allbery wrote:
>> Lawrence Greenfield <[email-masked]> writes:
> <snip>
>> (I'm surprised that more mail clients
>> don't default to showing the raw HTML with a button saying "render this";
>> users wouldn't press it for spam and it would make discarding spam quicker
>> as well as safer.)
> <snip>
>
> Like so?: http://www.mathematik.uni-bielefeld.de/~mmutz/html-warning.png

This looks rather sloppy to me.  It seems to say "Our HTML parser is
insecure, but if you as a user is ready to take on the blame if any
problems appear, just click here".  Users in general can't make
security decisions, and they should not be able to sign the security
of their system away by simply "clicking here".

It would be another thing if it said "For efficiency reasons, only the
raw HTML code is shown.  If you want to render it, click here".

From: Marc Mutz <[email-masked]>
Organization: KDE
To: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: text/html an abomination
Date: Tue, 7 May 2002 20:50:17 +0200
Content-Description: clearsigned data
Sender: [email-masked]

On Tuesday 07 May 2002 04:20, Russ Allbery wrote:
> Lawrence Greenfield <[email-masked]> writes:
<snip>
> (I'm surprised that more mail clients
> don't default to showing the raw HTML with a button saying "render this";
> users wouldn't press it for spam and it would make discarding spam quicker
> as well as safer.)
<snip>

Like so?: http://www.mathematik.uni-bielefeld.de/~mmutz/html-warning.png

Marc

- -- 
Marc Mutz <[email-masked]>

Date: Tue, 07 May 2002 13:14:36 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> If the MDA doesn't understand the extension, then delivery must fail

> it's not clear why such an option would be useful or implemented

I'm having trouble thinking of a good applicaiton for this, too. However,
if there is a need for warnings than an extension was not understood, then
there is a pretty good chance that there is a need for insisting that
delivery must fail, allowing the sender to change the options. There are
probably some good usages for this feature in the groupware space but I
haven't really thought about it. "Rescind" is a possibility, although I
would expect that a warning would be sufficient for that.

A trickier proposition is with the ~client settings. What do you do if an
extension listed in the Client-Alert isn't supported? Delete the mail?
Power down the PC?

> really wish that people would think about how to make more reliable,
> virus-proof, and spam-proof, and about how to solve other problems of
> a similar severity, rather than trying to justify designing a new
> protocol by promising new features of dubious value.

Recursive signatures of the envelope solve a big part of the spam problem
in that they (theoretically) make it impossible to lie about the transfer
path. As to whether or not it would be faster and easier to change SMTP to
add this feature is certainly open to debate. Personally I feel that it
would be easier to leverage this as part of a new system if it were a
mandatory part of deploying a new protocol. I don't think it is feasible
to mandate that all SMTP servers must support feature X by some date, but
it is feasible to mandate that some new protocol must support feature X
from the beginning.

In the same vein, it is possible to require that the content header blobs
must be signed if some new protocol/format is used, whereas S/MIME is and
always will be optional.

In other words, I don't think that forgery problems can be solved until
the solutions for them are mandatory.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Tue, 07 May 2002 13:56:08 -0400
Sender: [email-masked]

> However, if the extension only needs the support of the delivery agent,
> then it can demand that MDAs participate by listing the extension as a
> Delivery-Alert, and exclude the extension from the Transfer-Alert list. If
> the MDA doesn't understand the extension, then delivery must fail (this
> option is not available in SMTP today). 

it's not clear why such an option would be useful or implemented - nobody 
wants to cause a delivery to fail when the message has arrived at its
destination.

if we had the luxury of starting from scratch, I could see some benefit
of having the envelope protocol support a "delivery options" container 
that wasn't examined until final delivery, perhaps even with criticality
flags.  but experience with x.400 shows these things to be of limited 
utility.  so I don't think it really buys us very much, and I don't see
how whatever it would buy is likely to be worth the cost of transition.
that, and I don't even think a transition is feasible.

Keith

p.s. I realize that protocol engineers like to design new protocols. but I 
really wish that people would think about how to make more reliable,
virus-proof, and spam-proof, and about how to solve other problems of 
a similar severity, rather than trying to justify designing a new 
protocol by promising new features of dubious value.  Of course, these
are much harder problems to solve, and they involve not just protocol
design but also an understanding of operations and even politics,
which may explain why folks are reluctant to tackle them.  However,
this is where we need to be applying our efforts.

Date: Tue, 07 May 2002 12:46:50 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:
> 
> > > so the message gets to its destination, and the delivery report
> > > feature isn't supported, so an error is triggered.  since the
> > > destination doesn't know what the request is the only reasonable
> > > thing to do is to bounce the message.
> >
> > SMTP extensions work that way today.
> 
> no they don't.  the MTA that realizes that the next hop doesn't support
> the required feature is one that understands what the feature is.
> therefore it can downgrade, and this is generally what happens.

This exact same functionality would still be supported using a
Transfer-Alert. Delivery-Alerts would only be processed by the MDA, while
transfer alerts would be processed at each hop, exactly as now.

> With the model you're proposing the destination has no idea what to do
> with the feature.  A generic warning isn't necessarily appropriate -
> so specifying that behavior further limits the utility of the model.

The protocol (the extension) has to take the possible results into
consideration when it is designed.

If the extensions requires the active participation of the transfer
intermediaries (as with NOTIFY=SUCCESS does for the Relay action), then it
can demand that the MTAs participate by listing the extension as a
Transfer-Alert. If an MTA doesn't understand the extensions listed in the
Transfer-Alert, then that transfer hop must fail (this is exactly like
SMTP today). The extension may dictate that the current holder of the
message would then return a Relayed action notification, clear the
extension from the Transfer-Alert list, and retransmit, exactly like DSNs
currently behave.

Optionally, the protocol could specify that this is an elective extension,
which benefits from the MTA path but which does not require their
participation. At that point, the extension is listed in the extensions
list but not in the Alert list, thereby producing warnings instead of
failures.

However, if the extension only needs the support of the delivery agent,
then it can demand that MDAs participate by listing the extension as a
Delivery-Alert, and exclude the extension from the Transfer-Alert list. If
the MDA doesn't understand the extension, then delivery must fail (this
option is not available in SMTP today). Meanwhile the MTAs in the transfer
path would just ignore the extension.

This would make it possible to fully mimic the DSN extension (including
failures if the extension was not supported by all nodes in the transfer
path). HOWEVER there would be at least two new benefits. First is that
extensions which only apply to delivery (such as a "rescind" capability)
can be deployed without requiring the entire transfer path to be upgraded
beforehand. Secondarily, we can also deploy optional extensions that would
be sufficient with warnings rather than failures (maybe the rescind
extension doesn't need to trigger a delivery failure, but only needs to
trigger a delivery warning).

There is some danger of runaway envelope tags here if the delineation of
functions gets out of hand. I see the three important extensions being for
Transfer, Delivery and ~Client. So that would be six different envelope
tags (*-Options and *-Alert), assuming that extensions were defined within
any given message for all of them.

It would also mean that MDNs could be supported as Client-Options, with
the envelope data being used to seed the necessary information (eg,
OriginalRecipient), rather than relying on delivery servers to generate
the supporting header fields as is the current model.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: [email-masked], Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoftye 
Date: Tue, 07 May 2002 13:17:47 -0400
Sender: [email-masked]

> > I explained in detail why this is simply not the case
> 
> I know of no way to provide errors in the current model. You can produce
> cataclysmic failures with extensions, and you can produce silent errors
> with header fields, but you cannot produce a warning.

It's just the opposite.  In the current model an MTA can downgrade when the
next hop doesn't support the feature.  The degree to which this is appropriate
depends on the particular feature, but the negotiation mechanism makes
this possible.

With the model you're proposing the destination has no idea what to do
with the feature.  A generic warning isn't necessarily appropriate - 
so specifying that behavior further limits the utility of the model.

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Tue, 07 May 2002 13:15:12 -0400
Sender: [email-masked]

> > so the message gets to its destination, and the delivery report
> > feature isn't supported, so an error is triggered.  since the
> > destination doesn't know what the request is the only reasonable
> > thing to do is to bounce the message.
> 
> SMTP extensions work that way today. 

no they don't.  the MTA that realizes that the next hop doesn't support
the required feature is one that understands what the feature is.
therefore it can downgrade, and this is generally what happens.

Keith

Date: Tue, 07 May 2002 10:55:11 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoftye
Sender: [email-masked]

[email-masked] wrote:

> I explained in detail why this is simply not the case

I know of no way to provide errors in the current model. You can produce
cataclysmic failures with extensions, and you can produce silent errors
with header fields, but you cannot produce a warning.

Well, you *can* if you are willing to write another envelope extension and
have all of the MTAs deploy it.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 7 May 2002 08:55:10 -0700 (PDT)
From: Bart Schaefer <[email-masked]>
To: [email-masked]
Subject: Re: Dreaming about replacement
Sender: [email-masked]

On Tue, 7 May 2002, Eric A. Hall wrote:

> Addding seven more layers of bubblegum and baling wire to the existing
> service will not be as efficient or effective as designing for these
> features from a blank slate.

There hasn't been anything new said in this thread for several days now.

Go ahead, design this wonderful new service, and on the day it supplants
SMTP we'll all buy you a nice dinner.

Date: Tue, 07 May 2002 08:52:48 -0700 (PDT)
From: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
To: "Eric A. Hall" <[email-masked]>
Cc: Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Sender: [email-masked]

> > no it would have exactly the same problem - there's no way to know
> > whether the destination MTA supports the feature, so there's no way
> > to guarantee a delivery report.

> That's what the Delivery-Alert list is for. If the listed feature isn't
> supported, an error is triggered. This is substantially different from
> having header fields be ignored as would be the case in the scenario
> proposed by Paul -- that would be a silent error.

You obviously aren't reading the things other people say, preferring to argue
by repeating points like this one that have already been refuted.

A meaningful discussion cannot be had on these terms. I'm done here.

				Ned

Date: Tue, 07 May 2002 08:48:29 -0700 (PDT)
From: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
To: "Eric A. Hall" <[email-masked]>
Cc: Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Sender: [email-masked]

> Keith Moore wrote:

> > > >you have no idea about whether the ultimate destination MTA
> > > >supports such a feature.
> > >
> > > Agreed - but the same would apply to an 'envelope blob' mechanism
> > > (which is what we were talking about)
> >
> > right, but the envelope blob mechanism wouldn't be appropriate for
> > requesting delivery notifications either.

> Sure it would. In fact, an trace-level envelope flag was provided in the
> list of features I want, which has delivery success as one of the
> standardized values. I'm not sure about the appropriateness of that, but
> DSNs are very desirable as a standardized element. Even if it wasn't part
> of the core service, it would be simple to add as an envelope extension.

I explained in detail why this is simply not the case and why you cannot
usefully add success delivery notifications later as an extension of the sort
you describe. Similiar analyses apply to other extensions done this
way. This is why envelope blobs are nowhere near as useful as you persist
in claiming.

				Ned

Date: Tue, 07 May 2002 10:50:29 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> so the message gets to its destination, and the delivery report
> feature isn't supported, so an error is triggered.  since the
> destination doesn't know what the request is the only reasonable
> thing to do is to bounce the message.

SMTP extensions work that way today. The difference is that options not in
the alert list can produce a warning rather than a failure.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 07 May 2002 08:45:09 -0700
To: Keith Moore <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: text/html an abomination 
Cc: Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

At 11:35 AM 5/7/2002 -0400, Keith Moore wrote:
>fair enough.  but surely you'll grant that
>"automatic inclusion of text/html with gratuitous markup an abomination"

as a subject line, it might qualify as an abomination.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Tue, 07 May 2002 11:45:32 -0400
Sender: [email-masked]

> > no it would have exactly the same problem - there's no way to know
> > whether the destination MTA supports the feature, so there's no way
> > to guarantee a delivery report.
> 
> That's what the Delivery-Alert list is for. If the listed feature isn't
> supported, an error is triggered. This is substantially different from
> having header fields be ignored as would be the case in the scenario
> proposed by Paul -- that would be a silent error.
 
so the message gets to its destination, and the delivery report feature
isn't supported, so an error is triggered.  since the destination doesn't
know what the request is the only reasonable thing to do is to bounce the
message.

dream on.

Keith

Date: Tue, 07 May 2002 10:43:54 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> no it would have exactly the same problem - there's no way to know
> whether the destination MTA supports the feature, so there's no way
> to guarantee a delivery report.

That's what the Delivery-Alert list is for. If the listed feature isn't
supported, an error is triggered. This is substantially different from
having header fields be ignored as would be the case in the scenario
proposed by Paul -- that would be a silent error.

> > (eg, encrypt the content headers as a blob, sign the transfer
> > headers as a blob).
> >
> > You can't do that very easily with the existing setup.
> 
> sure you can.  all you have to do is prepend
>
> content-type: message/rfc822<CRLF><CRLF>

In the current model, you cannot ensure that all of the systems in the
etransfer path support the necessary feature without imposing an
additional break-point (such as a mandatory extension). This is
significantly more difficult than designing a protocol to use such a
mechanism by default.

> I must say that these arguments in favor of a new format aren't
> getting any more persuasive - all they're doing is demonstrating
> that we aren't even close to needing one.

Addding seven more layers of bubblegum and baling wire to the existing
service will not be as efficient or effective as designing for these
features from a blank slate.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: text/html an abomination 
Date: Tue, 07 May 2002 11:35:31 -0400
Sender: [email-masked]

> >i.e. it's not that people are making excessive use of boldface, font
> >changes, etc - it's that implementors are adding gratuitous markup
> >that typically wasn't specified by the author of the message and
> >adds absolutely nothing to the displayed output.
> 
> here, you offer a rather more constrained and pragmatic criticism than the
> text in the subject line or your previous comment.
>
> A call for more conservative use of 'automatic' html by implementors seems
> downright reasonable.

fair enough.  but surely you'll grant that 
"automatic inclusion of text/html with gratuitous markup an abomination" 
doesn't scan as well?

:)

Keith

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, Paul Smith <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Tue, 07 May 2002 11:34:25 -0400
Sender: [email-masked]

> > > >you have no idea about whether the ultimate destination MTA
> > > >supports such a feature.
> > >
> > > Agreed - but the same would apply to an 'envelope blob' mechanism
> > > (which is what we were talking about)
> > 
> > right, but the envelope blob mechanism wouldn't be appropriate for
> > requesting delivery notifications either.
> 
> Sure it would. 

no it would have exactly the same problem - there's no way to know whether
the destination MTA supports the feature, so there's no way to guarantee
a delivery report.

> The original intention is two-fold, first is to separate the transfer and
> content headers so that they can be handled separately (eg, encrypt the
> content headers as a blob, sign the transfer headers as a blob).
> 
> You can't do that very easily with the existing setup.

sure you can.  all you have to do is prepend

content-type: message/rfc822<CRLF><CRLF>

to the message.  of course, existing UAs may not handle this transparently.
but then again, they won't handle your proposal transparently either.

I must say that these arguments in favor of a new format aren't getting 
any more persuasive - all they're doing is demonstrating that we aren't
even close to needing one.

Keith

Date: Tue, 07 May 2002 16:21:14 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
Sender: [email-masked]

At 09:44 07/05/2002 -0500, Eric A. Hall wrote:
>Paul Smith wrote:
> >
> > At 10:43 04/05/2002 -0500, Eric A. Hall wrote:
> > >DSNs with NOTIFY=SUCCESS are an obvious example. The only thing I care
> > >about is that delivery succeeds, meaning that this is a negotation
> > >between the originator and the delivery server.
>
> > Well, the OBVIOUS answer to that, is to take the DSN information out of
> > the envelope and put it into the header.
>
>Couple of problems with that approach. First is that delivery servers
>would have to start poking through the headers of all incoming messages
>just to find out if the message required a delivery receipt.

Hmm, most servers seem to poke around in headers in any case, so checking 
for a particular header wouldn't be that much more difficult.

>That's a lot
>of work in contrast to simply being told to do so, as with the current
>model of envelope extensions.

Really? Would it more difficult than poking around in a 'blob' of extra 
envelope data?

>Secondarily, some of the recipient-specific
>fields can't be implemented as shared header fields, as they would affect
>all of the recipients. Which of the dozen Original-Recipient header fields
>is for me?

Agreed - but do MUAs regularly ask for different notify parameters for 
different recipients.

(I'm not suggesting that the current DSN system should be dumped in favour 
of putting the information in headers, I'm just saying that if the current 
failures of DSN are being used as a reason to break all existing code, then 
there are other alternatives which could be used without any disruption)

> > does this mean that all MUAs/MTAs would have to support BOTH message
> > format structures (or just the existing format..) for the forseeable
> > future?
>
>Mail readers wouldn't have to support two mailbox formats. It should be
>feasible to convert 822 mails into the new format pretty simply: dump the
>envelope into the envblob, dump the header section into the hdrblob, and
>tag it as a legacy message. The legacy body parts should render same.

Eh?

If I use MUA 'x' and I want to send a message to a 'legacy' MTA, I'd have 
to generate a message in the legacy format.

If I use MUA 'x' and I want to send a message to a 'blobby' MTA, I'd have 
to either generate the message in legacy format (and have the MTA convert 
it, but not have access to any of the new 'blobby' facilities), or generate 
the message in the new 'blobby' format.

So, as author of MUA 'x', either I have to write code to be able to 
generate both message formats, or I have to accept a restricted feature set 
and only generate legacy messages, or I have to write an MTA with an 
(initially) *very* limited customer base and go for the new format only.

Also, with your 'tagging the message as legacy' suggestion (which wouldn't 
help the above situation), the receiving MUA would need to be able to 
handle messages tagged as 'legacy' as well as those which weren't.

>The transition would require that delivery servers be capable of
>performing conversions based on the mailbox format in case a reader is
>using a legacy mail-store format.

Does this mean we'd need to have a replacement for POP3 and IMAP4 as well?

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Tue, 07 May 2002 08:15:39 -0700
To: Keith Moore <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: text/html an abomination 
Cc: [email-masked]
Sender: [email-masked]

At 08:27 AM 5/7/2002 -0400, Keith Moore wrote:
>i.e. it's not that people are making excessive use of boldface, font
>changes, etc - it's that implementors are adding gratuitous markup
>that typically wasn't specified by the author of the message and
>adds absolutely nothing to the displayed output.

here, you offer a rather more constrained and pragmatic criticism than the 
text in the subject line or your previous comment.

A call for more conservative use of 'automatic' html by implementors seems 
downright reasonable.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

Date: Tue, 07 May 2002 10:12:41 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> > >you have no idea about whether the ultimate destination MTA
> > >supports such a feature.
> >
> > Agreed - but the same would apply to an 'envelope blob' mechanism
> > (which is what we were talking about)
> 
> right, but the envelope blob mechanism wouldn't be appropriate for
> requesting delivery notifications either.

Sure it would. In fact, an trace-level envelope flag was provided in the
list of features I want, which has delivery success as one of the
standardized values. I'm not sure about the appropriateness of that, but
DSNs are very desirable as a standardized element. Even if it wasn't part
of the core service, it would be simple to add as an envelope extension.

  <Return-Path>foo@bar</Return-Path>
  <Forward-Path>bar@foo</Forward-Path>
  <Original-Recipient>baz@bar</Original-Recipient>
  <Env-Extension>dsn.cs.utk.edu (pick a namespace)</Env-Extension>
  <Delivery-Alerts>dsn.cs.utk.edu</Delivery-Alerts>

The delivery process would examine for any alerts, process them if they
were understood and supported, and return one of the (standardized)
DSN-like error responses if not. Failures also only need to use the
envelope data, and since the envelope is provided to the MUAs, the
originator system can act upon this information as appropriate.

> actually, it's not at all clear whether the envelope blob is useful
> for anything -

The original intention is two-fold, first is to separate the transfer and
content headers so that they can be handled separately (eg, encrypt the
content headers as a blob, sign the transfer headers as a blob).

With the ability to sign transfer headers, part of the problem with
verifiable mail can also be addressed. EG, we can do recursive envelope
signatures, thereby proving that the message could only have been sent
through the path it is claimed to have been sent through. This means that
faking the transfer path should become impossible.

  <Received-From>
    <Host-Id (eg an x509 subject)>submission.host.name</Host-Id>
    <Envelope-Sig>signature-of-envelope as received</Envelope-Sig>
  </Received-From>
  <Received-From>
    <Host-Identifier>originator.host.name</Host-Identifier>
    <Envelope-Sig>signature-of-envelope as received</Envelope-Sig>
  </Received-From>
  <Return-Path>foo@bar</Return-Path>
  <Forward-Path>baz@foo</Forward-Path>

Crypto is my weakpoint so I will assume that there is some problem with
the above that the experts here will point out and fix. I am aware of the
problem with key availability and have some things in mind for that.

You can't do that very easily with the existing setup.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

content-class: urn:content-classes:message
Subject: RE: BOF: Format-Flowed and Non-Western Charsets
Date: Tue, 7 May 2002 07:53:05 -0700
From: "Dan Kohn" <[email-masked]>
To: "Arnt Gulbrandsen" <[email-masked]>
Cc: "Simon Josefsson" <[email-masked]>, "Jacob Palme" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

In theory, there is no difference between theory and practice. But, in
practice, there is.
- Jan L. A. van de Snepscheut 

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>
Essays announced on <mailto:[email-masked]>

-----Original Message-----
From: Arnt Gulbrandsen [mailto:[email-masked]] 
Sent: Tuesday, May 07, 2002 01:32
To: Dan Kohn
Cc: Simon Josefsson; Jacob Palme; [email-masked]
Subject: Re: BOF: Format-Flowed and Non-Western Charsets

Dan Kohn <[email-masked]>
> Further, see that the first NOTE on page 19 of RFC 2046 is what
enables
> you (at least in theory) to have multiparts that are not broken by
CRLFs.

Yes. But, having an imapext archive handy, I looked at it now. The
x-mailer
fields from the first three messages that break the rule are:

X-Mailer: Mozilla 4.61 [en] (WinNT; U)
X-Mailer: Mulberry (MacOS) [2.0.0b2, s/n S1-000001]
X-Mailer: Mozilla 4.72 [en] (WinNT; U)

The IMAP torture test message also breaks it - ther's a GIF image you
can't decode without the trailing CRLF.

I infer that a MUA can't assume that a missing CRFL at the end of a
body-part means anything.

--Arnt

Date: Tue, 07 May 2002 09:44:32 -0500
From: "Eric A. Hall" <[email-masked]>
To: Paul Smith <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Paul Smith wrote:
> 
> At 10:43 04/05/2002 -0500, Eric A. Hall wrote:
> >DSNs with NOTIFY=SUCCESS are an obvious example. The only thing I care
> >about is that delivery succeeds, meaning that this is a negotation
> >between the originator and the delivery server.

> Well, the OBVIOUS answer to that, is to take the DSN information out of
> the envelope and put it into the header.

Couple of problems with that approach. First is that delivery servers
would have to start poking through the headers of all incoming messages
just to find out if the message required a delivery receipt. That's a lot
of work in contrast to simply being told to do so, as with the current
model of envelope extensions. Secondarily, some of the recipient-specific
fields can't be implemented as shared header fields, as they would affect
all of the recipients. Which of the dozen Original-Recipient header fields
is for me?

> does this mean that all MUAs/MTAs would have to support BOTH message
> format structures (or just the existing format..) for the forseeable
> future?

Mail readers wouldn't have to support two mailbox formats. It should be
feasible to convert 822 mails into the new format pretty simply: dump the
envelope into the envblob, dump the header section into the hdrblob, and
tag it as a legacy message. The legacy body parts should render same.

The transition would require that delivery servers be capable of
performing conversions based on the mailbox format in case a reader is
using a legacy mail-store format. The originator or submission server (in
those cases where they are the same process) would also need to perform
any auto-conversion of outbound messages which failed due to transfer
failure (and which were identifiable as such, which the enhanced envelope
should allow for).

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 07 May 2002 15:18:33 +0100
To: [email-masked]
From: Graham Klyne <[email-masked]>
Subject: Fwd: I-D ACTION:draft-klyne-hdrreg-mail-00.txt
Sender: [email-masked]

Further to the herader field registry proposal document [1], the new draft 
announced recently (announcement copy below) sketches the initial header 
field registrations for mail.   It is based on data from Jacob Palme's mail 
headers draft.

The data is still rough, and not yet entirely consistent with the 
registration proposal, but I thought it would be helpful to show how it 
works with real data.  I propose that a revised version of this draft will 
provide initial registry data for mail header fields.

The data is has also been generated as HTML: see [3].

[1] http://www.ietf.org/internet-drafts/draft-klyne-msghdr-registry-04.txt

[2] http://www.ietf.org/internet-drafts/draft-palme-mailext-headers-06.txt

[3] http://www.ninebynine.org/IETF/Messaging/HdrRegistry/Intro.html

#g
--

>A New Internet-Draft is available from the on-line Internet-Drafts 
>directories.
>
>
>         Title           : Registration of mail header fields
>         Author(s)       : G. Klyne
>         Filename        : draft-klyne-hdrreg-mail-00.txt
>         Pages           : 91
>         Date            : 06-May-02
>
>This document defines the initial IANA registration for some mail
>message header fields.
>
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-klyne-hdrreg-mail-00.txt

-------------------
Graham Klyne
<[email-masked]>

From: Keith Moore <[email-masked]>
To: Paul Smith <[email-masked]>
cc: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Tue, 07 May 2002 10:07:20 -0400
Sender: [email-masked]

> > > Well, the OBVIOUS answer to that, is to take the DSN information out of the
> > > envelope and put it into the header. This won't break any existing MTAs,
> > > and the header gets passed through and ignored any MTA which doesn't
> > > understand it.
> >
> >and you lose your assurance of getting a delivery report, because you have
> >no idea about whether the ultimate destination MTA supports such a feature.
> 
> Agreed - but the same would apply to an 'envelope blob' mechanism (which is
> what we were talking about) 

right, but the envelope blob mechanism wouldn't be appropriate for requesting
delivery notifications either.

actually, it's not at all clear whether the envelope blob is useful for anything -
arguments have mostly been of the form "it sure would be nice if..." without
explaining how that would benefit the user.

Keith

Date: Tue, 07 May 2002 14:13:34 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
Sender: [email-masked]

At 08:39 07/05/2002 -0400, you wrote:
> > >DSNs with NOTIFY=SUCCESS are an obvious example. The only thing I care
> > >about is that delivery succeeds, meaning that this is a negotation between
> > >the originator and the delivery server. The submission server and any
> > >relays in between are completely irrelevant, yet due to the current
> > >envelope-exchange design, they dictate the success of that feature.
> >
> > Well, the OBVIOUS answer to that, is to take the DSN information out of the
> > envelope and put it into the header. This won't break any existing MTAs,
> > and the header gets passed through and ignored any MTA which doesn't
> > understand it.
>
>and you lose your assurance of getting a delivery report, because you have
>no idea about whether the ultimate destination MTA supports such a feature.

Agreed - but the same would apply to an 'envelope blob' mechanism (which is 
what we were talking about) - at least putting the information into the 
header doesn't break existing software :-)

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

From: Keith Moore <[email-masked]>
To: Paul Smith <[email-masked]>
cc: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Tue, 07 May 2002 08:39:14 -0400
Sender: [email-masked]

> >DSNs with NOTIFY=SUCCESS are an obvious example. The only thing I care
> >about is that delivery succeeds, meaning that this is a negotation between
> >the originator and the delivery server. The submission server and any
> >relays in between are completely irrelevant, yet due to the current
> >envelope-exchange design, they dictate the success of that feature.
> 
> Well, the OBVIOUS answer to that, is to take the DSN information out of the
> envelope and put it into the header. This won't break any existing MTAs,
> and the header gets passed through and ignored any MTA which doesn't
> understand it.

and you lose your assurance of getting a delivery report, because you have
no idea about whether the ultimate destination MTA supports such a feature.

Keith

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Tue, 07 May 2002 08:33:53 -0400
Sender: [email-masked]

> >do you really think that OS vendors are going to rewrite *every* form
> >of input (not just keyboard drivers but copy-and-paste, files downloaded
> >from the net, etc) to do normalization?  if they did, it would break
> >far more things than it would fix.
> 
> The idea is that non-normalized stuff will never exist in files, displays
> to be copied/pasted, and the like simply because all such material
> ultimately derives from something that was keyed in on a normalizing
> keyboard.

it's a nice idea, but keyboard drivers and file formats are far removed 
from anything over which we have significant influence.    that, and 
it's not sufficient - some text is generated by computer programs,
and we can't force those programmers to always generate normalized text
either.

the ironic thing is that Unicode isn't inherently normalized because
of a need for backward compatability with existing character sets -
you were supposed to be able to translate to and from Unicode without
loss.  so people who are griping about the insistence on ASCII-
compatibility in email and insisting that we move to "pure" utf-8 are 
really just saying we should trade one kind of backward compatibility 
problem for another.

Keith

From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
cc: [email-masked]
Subject: Re: text/html an abomination 
Date: Tue, 07 May 2002 08:27:55 -0400
Sender: [email-masked]

> When people first get access to mechanisms for formatting, fonts, etc.,
> they tend to overuse it.  Over time, they learn to be more judicious.

true.  but in this case it's not the users who are causing the problem,
it's the implementors.  

i.e. it's not that people are making excessive use of boldface, font 
changes, etc - it's that implementors are adding gratuitous markup
that typically wasn't specified by the author of the message and 
adds absolutely nothing to the displayed output.

Date: Tue, 07 May 2002 09:57:02 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoftye
Sender: [email-masked]

At 15:16 04/05/2002 -0500, Eric A. Hall wrote:
> > Header signatures and encryption can be accomplished by using nested
> > messages.
>
>Completely impractical. I have to send a message to myself (or save it to
>the Drafts folder) and then forward the resulting message, which destroys
>secondary context such as threading, unless of course I have a tool that
>does all of this on my behalf.

Exactly, you could have a tool which does all that. The message formatting 
supports what you want.

If header signatures & encryption was that important to people, someone 
would have written an MUA to support it using the existing infrastructure.

Modifying the message format won't force people to write an MUA to support 
these things, it'll still need someone to produce the MUA software to 
encrypt the message and put it into your message format for you.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Tue, 07 May 2002 09:46:35 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
Sender: [email-masked]

At 10:43 04/05/2002 -0500, Eric A. Hall wrote:
>Keith Moore wrote:
>DSNs with NOTIFY=SUCCESS are an obvious example. The only thing I care
>about is that delivery succeeds, meaning that this is a negotation between
>the originator and the delivery server. The submission server and any
>relays in between are completely irrelevant, yet due to the current
>envelope-exchange design, they dictate the success of that feature.

Well, the OBVIOUS answer to that, is to take the DSN information out of the 
envelope and put it into the header. This won't break any existing MTAs, 
and the header gets passed through and ignored any MTA which doesn't 
understand it.

There is probably a good reason why this isn't done, but it'd work with the 
existing infrastructure, so maybe that should be investigated as an option..

A worry I'd have about having an 'envelope blob' is that people might start 
using that as an alternative to the header for other things (after all, 
there's really NOTHING in the message header that you couldn't put into the 
'envelope' if you wanted to, and if it was capable of handling it...). Non-SMTP

>We need a new message format if we want to allow UTF-8 directly in the
>message headers. You will probably argue that a UI facade will work just
>as well for this, but you will be wrong.

Do we want UTF-8 directly in the message headers ENOUGH to break everything 
else?

>We need a new message format if we want to be able to encrypt the full
>message (message headers and body, rather than just the body), since the
>commingling of transfer and message headers prevents that now.

Hmm - what 'headers' are we talking about here. The only one which would be 
worth encryting would be the subject - the recipients have to be stored in 
plain text in the envelope anyway, the date is approximately deducable, and 
other header fields may be needed to be used by the MTA/MUA automatically 
anyway (eg priority). Surely encrypted subject lines could be handled some 
other way rather than breaking everything else.

>As single items, none of those warrant a new message format. Collectively
>(especially with other benefits that become viable with a new format, as
>with the envelope modifications above), they do warrant it.

> > > On the other hand, what is it that is not feasible about a new
> > > message and transfer service?
> >
> > 1. getting users to buy into it when it is disruptive and it doesn't
> > provide them with any immediate benefit.
>
>If the features were desirable and it simplified implementation, people
>would use it. It wouldn't happen overnight, we can agree on that.

But when WOULD it happen? Because it wouldn't happen overnight, all the 
MTAs/MUAs would have to be able to fall back to the existing mode (with 
possible message reformatting being done by MTAs (thus possibly breaking 
things)).

The example of DSN shows what a problem it will be. DSN has been around for 
more than 6 years, but lots of MTAs don't support it. I suspect that in 
another 5 or 10 years time there will still be MTAs which don't support it, 
and can you guarantee that in 20 years time there STILL won't be MTAs which 
don't support it?

So, does this mean that all MUAs/MTAs would have to support BOTH message 
format structures (or just the existing format..) for the forseeable 
future? Not only that, but MTAs which support the proposed format would 
have to have the ability to convert the proposed format into the existing 
format reliably, so they'd be made even more complex than ever.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Tue, 7 May 2002 10:32:03 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Dan Kohn <[email-masked]>
Cc: Simon Josefsson <[email-masked]>, Jacob Palme <[email-masked]>, [email-masked]
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
Sender: [email-masked]

Dan Kohn <[email-masked]>
> Further, see that the first NOTE on page 19 of RFC 2046 is what enables
> you (at least in theory) to have multiparts that are not broken by CRLFs.

Yes. But, having an imapext archive handy, I looked at it now. The x-mailer
fields from the first three messages that break the rule are:

X-Mailer: Mozilla 4.61 [en] (WinNT; U)
X-Mailer: Mulberry (MacOS) [2.0.0b2, s/n S1-000001]
X-Mailer: Mozilla 4.72 [en] (WinNT; U)

The IMAP torture test message also breaks it - ther's a GIF image you
can't decode without the trailing CRLF.

I infer that a MUA can't assume that a missing CRFL at the end of a
body-part means anything.

--Arnt

To: [email-masked]
Subject: Re: text/html an abomination
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 06 May 2002 19:31:07 -0700
Lines: 46
Sender: [email-masked]

Dave Crocker <[email-masked]> writes:

> When people first get access to mechanisms for formatting, fonts, etc.,
> they tend to overuse it.  Over time, they learn to be more judicious.

Yeah, that's what I said when HTML first gained widespread use and first
gained those capabilities.

I'm still waiting for the learning to be more judicious part.  :)

Microsoft's generated HTML sets the font for practically every word.  Some
of the people who write these HTML generators should be forced to write
several physics lab manuals entirely in WordPerfect's view codes mode.  It
certainly taught me an attention to tag cleanliness that I've never lost.
*grin*

> The idea that email content should be required to have less visual
> control than a web page or a document simply doesn't make any sense.

It doesn't make sense to require it.

It does make sense to recognize that most e-mail is written quickly
without a lot of attention paid to design, and that therefore complex
visual layout languages are frequently inappropriate for e-mail in the
same way that one doesn't generally use FrameMaker to maintain one's
shopping list.  Trying to shoehorn normal e-mail into a page design
paradigm seems very wrong to me, and the cause of a lot of the worst of
the brokenness of multipart/alternative as it's seen in the wild.

There's also the entirely practical and non-standards-based, but
incredibly persuasive, argument that something like 98% of pure HTML
e-mail is spam.  Regular e-mail senders are, by and large, not using it.
This is creating a fast-growing barrier to *ever* using HTML as the
preferred markup language for e-mail; it's easy to get things into
people's spam filters, and then *extremely* hard to get them back out
again.  Try to send HTML mail to many people these days and they may not
even know that you sent it, because the last 500 messages of that type
that they've gotten were unwanted and offensive and made naked women pop
up all over their screen until they had to reboot their computer, and they
got tired of checking for the to-that-point completely nonexistent false
positives.

The well is getting pretty thoroughly poisoned.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Subject: Re: text/html an abomination
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 06 May 2002 19:20:20 -0700
Lines: 26
Sender: [email-masked]

Lawrence Greenfield <[email-masked]> writes:

> Who cares?  Your HTML renderer looks at the "cruft" and does the right
> thing and you never see it.

It makes the HTML renderer more complex.  More complex -> more bugs,
including security vulnerabilities, and the entire code path that parses
an e-mail message is security-sensitive and dealing with untrusted data.
Which sucks, because that's a *lot* of code to expose to a security layer.

It's bad enough in a web browser.  At least with a web browser, you have
some control over what sites you go to.  What with the growth of mail
spam, it's rarer and rarer that you can establish that sort of control
over what mail messages you open.  (I'm surprised that more mail clients
don't default to showing the raw HTML with a button saying "render this";
users wouldn't press it for spam and it would make discarding spam quicker
as well as safer.)

Right now, what with the HTML that some things are generating, one needs a
full-blown HTML 4.0 rendering engine complete with CSS level two support
and possibly even DOM support to actually "do the right thing" with all
that "cruft."  That seems a little excessive for just adding bold text to
a mail message, which is what the original author probably actually did.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: "Ken Hirsch" <[email-masked]>
To: <[email-masked]>
Subject: Re: MHonArc support for format=flowed
Date: Mon, 6 May 2002 22:17:07 -0400
Sender: [email-masked]

Very
Alan Barrett <[email-masked]> wrote:
> In UTF-7, "+ietf-" decodes to U+89EB followed by a decoding error.  I
> have no idea why IE6 is trying to interpret the page as UTF-7.
>

Very interesting.  UTF-7 seems like an encoding that shouldn't be guessed at, but
only used when explicitly coded.

I've just taken a look at Hotmail's support for format=flowed.  While it does take
the opportunity to nicely mark quoted messages, it doesn't actually reflow
paragraphs!

Would it be a good idea to create a short message demonstrating the features of
format=flowed to be included in the mimetest (http://www.imc.org/mimetest/) suite?

One last question?  Has there been any effort to come up with a more up-to-date name
than RFC822 format?  I know rfc2822 calls it Internet Message Format, but that leads
to acronyms that are more suggestive of instant messaging or the International
Monetary Fund than email. Perhaps something that includes H for headers or E for
email, like EHF or IMMHF or something like that?

Ken Hirsch

To: [email-masked]
Xref: clerew local.mime:1396
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Mon, 6 May 2002 15:33:54 GMT
Lines: 65
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>The time that programmers waste dealing with today's horrifying mess of
>character encodings is time taken away from providing real features for
>the users. It's so much time, in fact, that programmers invariably cut
>corners, producing occasional failures for the users.

>The obvious solution to both of these problems is to settle on a single
>character encoding. We've chosen the UTF-8 encoding of Unicode, because
>it's compatible with ASCII, it's self-synchronizing, it includes all the
>characters we need, etc. The transition plan is straightforward:

>   (1) add support for that encoding to all readers; then
>   (2) switch all writers to that encoding; then
>   (3) convert all stored data to that encoding; then
>   (4) stop worrying about other encodings.

Not quite, because it is hard to persuade implementors to add a feature to
*all* readers that will not result in some immediately visible benefit.
But there are other ways to do it.

We start from two basic situations:

1. UTF-8 is simple to generate, but current transports don't transport it,
and readers do not display it. Otherwise it is fine :-) .

2. Codecs such as RFC 2047 are difficult to implement (evidence: so many
implementations do it wrong) but at least they sort-of work at present and
sort-of comply with current standards. Adding IDNA as an additional codec
will surely give us more of the same.

SO you define an SMTP extension (let's call it "I18N") which accepts and
tranports UTF-8, and which does the "right thing" as regards DNS (such as
doing the IDNA conversion if is was not already done as received).

At the same time, you define a UTF-8 extension to the mail format, to be
used as an alternative to the existing RFC 2047 etc. Clearly, UTF-8
extended messages can only be used with MTAs which support I18N.

So what happens? Initially, nothing. But gradually I18N transports begin
to appear, and a few brave people start sending UTF-8 messages. That
encourages the appearance of more I18N transports.

Of course, use of RFC 2047 continues in parallel, but there comes a time
when it becomes apparent that UTF-8 messages are being transported more
reliably than the RFC 2047 ones (simply because they are more easily
implemented and therefore their implementations are more robust). At that
point, there will be a rush to switch over.

And finally, there will be a long and decreasing tail of old-style usage,
which will give worse and worse service as less and less effort is
expended on supporting those old formats. Eventually, after many years,
the old formats will disappear from the standards.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1398
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Mon, 6 May 2002 16:03:22 GMT
Lines: 51
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>Keith Moore <[email-masked]> writes:

>>> Yes. Unfortunately, the IDNA proponents are refusing to consider local
>>> parts, login names, etc.
>>
>> perhaps because it's not in their charter to do so?

>They seem to assume that other groups adopt their solution to solve
>such problems though.  This group seem to look at adopting the IDN
>solutions for localpart.  Which may be a bad idea since the IDNA
>proposals wasn't developed to solve this issues.

>The IDN WG should have been chartered to achieve IDNs in core internet
>protocols, not just IDNs as an isolated entity.  Well, too late now.

So it is left to groups like us to do the necessary standardization work
for Email. Which means that we should be looking at it carefully now, and
feeding back any problems into the IDNA people while there is still time
to change things.

However, I am less sure than I was that adapting IDNA for the local part
is the best way. How do you know that <[email-masked]> is intended
to be displayed as <[email-masked]>? Perhaps "zz--jrgen-kva" really *was*
the local-part for the entity intended to receive the message.

OTOH, you are going to get this problem with any encoding that satisfies
the existing syntax of a local-part.

Another difficulty is that the genuine local-part might contain
significant uppercase letters, and the IDNA encoding loses that
information (that is an example of a matter that could be fed into the
ongoing punycode work).

So perhaps we should still be looking at other possibilities for local
part, such as RFC 2047. Yes that needs care to ensure it is done
canonically, but that is just a matter of insisting that the whole
local-part be treated as exactly one encoded-word.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1397
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Mon, 6 May 2002 15:43:13 GMT
Lines: 44
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:
>> 
>> Do you really think Bill Gates is going to lower himself to using code
>> written by somebody else? No, he will cut his own and get it wrong
>> (probably deliberately) :-( .

>what we think Bill Gates will or will not do is not a sane basis for
>making a technical decision on a standard.  

What do you think the smiley was there for?

Sadly, however, what Bill Gates does will have more effect than what we
write in any standard :-( .

Another smiley, in case you didn't notice.

>do you really think that OS vendors are going to rewrite *every* form
>of input (not just keyboard drivers but copy-and-paste, files downloaded
>from the net, etc) to do normalization?  if they did, it would break
>far more things than it would fix.

The idea is that non-normalized stuff will never exist in files, displays
to be copied/pasted, and the like simply because all such material
ultimately derives from something that was keyed in on a normalizing
keyboard.

A risky strategy? Perhaps, but good enough for many non-critical
applications (critical applications would check, of course). The plain
fact of the matter is that writers of "ordinary" programs will not want to
bother with normalization (they will just expect utilities like 'tolower'
and 'strcmp' to do the Right Thing). And if ever they did try to do the
normalization, they would likely get it wrong anyway :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 06 May 2002 18:46:46 -0700
To: [email-masked]
From: Dave Crocker <[email-masked]>
Subject: Re: text/html an abomination 
Sender: [email-masked]

At 11:43 PM 5/5/2002 -0400, Keith Moore wrote:
>that wouldn't bother me a bit if such messages didn't have so much gratuitous
>markup.  in the vast majority of html messages I see, the html contains no 
>more
>information than could have been conveyed in plain text, but has so much
>cruft in it that it essentially cannot be read as text.

When people first get access to mechanisms for formatting, fonts, etc., 
they tend to overuse it.  Over time, they learn to be more judicious.

Formatting that is done well aids the communication, rather than detracting 
from it, and adds another dimension to the communication.

Let's not damn the capability just because some people have not yet learned 
to use it.

The idea that email content should be required to have less visual control 
than a web page or a document simply doesn't make any sense.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

From: Keith Moore <[email-masked]>
To: Lawrence Greenfield <[email-masked]>
cc: Sam Roberts <[email-masked]>, Keith Moore <[email-masked]>, [email-masked]
Subject: Re: text/html an abomination 
Date: Mon, 06 May 2002 21:17:33 -0400
Sender: [email-masked]

> Who cares?  Your HTML renderer looks at the "cruft" and does the right
> thing and you never see it.

some folks are still using text-only mailers.

some mail robots can't deal with HTML in body parts, but it would work
okay if the HTML weren't included except when needed.

lots of bandwidth is being wasted by mp/alternative of two parts
that contain the same information.

Keith

Date: Mon, 6 May 2002 21:14:27 -0400
From: Lawrence Greenfield <[email-masked]>
To: Sam Roberts <[email-masked]>, Keith Moore <[email-masked]>
Cc: [email-masked]
Subject: Re: text/html an abomination 
Sender: [email-masked]

   From: Keith Moore <[email-masked]>
   cc: [email-masked]
   Date: Sun, 05 May 2002 23:43:52 -0400

   > I also have friends with webmail accounts, and some of those
   > webmailers default to text/html, WITHOUT the
   > multipart/alternative of text/plain. Gagh.

   that wouldn't bother me a bit if such messages didn't have so much
   gratuitous markup.  in the vast majority of html messages I see,
   the html contains no more information than could have been conveyed
   in plain text, but has so much cruft in it that it essentially
   cannot be read as text.

Who cares?  Your HTML renderer looks at the "cruft" and does the right
thing and you never see it.

Larry

To: [email-masked]
Subject: Text with many charsets splitted into many MIME parts
From: Simon Josefsson <[email-masked]>
Date: Mon, 06 May 2002 23:00:33 +0200
Lines: 47
Sender: [email-masked]

--=-=-=
Content-Type: text/plain; charset=iso-2022-7bit
Content-Transfer-Encoding: quoted-printable

This is a list of ways to say hello in various languages.
Its purpose is to illustrate a number of scripts.

---------------------------------------------------------
Amharic	(=1B$(3"c!<!N"^=1B(B)	=1B$(3!A!,!>=1B(B
Arabic			=1B(38R=1B(47d=1B(3T!JSa=1B(4W=1B(3W=1B(B

--=-=-=
Content-Type: text/plain; charset=iso-8859-2
Content-Transfer-Encoding: quoted-printable

Czech (=E8esky)		Dobr=FD den
Danish (Dansk)		Hej, Goddag
English			Hello

--=-=-=
Content-Type: text/plain; charset=iso-8859-3
Content-Transfer-Encoding: quoted-printable

Esperanto		Saluton (E=B6o=FEan=F8o =E6iu=BCa=FDde)
Estonian		Tere, Tervist
FORTRAN			PROGRAM
Finnish (Suomi)		Hei

--=-=-=
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

French (Français)	Bonjour, Salut
German (Deutsch Nord)	Guten Tag
German (Deutsch Süd)	Grüß Gott

--=-=-=
Content-Type: text/plain; charset=iso-8859-7
Content-Transfer-Encoding: base64

R3JlZWsgKMXr6+ft6ercKQnD5encIPPh8g0K
--=-=-=
Content-Type: text/plain; charset=iso-8859-8
Content-Transfer-Encoding: base64

SGVicmV3CQkJ+ezl7Q0K
--=-=-=
Content-Type: text/plain; charset=devanagari
Content-Transfer-Encoding: quoted-printable

Hindi			=1B$(6!F!L"W!B!a=1B(B, =1B$(6!F!L"W=B3!Z!O=1B(B =1B$(6!j=1B(B
Italian (Italiano)	Ciao, Buon giorno

--=-=-=
Content-Type: text/plain; charset=lao
Content-Transfer-Encoding: base64

TGFvKL7SytLF0scpICAgICAgICAgICAgytC60q201Swgou3jy+niqqG01Q0K
--=-=-=
Content-Type: text/plain; charset=iso-8859-3
Content-Transfer-Encoding: quoted-printable

Maltese (Malti)		Bon=F5u, Sa=B1=B1a
Nederlands, Vlaams	Hallo, Dag
Norwegian (Norsk)	Hei, God dag

--=-=-=
Content-Type: text/plain; charset=iso-8859-2
Content-Transfer-Encoding: quoted-printable

Polish			Dzie=F1 dobry! Cze=B6=E6!

--=-=-=
Content-Type: text/plain; charset=iso-8859-5
Content-Transfer-Encoding: base64

UnVzc2lhbiAowOPh4drY2SkJt9Tg0NLh4tLj2eLVIQ0K
--=-=-=
Content-Type: text/plain; charset=iso-8859-2
Content-Transfer-Encoding: quoted-printable

Slovak (slovensky)	Dobr=FD de=F2

--=-=-=
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

Spanish (Español)	¡Hola!
Swedish (Svenska)	Hej, Goddag

--=-=-=
Content-Type: text/plain; charset=thai-tis620
Content-Transfer-Encoding: base64

VGhhaSAowNLJ0uS3wikJCcrH0cq01aTD0bosIMrH0cq01aTo0A0K
--=-=-=
Content-Type: text/plain; charset=tibetan
Content-Transfer-Encoding: base64

VGliZXRhbiAoorei3aKyobuix6OhotGisqG7KQmit6Kho8Oi0aG7osWi06LHobuit6Kyotuhu6LE
otuio6LHob4NCg==
--=-=-=
Content-Type: text/plain; charset=iso-2022-7bit
Content-Transfer-Encoding: quoted-printable

Tigrigna (=1B$(3"8#r!N"^=1B(B)	=1B$(3!Q!,!<"8=1B(B

--=-=-=
Content-Type: text/plain; charset=iso-8859-9
Content-Transfer-Encoding: quoted-printable

Turkish (T=FCrk=E7e)	Merhaba

--=-=-=
Content-Type: text/plain; charset=viscii
Content-Transfer-Encoding: quoted-printable

Vietnamese (Ti=AAng Vi=AEt)	Ch=E0o b=D5n

--=-=-=
Content-Type: text/plain; charset=euc-jp
Content-Transfer-Encoding: base64

DQpKYXBhbmVzZSAoxvzL3LjsKQkJpLOk86TLpMGkzywg
--=-=-=
Content-Type: text/plain; charset=euc-jp
Content-Transfer-Encoding: base64

jrqO3Y7GjsGOyg0K
--=-=-=
Content-Type: text/plain; charset=gb2312
Content-Transfer-Encoding: base64

Q2hpbmVzZSAo1tDOxCzG1c2ou7AsurrT7ykJxOO6ww0K
--=-=-=
Content-Type: text/plain; charset=big5
Content-Transfer-Encoding: base64

Q2FudG9uZXNlICi4Zrt5LLxzqka43CkJCaatseEsIKdBpm4NCg==
--=-=-=
Content-Type: text/plain; charset=euc-kr
Content-Transfer-Encoding: base64

S29yZWFuICjH0bHbKQkJCb7Is+fHz7y8v+QsIL7Is+fHz73KtM+x7g0K
--=-=-=
Content-Type: text/plain; charset=gb2312
Content-Transfer-Encoding: quoted-printable

Difference among chinese characters in GB, JIS, KSC, BIG5:
	GB   -- =D4=AA=C6=F8  =BF=AA=B7=A2

--=-=-=
Content-Type: text/plain; charset=euc-jp
Content-Transfer-Encoding: base64

CUpJUyAgLS0guLW1pCAgs6vIrw0K
--=-=-=
Content-Type: text/plain; charset=euc-kr
Content-Transfer-Encoding: base64

CUtTQyAgLS0g6qrRqCAgy9LboQ0K
--=-=-=
Content-Type: text/plain; charset=big5
Content-Transfer-Encoding: base64

CUJJRzUgLS0gpLiu8CAgtn21bw0K
--=-=-=
Content-Type: text/plain; charset=euc-jp
Content-Transfer-Encoding: quoted-printable

Just for a test of JISX0212: =F1=D6=8F=E9=D1 (the second character is of JI=
SX0212)

--=-=-=--

To: Marc Mutz <[email-masked]>
Cc: [email-masked]
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
From: Simon Josefsson <[email-masked]>
Date: Mon, 06 May 2002 22:58:46 +0200
Lines: 17
Sender: [email-masked]

Marc Mutz <[email-masked]> writes:

> On Monday 06 May 2002 10:05, Simon Josefsson wrote:
> <snip>
>> You can use MIME parts for this as well, mark the parts as inline and make
>> sure you don't add newlines during encoding.  The processing rules
>> describes fairly well how to decode it back together.
> <snip>
>
> Euhm? Where did you read this? The only rfc I know of that defines
> presentation of mails is rfc2183, defining the C-D header (plus
> others that add new C-D values). RFC2183 basically interprets
> "inline" as "non-iconic" or "expanded". In HTML speak this means
> <div>, not <span>.

Core MIME specs, Dan gave the reference.  For entertainment, I'll send
the HELLO file in the next message.

Date: Mon, 06 May 2002 18:46:43 +0200
From: Marc Mutz <[email-masked]>
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
To: Simon Josefsson <[email-masked]>
Cc: [email-masked]
Organization: KDE
Content-description: clearsigned data
Sender: [email-masked]

On Monday 06 May 2002 10:05, Simon Josefsson wrote:
<snip>
> You can use MIME parts for this as well, mark the parts as inline and make
> sure you don't add newlines during encoding.  The processing rules
> describes fairly well how to decode it back together.
<snip>

Euhm? Where did you read this? The only rfc I know of that defines 
presentation of mails is rfc2183, defining the C-D header (plus others that 
add new C-D values). RFC2183 basically interprets "inline" as "non-iconic" or 
"expanded". In HTML speak this means <div>, not <span>.

Marc

- -- 
Marc Mutz <[email-masked]>

content-class: urn:content-classes:message
Subject: RE: BOF: Format-Flowed and Non-Western Charsets
Date: Mon, 6 May 2002 13:16:23 -0700
From: "Dan Kohn" <[email-masked]>
To: "Simon Josefsson" <[email-masked]>, "Jacob Palme" <[email-masked]>
Cc: "Arnt Gulbrandsen" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

For an example, see page 9 of RFC 2152.  Unfortunately, few mailers in practice support re-assembling separate text parts, even when they are all marked as inline.

Further, see that the first NOTE on page 19 of RFC 2046 is what enables you (at least in theory) to have multiparts that are not broken by CRLFs.

http://www.ietf.org/rfc/rfc2046.txt
http://www.ietf.org/rfc/rfc2152.txt

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>
Essays announced on <mailto:[email-masked]>

-----Original Message-----
From: Simon Josefsson [mailto:[email-masked]] 
Sent: Monday, May 06, 2002 01:05
To: Jacob Palme
Cc: Arnt Gulbrandsen; [email-masked]
Subject: Re: BOF: Format-Flowed and Non-Western Charsets

On Sun, 5 May 2002, Jacob Palme wrote:

> At 12:51 +0100 02-03-19, Arnt Gulbrandsen wrote:
> >No. Multi-language mail is more common than you might think. Two examples:
> >
> >I got mail from someone in France this morning: The message was in English
> >but the .sig was in French.
> >
> >When a fellow Norwegian posts in English on a mailing list and I answer
> >privately, I might well answer in Norwegian, quoting the English posting.
> 
> That is why we have MIME body parts. Put each language piece
> into a separate body part, with content-language to indicate
> its language.
> 
> The body part solution will work when the border between
> languages is at a line boundary. However, what about text
> like the following:
> "I spent three years at Universät Lüneburg in Germany"
> where the language break occurs twice in the middle of
> a line!

You can use MIME parts for this as well, mark the parts as inline and make
sure you don't add newlines during encoding.  The processing rules 
describes fairly well how to decode it back together.

Gnus supports this.  The extreme example is the Emacs "HELLO" page with
strings for all languages Emacs support -- sending the file as an email
results in 20-30 or so parts.  Of course, it would be better to encode it
all as Unicode, but Emacs does not support Unicode well yet.

I haven't seen a client that handle the resulting mail in any sane way 
though, people usually don't bother to implement the entire MIME specs.

Date: Mon, 06 May 2002 13:19:43 -0500
From: "Eric A. Hall" <[email-masked]>
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Charles Lindsey wrote:

> Not a new message format surely? Just an extension opf the present
> format.

I don't think a new format is justifiable if it only meets one demand
(UTF-8 in this particular case). The benefit of a new format is that it
allows multiple problems to be resolved, including allowing UTF-8 values,
but also facilitating easier parsing of various header fields (display
names vs comments vs group names vs ...), facilitating separation of
transfer headers from content headers, facilitating desirable features
such as longer lines of multilingual content and inline attributable
quoting, and so forth.

> Do you mean signature, or encryption, or both? Eitherway, selective
> signature of just some headers is already deployed (pgpverify) and
> something similar should be possoble for encryption. See
> draft-lindsey-usefor-signed-01.txt (which was designed primarily for
> news, but should work in mail just as well).

I appreciate this work as necessary and useful. However, this model
imposes another additional layer on top of the existing rules. Systems
already cannot generate 2047_Group:idna; and it seems unlikely that they
will have greater success creating/reading Signed:2047_Group:idna;. This
doesn't mean that the work is not useful, it just highlights that all of
the codecs and layers do not actually solve the canonical problem, which
is that 822 syntax was not designed with the modern Internet in mind. By
all means, please continue this work, but at some point, we will need to
realize that coming up with a message format that suits the modern
requirements is going to be more productive than adding more layers of
bubblegum and baling wire.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: local-part conventions and enhancements 
Date: Mon, 06 May 2002 14:08:02 -0400
Sender: [email-masked]

> >I must have missed something.  If interpretation of the local-part is
> >entirely a matter for the domain, why do we need to standardize the
> >structure of the local-part?
> 
> yeah.  it IS confusing.
> 
> the phone-related template is a global standard.  the benefit of this is
> that each recipient (offramp) will not have different syntax or different
> basic semantics.  this is a big win.

okay, if that's the only goal, then applying the IDNA encoding to a
local-part with some internationalized parameters would still suffice.
the offramps would have to learn to decode that encoding before parsing
it, but they'd have to learn a new syntax anyway.

Keith

Date: Mon, 06 May 2002 11:00:31 -0700
To: Keith Moore <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: local-part conventions and enhancements 
Cc: [email-masked], [email-masked]
Sender: [email-masked]

At 01:54 PM 5/6/2002 -0400, Keith Moore wrote:

>I must have missed something.  If interpretation of the local-part is
>entirely a matter for the domain, why do we need to standardize the
>structure of the local-part?

yeah.  it IS confusing.

the phone-related template is a global standard.  the benefit of this is 
that each recipient (offramp) will not have different syntax or different 
basic semantics.  this is a big win.

the theory is that there is benefit in extending this model for other 
parametric local-part situations.  the clear requirement, however, is 
backward compatibility so that the more popular free-form local-part continues.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

From: Keith Moore <[email-masked]>
To: Dave Crocker <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: local-part conventions and enhancements 
Date: Mon, 06 May 2002 13:54:23 -0400
Sender: [email-masked]

I must have missed something.  If interpretation of the local-part is
entirely a matter for the domain, why do we need to standardize the 
structure of the local-part?

Keith

Date: Mon, 06 May 2002 10:25:40 -0700
To: [email-masked]
From: Dave Crocker <[email-masked]>
Subject: Re: local-part conventions and enhancements
Cc: [email-masked]
Sender: [email-masked]

At 09:22 AM 5/6/2002 -0700, [email-masked] wrote:
>>what makes the template inappropriate for other sorts of services, some of
>>which are not offramp, but instead something on the targeting host?
>
>It is approprate as long as interpretation is left up to the system that
>"owns" the local-part.

Agree entirely.  One of the major bits of superiority in Internet mail 
technology is that the local part was left as a local matter.  It permits 
all sorts of flexibility that the well-intentioned, but over-specified, 
X.400 addressing prevented.  Alas, I tend to view the superiority as having 
happened more by luck than intent.  In effect it was the result of the 
strong independence of Arpanet research sites.  Getting agreement on the 
right-hand side was obviously required, but messing with the left-hand side 
would have been viewed as incursion.  (Hmmm, I find myself thinking of U.S. 
"states' rights" arguments.)

However I wonder about standardizing the SYNTAX in local-part.  The current 
free form would remain, but the set of restricted characters would grow a 
bit, with the use of the newly-restricted characters limited to this 
parametric form.

This presumes that there is global benefit to standardizing the syntax, and 
that translates into benefit into having a standard scheme for generating 
and parsing this type of multi-field local-part.

>What's can be problematic is the combination of a parameter structure and
>encoding of non-ASCII characters. This sort of encoding implies some level of
>interpretation of the local-part going on in the context of systems that 
>do not
>"own" it but want to "display" it.

Exactly.  And, yes, merging this with international character support makes 
things more... interesting.

>There's a slippery slope lurking in this neighborhood. I don't think we're on
>it yet, but we do need to be careful.

If we pursue standardization, we are definitely incurring the slippage.  I 
would not make light of the implications.

I've come to believe that we do need a standard way of doing local-part 
parameterization.  It's a requirement that has shown up a few times and we 
might as well tackle it.  Clearly, international characters are also required.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

Date: Mon, 06 May 2002 09:22:52 -0700 (PDT)
From: [email-masked]
Subject: Re: local-part conventions and enhancements
To: Dave Crocker <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> At 11:52 PM 5/5/2002 -0700, [email-masked] wrote:
> >Agreed. Additionally, the optional fields in 3192 and 3193 are associated with
> >offramp functionality, which makes the concerns about being able to map the
> >local-parts of such addresses to mailboxes moot.

> the purpose of 3192/3193 was for offramp, yes.  however that's really a
> 'local' decision of the system processing the local-part.

> what makes the template inappropriate for other sorts of services, some of
> which are not offramp, but instead something on the targeting host?

It is approprate as long as interpretation is left up to the system that
"owns" the local-part.

What's can be problematic is the combination of a parameter structure and
encoding of non-ASCII characters. This sort of encoding implies some level of
interpretation of the local-part going on in the context of systems that do not
"own" it but want to "display" it.

There's a slippery slope lurking in this neighborhood. I don't think we're on
it yet, but we do need to be careful.

				Ned

To: [email-masked]
Xref: clerew local.mime:1395
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
Date: Mon, 6 May 2002 15:11:24 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> "Eric A. Hall" <[email-masked]> writes:

>We need a new message format if we want to allow UTF-8 directly in the
>message headers. You will probably argue that a UI facade will work just
>as well for this, but you will be wrong.

Not a new message format surely? Just an extension opf the present format.

>We need a new message format if we want to be able to encrypt the full
>message (message headers and body, rather than just the body), since the
>commingling of transfer and message headers prevents that now.

Do you mean signature, or encryption, or both? Eitherway, selective
signature of just some headers is already deployed (pgpverify) and
something similar should be possoble for encryption. See
draft-lindsey-usefor-signed-01.txt (which was designed primarily for news,
but should work in mail just as well).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, Graham Klyne <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Mon, 06 May 2002 11:57:29 -0400
Sender: [email-masked]

> We are both trying to solve known problems, we are just taking different
> approaches. I would hope that you would recognize that I am trying to take
> the long-view by advocating a message format and protocol that addresses
> the multiple roots of the multiple known problems. All of the evidence
> shows that codec-based cures introduce additional interoperability
> failures of their own, so I don't see them as being anything other than
> short-term patches.

sure, but the evidence also shows that new formats and protocols introduce
interoperability problems and disruption.  I'm sure we will need new 
formats and protocols eventually, but I don't think they're justified
until they can provide a substantially better level of service. 
I haven't seen any evidence that we even know how to significantly
improve mail service by any means, much less one that requires a
change ot the mssage format.
 
> To wit, 2047_Group:idna; is likely to malfunction for the majority of the
> mailers

it won't malfunction any more often than "ascii-group: ascii-address;"

Keith

Date: Mon, 06 May 2002 10:30:44 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Graham Klyne <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith "Codec" Moore wrote:

> my comment wasn't about your document - it was about the idea that
> we should define a new message format rather than try to solve
> known problems.

We are both trying to solve known problems, we are just taking different
approaches. I would hope that you would recognize that I am trying to take
the long-view by advocating a message format and protocol that addresses
the multiple roots of the multiple known problems. All of the evidence
shows that codec-based cures introduce additional interoperability
failures of their own, so I don't see them as being anything other than
short-term patches.

To wit, 2047_Group:idna; is likely to malfunction for the majority of the
mailers, just based on the the known implementation problems with 2047 and
Group addresses, even without adding another codec. If you know of another
way to solve all three of these problems without defining a unified syntax
as part of a new message, I'd love to hear about it.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: Graham Klyne <[email-masked]>
cc: Keith Moore <[email-masked]>, "Eric A. Hall" <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Mon, 06 May 2002 11:06:54 -0400
Sender: [email-masked]

> Maybe your comment isn't aimed at the proposal I mentioned, but just for
> the record, it *was* prepared in response a real problem - IM (though in
> the end not used for that problem), and it was actually used in a product
> prototype. 

my comment wasn't about your document - it was about the idea that we
should define a new message format rather than try to solve known problems.

Date: Mon, 06 May 2002 12:35:35 +0100
To: Keith Moore <[email-masked]>
From: Graham Klyne <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Cc: "Eric A. Hall" <[email-masked]>, [email-masked]
Sender: [email-masked]

Keith,

Maybe your comment isn't aimed at the proposal I mentioned, but just for 
the record, it *was* prepared in response a real problem - IM (though in 
the end not used for that problem), and it was actually used in a product 
prototype.  And I tried to be quite clear both in the document and in 
discussions that it wasn't considered a replacement for any existing 
messaging infrastructure.

I happen to see a requirement for extensible message metadata.  Beyond just 
sending messages, that is.  I also think it is valuable if such metadata is 
exchangeable between systems.

#g
--

At 06:33 PM 5/3/02 -0400, Keith Moore wrote:
> > It seems to me that we should be cataloging the kinds of requirements that
> > we want from a new message.
>
>It seems to me that we should be trying to solve real problems rather than
>trying to invent new things that serve no purpose except to disrupt the
>installed base.
>
>If you want to do something useful for email, how about figuring out how to
>make it really reliable, or spam-proof, or virus proof?
>
>Keith

-------------------
Graham Klyne
<[email-masked]>

Date: Mon, 06 May 2002 13:01:38 +0100
To: "Eric A. Hall" <[email-masked]>
From: Graham Klyne <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Cc: [email-masked]
Sender: [email-masked]

At 03:07 PM 5/3/02 -0500, Eric A. Hall wrote:
>It seems to me that we should be cataloging the kinds of requirements that
>we want from a new message. My bet is that some of these wishes will also
>indicate a requirement for a new transfer protocol as well. Some of the
>things I would like to see:

[long and varied list snipped]

I was once involved in a messaging project to deploy a wide range of 
similar features in a monolithic protocol loosely based on Internet mail, 
which turned out to be a not-very-effective [*] way to proceed ([*] British 
style understatement ;-).

It seems that in an Internet context, it's far easier in most cases to make 
progress by tackling the problems individually, and within the constraints 
of email architecture and practice.  There are a very few problems which 
are sufficiently hard to be practically impossible to do this way (e.g. 
some of the requirements for instant messaging), some which permit coherent 
designs but uncertain deployability (e.g. content negotiation for Internet 
fax) and many for which workable solutions clearly do exist.

But mainly, I think most requirements are not fundamentally limited by the 
message format in Internet mail.

Internet mail is an important application, and current protocols generally 
work very well for that application.  But I think not every application 
that looks vaguely like mail is a good candidate for use of SMTP (just as 
not every application that looks a bit like a request-response interaction 
is an  ideal candidate for HTTP).

Regarding your list of requirements, I'd suggest not lumping them all into 
a single wish-bag.  I prefer to find architectures that allow one to 
separate concerns;  the RFC822-XML proposal started out as an attempt to 
leverage the experience with RFC822 to create a piece for use with instant 
messaging.  As it happens, it hasn't been adopted there, but was used in a 
prototype message archive development.  A solution that tried to tackle too 
many problems at once probably wouldn't have been useful.

Which is roughly where I started above.

#g

-------------------
Graham Klyne
<[email-masked]>

Date: Mon, 6 May 2002 13:15:52 +0200 (CEST)
From: [email-masked]
Subject: Re: local-part conventions and enhancements
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

On  6 May, Nick Shelness wrote:
> how to get there. What I was objecting to was defining an
> over-arching encoding scheme for local-parts for the sole purpose of
> encoding accented names in the local-part.

A small correction here. It is not just accented characters. In Sweden
we have three additional characters "åäö". They may look like they are
just accented but the are really three distinct characters with their
own sounds and collating positions.

The frequency of the most common of these "ä" is about the same as for
"f" in English. So imagine that you couldn't use "f" in email-addresses
just because some other country had decided that they do not use it so
there can be no reason you need to use it either:-)

	/MaF

To: Dave Crocker <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: local-part conventions and enhancements
From: Nick Shelness<[email-masked]>
Date: Mon, 6 May 2002 10:41:32 +0100
Sender: [email-masked]

Dave,

> > I wrote:
> > I think that
> > achieving a similar consensus for local-parts will be difficult. I for
> > one, will take a lot of convincing.
>
> You wrote: 
> RFC3192 and RFC3193 focus on using a template for local-part that 
pertains 
> to encoding telephone information.

As I think you would expect, I have absolutely no objection to local-part 
encoding, per se. For that matter, I have no philosophical objection to 
the use of UTF8 in RFC 2822 headers, I'm just not sure how to get there. 
What I was objecting to was defining an over-arching encoding scheme for 
local-parts for the sole purpose of encoding accented names in the 
local-part. 

Nick

Date: Mon, 06 May 2002 01:16:14 -0700
To: [email-masked]
From: Dave Crocker <[email-masked]>
Subject: Re: local-part conventions and enhancements
Cc: [email-masked]
Sender: [email-masked]

At 11:52 PM 5/5/2002 -0700, [email-masked] wrote:
>Agreed. Additionally, the optional fields in 3192 and 3193 are associated with
>offramp functionality, which makes the concerns about being able to map the
>local-parts of such addresses to mailboxes moot.

the purpose of 3192/3193 was for offramp, yes.  however that's really a 
'local' decision of the system processing the local-part.

what makes the template inappropriate for other sorts of services, some of 
which are not offramp, but instead something on the targeting host?

>Some of the X.400 fields had a flavor similar to the FAX parameters (the 
>postal
>delivery stuff, for example) but it was never entirely clear to me where the
>boundary between matched and unmatched fields was. And there were around 48
>possible fields in an X.400 address... What a mess.

i think the fundamental error was to start with an all-singing, 
all-dancing, all-encompassing and too rigid syntax.  by contrast we've had 
an entirely loose one that works well, and the idea behind retro-fitting 
these <service>"=" local-part templates is that each one would have a 
well-defined -- ie, constrained -- use.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

Date: Mon, 6 May 2002 10:05:00 +0200 (CEST)
From: Simon Josefsson <[email-masked]>
To: Jacob Palme <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, <[email-masked]>
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
Sender: [email-masked]

On Sun, 5 May 2002, Jacob Palme wrote:

> At 12:51 +0100 02-03-19, Arnt Gulbrandsen wrote:
> >No. Multi-language mail is more common than you might think. Two examples:
> >
> >I got mail from someone in France this morning: The message was in English
> >but the .sig was in French.
> >
> >When a fellow Norwegian posts in English on a mailing list and I answer
> >privately, I might well answer in Norwegian, quoting the English posting.
> 
> That is why we have MIME body parts. Put each language piece
> into a separate body part, with content-language to indicate
> its language.
> 
> The body part solution will work when the border between
> languages is at a line boundary. However, what about text
> like the following:
> "I spent three years at Universät Lüneburg in Germany"
> where the language break occurs twice in the middle of
> a line!

You can use MIME parts for this as well, mark the parts as inline and make
sure you don't add newlines during encoding.  The processing rules 
describes fairly well how to decode it back together.

Gnus supports this.  The extreme example is the Emacs "HELLO" page with
strings for all languages Emacs support -- sending the file as an email
results in 20-30 or so parts.  Of course, it would be better to encode it
all as Unicode, but Emacs does not support Unicode well yet.

I haven't seen a client that handle the resulting mail in any sane way 
though, people usually don't bother to implement the entire MIME specs.

Date: Mon, 06 May 2002 00:07:13 -0700 (PDT)
From: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
To: "Eric A. Hall" <[email-masked]>
Cc: Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

> Keith Moore wrote:

> > > I would suggest that the encoding above is much simpler and cleaner
> > > than a group address with multiple ACE sequences.
> >
> > nope, because the ACE parses just like any other domain name.

> We were talking about benefit to humans.

>  <PrimaryRecipient>
>    <DisplayName>Lõçal Admins</DisplayName>
>    <Mailbox>fred@lõçal</Mailbox>
>    <Mailbox>joe@lõçal</Mailbox>
>  </PrimaryRecipient>

> versus

>  To:=?iso-8859-1?Q?L=F5=E7?=al_Admins: fred@zz-adsadfsd, joe@zz-adsadfsd;

> As postmaster for your org, which would you rather troubleshoot?

I prefer the latter by far. It is far easier to pick out the truly relevant
parts. And it being all-ASCII is a HUGE advantage, as many tools tools don't
handle non-ASCII character output or input well at all. This isn't much of a
problem when dealing with Latin-1, but it gets pretty tense when dealing
with, say, Japanese or Chinese characters.

				Ned

Date: Sun, 05 May 2002 23:52:50 -0700 (PDT)
From: [email-masked]
Subject: Re: local-part conventions and enhancements
To: Dave Crocker <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> At 10:04 AM 5/3/2002 +0100, Nick Shelness wrote:
> >Moving on, there is clearly rough consensus that it makes no sense to
> >restrict domain-names (yes, names) to an ASCII alphabet, though clearly a
> >pure ASCII encoding is required for backward compatibility.

> yes.

> >   I think that
> >achieving a similar consensus for local-parts will be difficult. I for
> >one, will take a lot of convincing.

> RFC3192 and RFC3193 focus on using a template for local-part that pertains
> to encoding telephone information.

> However if one looks at the specifications with from a broader perspective
> they will note that a generic template is provided, and alas, it has an
> X.400 parametric tone to it.

> Simplistically, the template is of the style:

>          <type> "=" <prime-string> *( "/" <parameter-name> "="
> <parameter-value> )

> such as:

>          FAX=+12027653000/T33S=6377

> However it is an optional format, to be used only when the local-part
> string really needs to be parametric.  This is in contrast with X.400,
> where the structure was required even when it was not useful.

Agreed. Additionally, the optional fields in 3192 and 3193 are associated with
offramp functionality, which makes the concerns about being able to map the
local-parts of such addresses to mailboxes moot.

Some of the X.400 fields had a flavor similar to the FAX parameters (the postal
delivery stuff, for example) but it was never entirely clear to me where the
boundary between matched and unmatched fields was. And there were around 48
possible fields in an X.400 address... What a mess.

I think we'll be OK as long as we stay clear of adding any complexity
to the mapping from local-parts to mailboxes.

				Ned

Date: 6 May 2002 04:50:58 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Sender: [email-masked]

Keith Moore writes:
> the last thing I'd want is to have the IDN people making changes to
> several internet protocols - their expertise is much more in the
> technical and/or political problems associated with domains

The IDNA proponents are blatantly ignoring public written objections
from hundreds of people, including fifteen regular IDN WG participants.
Even if you weren't one of the IDNA proponents, your claim of technical
and political expertise wouldn't be credible.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

From: Keith Moore <[email-masked]>
To: Sam Roberts <[email-masked]>
cc: [email-masked]
Subject: Re: text/html an abomination 
Date: Sun, 05 May 2002 23:43:52 -0400
Sender: [email-masked]

> I also have friends with webmail accounts, and some of those webmailers
> default to text/html, WITHOUT the multipart/alternative of text/plain. Gagh.

that wouldn't bother me a bit if such messages didn't have so much gratuitous 
markup.  in the vast majority of html messages I see, the html contains no more
information than could have been conveyed in plain text, but has so much
cruft in it that it essentially cannot be read as text.

From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
cc: Keith Moore <[email-masked]>, "D. J. Bernstein" <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Sun, 05 May 2002 23:14:57 -0400
Sender: [email-masked]

> >> Yes. Unfortunately, the IDNA proponents are refusing to consider local
> >> parts, login names, etc.
> >
> > perhaps because it's not in their charter to do so?
> 
> They seem to assume that other groups adopt their solution to solve
> such problems though.  This group seem to look at adopting the IDN
> solutions for localpart.  Which may be a bad idea since the IDNA
> proposals wasn't developed to solve this issues.

perhaps - though I think that the problems are similar enough that 
IDNA will be a good candidate for the local-part solution.

> The IDN WG should have been chartered to achieve IDNs in core internet
> protocols, not just IDNs as an isolated entity.  

not clear.  the last thing I'd want is to have the IDN people
making changes to several internet protocols - their expertise is
much more in the technical and/or political problems associated 
with domains, not in email, news, web, directories, etc..

Keith

Date: Sun, 5 May 2002 19:54:27 -0400
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: text/html an abomination
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Sun, May 05, 2002 at 08:23:50PM +0200:
> At 11:11 -0500 02-03-19, Keith Moore wrote:
> >problem is, text/html has turned out to be an abomination -
> >it introduces security holes via java, javascript, etc., and
> >lots of folks I know filter it because the vast majority of text/html
> >mail they receive is spam.
> 
> Would that mean that you filter out all messages sent
> by Outlook Express? But you are right that spam seems
> to use text/html much more than non-spam messages.
> 
> Many mailers handle text/html, but only for a limited
> part of HTML, not including java and javascript. It
> is mainly used
> 
> (a) for simple formatting like bold facing, larger
>      font and monospacing fonts.
> (b) Including inline graphics in the e-mail.

I'm no fan of html mail, and I routinely get folks I know to change their
mailer settings to not send it, but text/html isn't a security hole, its the
processing of it as if it was a web page that is (and insecure web page
processing, at that).

My mailer runs html through lynx -local_host, I see it and reply to it as
text, no java or javascript runs, and no links (aka webbugs) are followed.
Outlook's security problems are its fault, not text/htmls.

I also have friends with webmail accounts, and some of those webmailers
default to text/html, WITHOUT the multipart/alternative of text/plain. Gagh.

Sam

-- 
Sam Roberts <[email-masked]> (Vivez sans temps mort!)

To: [email-masked]
Subject: Re: text/html an abomination
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sun, 05 May 2002 16:10:01 -0700
Lines: 32
Sender: [email-masked]

Jacob Palme <[email-masked]> writes:
> At 11:11 -0500 02-03-19, Keith Moore wrote:

>> problem is, text/html has turned out to be an abomination - it
>> introduces security holes via java, javascript, etc., and lots of folks
>> I know filter it because the vast majority of text/html mail they
>> receive is spam.

> Would that mean that you filter out all messages sent by Outlook
> Express?

No.  Outlook Express generates multipart/alternative messages.  Pure
text/html messages are 100% spam within my incoming mail; I use it as a
spam filtering method and I've never had a false positive, and I exchange
mail with a lot of people who use Outlook Express.  (Other people
subscribed to business news mailing lists and the like may have a
different experience.)

> Many mailers handle text/html, but only for a limited part of HTML, not
> including java and javascript. It is mainly used

> (a) for simple formatting like bold facing, larger
>      font and monospacing fonts.
> (b) Including inline graphics in the e-mail.

It's hard for me to tell what it's used for these days, since much of the
HTML that I see is generated by Microsoft Word or some equivalent
generator and contains so much embedded style data and formatting data
that it's almost impossible to tell what's actually being done.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sun, 5 May 2002 20:18:26 +0200
To: Arnt Gulbrandsen <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
Cc: [email-masked]
Sender: [email-masked]

At 12:51 +0100 02-03-19, Arnt Gulbrandsen wrote:
>No. Multi-language mail is more common than you might think. Two examples:
>
>I got mail from someone in France this morning: The message was in English
>but the .sig was in French.
>
>When a fellow Norwegian posts in English on a mailing list and I answer
>privately, I might well answer in Norwegian, quoting the English posting.

That is why we have MIME body parts. Put each language piece
into a separate body part, with content-language to indicate
its language.

The body part solution will work when the border between
languages is at a line boundary. However, what about text
like the following:
"I spent three years at Universät Lüneburg in Germany"
where the language break occurs twice in the middle of
a line!
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Sun, 5 May 2002 20:23:50 +0200
To: Keith Moore <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: text/html an abomination
Cc: Barry Leiba <[email-masked]>, [email-masked]
Sender: [email-masked]

At 11:11 -0500 02-03-19, Keith Moore wrote:
>problem is, text/html has turned out to be an abomination -
>it introduces security holes via java, javascript, etc., and
>lots of folks I know filter it because the vast majority of text/html
>mail they receive is spam.

Would that mean that you filter out all messages sent
by Outlook Express? But you are right that spam seems
to use text/html much more than non-spam messages.

Many mailers handle text/html, but only for a limited
part of HTML, not including java and javascript. It
is mainly used

(a) for simple formatting like bold facing, larger
     font and monospacing fonts.
(b) Including inline graphics in the e-mail.
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Sun, 05 May 2002 13:55:05 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: [email-masked], Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyonetellMicrosoftye
Sender: [email-masked]

[email-masked] wrote:

> In the particular case I mentioned the messages my MUA sees are
> entirely normal. They are packaged up as part of the submission
> process and unpackaged as part of the delivery process.

So my original statement is entirely true (nb the "tool" part):

 | I have to send a message to myself (or save it to the Drafts folder)
 | and then forward the resulting message, which destroys secondary
 | context such as threading, unless of course I have a tool that does
 | all of this on my behalf. Even then, somebody can take the embedded
 | message out of its context and all of the intended features and
 | benefits will be lost.

We are left to disagree on the practicality of having external monitoring
agents which encode and decode such messages on our behalf, according to
some heurestic, rather than having this occur within the scope of the MUA
itself. Personally I consider decoupling the transfer headers from the
message headers as the more appropriate long-term objective for multiple
reasons (obviously), not the least of which is the ability to have the
MUAs do this stuff.

> A more interesting question is how to merge Received: field
> information in such a setup.

Transfer headers are naturally related to the envelope ("here are the
delivery instructions"; "here is how I complied with the instructions"),
and are not naturally related to the contents of the message.

The undiscussed secondary point here is that MUAs should also get the
envelopes (along with the messages), rather than having delivery agents
discard the envelope as is the current practice. See RFC 2298, which
requires MDAs to convert ORCPT parameter values to Original-Recipient
header fields for the MUA's benefit as an example of where this separation
between the envelope and message content is not currently preserved, and
how MUAs would benefit from having the envelope.

> But more generally, if a desire for increased security warrants pushing
> encryption further towards the user, a tension necessarily develops
> between the need to keep things hidden and the need to expose things
> for indexing and other similar activities not directly associated with
> the handling of a message in isolation. Unfortunately, this tension is
> fundamental and chosing different syntaxes for things doesn't change
> it significantly.

Agree that there is a natural conflict here. The basic information in an
envelope tells hostile parties a lot: "Joe sent Fred a mesage at 9:02pm
from his home computer" is a lot of information if you have other
contextual data. There has to be a point where the underlying network
security takes over, obviously. But today the line exists at a point where
you not only know that Joe sent a message, but you also get to see the
Subject of the message, the References threading details, and a bunch of
other information that is much more useful than the envelope alone. The
difference between these lines is the difference between pen registers and
annotated summaries.

> Virus scanning signed or encrypted packages is problematic, for
> instance.

Those are still available, but at the recipient's system instead of at the
gateway. Certainly this remains as a valid concern however.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sun, 05 May 2002 10:32:36 -0700 (PDT)
From: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyonetellMicrosoftye
To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked], [email-masked], Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

> > > > Header signatures and encryption can be accomplished by using
> > > > nested messages.
> >
> > > Completely impractical.
> >
> > Funny, I use this capability on a daily basis as part of an internal
> > setup at Sun.

> I'd like to see one of these messages, please. I'm particularly interested
> to see how it handles the Subject header field in such a way as to protect
> the message's privacy while at the same time functioning to serve
> Subject-based threading.

In the particular case I mentioned the messages my MUA sees are entirely
normal. They are packaged up as part of the submission process and unpackaged
as part of the delivery process. The result is that threading works perfectly. 

A more interesting question is how to merge Received: field information in
such a setup.

But more generally, if a desire for increased security warrants pushing
encryption further towards the user, a tension necessarily develops between the
need to keep things hidden and the need to expose things for indexing and other
similar activities not directly associated with the handling of a message in
isolation. Unfortunately, this tension is fundamental and chosing different
syntaxes for things doesn't change it significantly.

There are many tradeoffs involved with using packaging techniques, especially
signed or encrypted packages. Virus scanning signed or encrypted packages is
problematic, for instance. There are plenty of other issues as well.

				Ned

To: Keith Moore <[email-masked]>
Cc: "D. J. Bernstein" <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
From: Simon Josefsson <[email-masked]>
Date: Sun, 05 May 2002 18:15:00 +0200
Lines: 14
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> Yes. Unfortunately, the IDNA proponents are refusing to consider local
>> parts, login names, etc.
>
> perhaps because it's not in their charter to do so?

They seem to assume that other groups adopt their solution to solve
such problems though.  This group seem to look at adopting the IDN
solutions for localpart.  Which may be a bad idea since the IDNA
proposals wasn't developed to solve this issues.

The IDN WG should have been chartered to achieve IDNs in core internet
protocols, not just IDNs as an isolated entity.  Well, too late now.

Date: Sun, 05 May 2002 09:05:56 -0700
To: [email-masked]
From: Dave Crocker <[email-masked]>
Subject: local-part conventions and enhancements
Sender: [email-masked]

At 10:04 AM 5/3/2002 +0100, Nick Shelness wrote:
>Moving on, there is clearly rough consensus that it makes no sense to
>restrict domain-names (yes, names) to an ASCII alphabet, though clearly a
>pure ASCII encoding is required for backward compatibility.

yes.

>   I think that
>achieving a similar consensus for local-parts will be difficult. I for
>one, will take a lot of convincing.

RFC3192 and RFC3193 focus on using a template for local-part that pertains 
to encoding telephone information.

However if one looks at the specifications with from a broader perspective 
they will note that a generic template is provided, and alas, it has an 
X.400 parametric tone to it.

Simplistically, the template is of the style:

         <type> "=" <prime-string> *( "/" <parameter-name> "=" 
<parameter-value> )

such as:

         FAX=+12027653000/T33S=6377

However it is an optional format, to be used only when the local-part 
string really needs to be parametric.  This is in contrast with X.400, 
where the structure was required even when it was not useful.

The template also has the advantage of being backward compatible.  It does 
not get in the way of existing local-parts, as far as we could tell.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.850.1850

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Sun, 05 May 2002 11:33:03 -0400
Sender: [email-masked]

 >> actually I haven't seen a single compelling argument to move to utf-8.
 >
 >The time that programmers waste dealing with today's horrifying mess of
 >character encodings is time taken away from providing real features for
 >the users. It's so much time, in fact, that programmers invariably cut
 >corners, producing occasional failures for the users.

true enough.  but the pain in transition is an even more horrifying mess,
and there's no assurance that the eventual result of transitioning won't 
be at least as horrifying as the current state of affairs.  

it's far easier to explain to programmers how to implement the current
specifications well, than to explain to programmers how to implement 
the transition well.  

Keith

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Sun, 05 May 2002 11:30:21 -0400
Sender: [email-masked]

> > Does anyone know if they plan to change that in some foreseeable future?
> 
> They don't even admit that it's a bug. In fact, according to the IETF
> specifications, it's perfectly acceptable to mangle all 8-bit bytes,
> including ISO 8859-1 bytes in the message body.

and if it conforms to the specs, it's not a bug.

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Sun, 05 May 2002 11:29:34 -0400
Sender: [email-masked]

> >second, because the code necessary to do normalization and encoding
> >will be available in libraries that everyone can use - the MUA vendor
> >won't have to get that part right.
> 
> Do you really think Bill Gates is going to lower himself to using code
> written by somebody else? No, he will cut his own and get it wrong
> (probably deliberately) :-( .

what we think Bill Gates will or will not do is not a sane basis for
making a technical decision on a standard.  

> >and of course the normalization would have to be done even with pure UTF-8.
> >the ASCII encoding step is fairly simple and isn't likely to introduce
> >many errors.
> 
> Not necessarily for normalization (NFC or NFKC). That is a job for
> keyboard drivers and other features of the operating system. 

do you really think that OS vendors are going to rewrite *every* form
of input (not just keyboard drivers but copy-and-paste, files downloaded
from the net, etc) to do normalization?  if they did, it would break
far more things than it would fix.

> It should
> normally be possible for people to write applications (and MUAs are
> applications) on the assumption that all the string in the system are
> already normalized. 

it's possible to write applications with those assumptions.  of course, 
they won't work reliably, because you can't control every possible input.

Keith

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Sun, 05 May 2002 11:23:44 -0400
Sender: [email-masked]

> Yes. Unfortunately, the IDNA proponents are refusing to consider local
> parts, login names, etc.

perhaps because it's not in their charter to do so?

Date: 5 May 2002 11:06:24 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Sender: [email-masked]

Kai Henningsen writes:
> There was *WHAT*?!

There was, for example, Mark Crispin saying

   7-bit MTA's will never go away. Ever. Mail goes on many paths besides
   IP, some of which are forever 7-bits. You will always have to solve
   the problem of getting your 8-bit mail into 7-bits for some 7-bit MTA.

and similar stupidity from Paul Vixie, Keith Moore, et al.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 5 May 2002 09:35:16 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Sender: [email-masked]

Charles Lindsey writes:
  [ on sendmail's destruction of bytes 128-159 in message headers ]
> Does anyone know if they plan to change that in some foreseeable future?

They don't even admit that it's a bug. In fact, according to the IETF
specifications, it's perfectly acceptable to mangle all 8-bit bytes,
including ISO 8859-1 bytes in the message body.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 5 May 2002 09:29:59 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Sender: [email-masked]

Keith Moore writes:
> actually I haven't seen a single compelling argument to move to utf-8.

The time that programmers waste dealing with today's horrifying mess of
character encodings is time taken away from providing real features for
the users. It's so much time, in fact, that programmers invariably cut
corners, producing occasional failures for the users.

The obvious solution to both of these problems is to settle on a single
character encoding. We've chosen the UTF-8 encoding of Unicode, because
it's compatible with ASCII, it's self-synchronizing, it includes all the
characters we need, etc. The transition plan is straightforward:

   (1) add support for that encoding to all readers; then
   (2) switch all writers to that encoding; then
   (3) convert all stored data to that encoding; then
   (4) stop worrying about other encodings.

The costs of #1, #2, and #3 are quite serious, but they're outweighed by
the massive continuing benefits of #4.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Sun, 05 May 2002 01:58:42 -0400
Sender: [email-masked]

> As postmaster for your org, which would you rather troubleshoot?

the latter.   it's actually easier for my eye to parse, since
I don't have to keep track of so many layers of nesting that
all use the same syntax, or scan so many lines, in order to see
the recipient addresses. also, the addresses in the latter example 
are valid anywhere, and can be typed on almost any keyboard,
whereas the addresses in your XML example are only valid on 
systems that know how to interpret unicode and can only be typed
effectively on a keyboard that has the right characters
(assuming I can even manage to figure out the input conventions
for some language that I don't know)

Keith

To: [email-masked]
Xref: clerew local.mime:1360
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Sat, 4 May 2002 20:33:26 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>second, because the code necessary to do normalization and encoding
>will be available in libraries that everyone can use - the MUA vendor
>won't have to get that part right.

Do you really think Bill Gates is going to lower himself to using code
written by somebody else? No, he will cut his own and get it wrong
(probably deliberately) :-( .

>and of course the normalization would have to be done even with pure UTF-8.
>the ASCII encoding step is fairly simple and isn't likely to introduce 
>many errors.

Not necessarily for normalization (NFC or NFKC). That is a job for
keyboard drivers and other features of the operating system. It should
normally be possible for people to write applications (and MUAs are
applications) on the assumption that all the string in the system are
already normalized. At least that is my understanding of the "Early
Normalization" principle in the draft by Duerst that I referred to before.

Anyway, for sure anything that does the actual DNS lookup (where correct
normalization is essential) will likely inspect (and even correct) the
string given to it. So it is not the end of the world if the MUA does it
less than perfectly.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 5 May 2002 01:40:49 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Sender: [email-masked]

Lawrence Greenfield writes:
> This is very important to me---the local part and the domain should be
> encoded in the same way whenever possible.

Yes. Unfortunately, the IDNA proponents are refusing to consider local
parts, login names, etc.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 5 May 2002 01:38:02 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Moving to UTF-8
Sender: [email-masked]

Charles Lindsey writes:
> Has anybody looked into the transitional issues that would arise if
> Email was to be moved into allowing UTF-8 headers?

http://cr.yp.to/proto/idnc3.html
http://pi.cr.yp.to

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sat, 04 May 2002 20:30:04 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> > I would suggest that the encoding above is much simpler and cleaner
> > than a group address with multiple ACE sequences.
> 
> nope, because the ACE parses just like any other domain name.

We were talking about benefit to humans.

 <PrimaryRecipient>
   <DisplayName>Lõçal Admins</DisplayName>
   <Mailbox>fred@lõçal</Mailbox>
   <Mailbox>joe@lõçal</Mailbox>
 </PrimaryRecipient>

versus

 To:=?iso-8859-1?Q?L=F5=E7?=al_Admins: fred@zz-adsadfsd, joe@zz-adsadfsd;

As postmaster for your org, which would you rather troubleshoot?

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sat, 04 May 2002 20:04:12 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: [email-masked], Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyonetellMicrosoftye
Sender: [email-masked]

[email-masked] wrote:

> > I'm not sure how productive it would be to set expectations and
> > requirement thresholds higher than the current service offers.
> 
> Which greatly diminishes the value of any replacement format.

I don't know if you think I've promised something grand from this. All I
promised from this particular feature would be the capability to transfer
options through routers without having to upgrade the router. That is
still there.

> But as my analysis shows this doesn't buy you as much as you originally
> claimed.

It buys you the option to implement features on top of mail without having
to get every MTA on the planet to salute first, and without having to
write things into the message headers (the other current option).

I want a "rescind" feature. I don't want to generate another message, and
I don't want the delivery server to have to poke around in message headers
since the original message may have been encrypted. An envelope feature is
the right way to do this today, but every MTA in between me and the
recipients has to be upgraded before it will work. Under the
pseudo-proposed system, it is an option with an alert. If the delivery
server doesn't support the option, then I get a failure-trace message back
from the server telling me so.

Yes, you could add this feature to SMTP today. You could add lots of
things to SMTP today. Lots of hammering on lots of code to make any of
them work on a broad scale.

> > ...once you upgraded the infrastructure to allow the new extension
> > to work through all of the hops. :/
> 
> A vastly simpler task than getting everything to use an entirely
> different format for everything.

I don't know if you've think I promised everything would use it. I am
pretty sure I said that backwards compatibility with legacy servers and
mailboxes is necessary.

> > > Header signatures and encryption can be accomplished by using
> > > nested messages.
> 
> > Completely impractical.
> 
> Funny, I use this capability on a daily basis as part of an internal
> setup at Sun.

I'd like to see one of these messages, please. I'm particularly interested
to see how it handles the Subject header field in such a way as to protect
the message's privacy while at the same time functioning to serve
Subject-based threading.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sat, 04 May 2002 14:36:54 -0700 (PDT)
From: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoftye
To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked], Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> > Pretty quickly you arrive at something where you'll have submission,
> > relay, and delivery criticality at a minimum. This gets to be complex

> I agree that the function-centric nature of SMTP participants is one of
> the core dilemmas in such a design.

> This problem also exists in the current model, of course. If the first-hop
> server doesn't support BDAT then the average joe is likely to get an error
> which they do not understand. As far as that goes, I'm not sure how
> productive it would be to set expectations and requirement thresholds
> higher than the current service offers.

Which greatly diminishes the value of any replacement format.

> With that in mind, if an extension actively requires the participation of
> a router and must be rejected if that router doesn't understand it, then
> we are at the same level of functionality as we have today. However, we
> would also have the ability to define a default position of "ignore"
> whereby unrecognized options could be safely passed.

But as my analysis shows this doesn't buy you as much as you originally
claimed.

> > Actually, an (admittedly complex) single SMTP extension to allow for
> > envelope elements that can pass through relays that don't understand
> > them could be defined now.

> ...once you upgraded the infrastructure to allow the new extension to work
> through all of the hops. :/

A vastly simpler task than getting everything to use an entirely different
format for everything.

> > Header signatures and encryption can be accomplished by using nested
> > messages.

> Completely impractical.

Funny, I use this capability on a daily basis as part of an internal setup at
Sun. it works just fine. So it doesn't seem impractical to me at all.

> I have to send a message to myself (or save it to
> the Drafts folder) and then forward the resulting message, which destroys
> secondary context such as threading, unless of course I have a tool that
> does all of this on my behalf.

That's only true if you're trying to do this manually. Software can and has
been written to do it automatically. There's even been discussion of
standardizing markers for it in the S/MIME area.

> Even then, somebody can take the embedded
> message out of its context and all of the intended features and benefits
> will be lost.

Again you are assuming manual operation. It doesn't have to be done this way.

				Ned

Date: Sat, 04 May 2002 15:16:52 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoftye
Sender: [email-masked]

[email-masked] wrote:

> Pretty quickly you arrive at something where you'll have submission,
> relay, and delivery criticality at a minimum. This gets to be complex

I agree that the function-centric nature of SMTP participants is one of
the core dilemmas in such a design.

This problem also exists in the current model, of course. If the first-hop
server doesn't support BDAT then the average joe is likely to get an error
which they do not understand. As far as that goes, I'm not sure how
productive it would be to set expectations and requirement thresholds
higher than the current service offers.

With that in mind, if an extension actively requires the participation of
a router and must be rejected if that router doesn't understand it, then
we are at the same level of functionality as we have today. However, we
would also have the ability to define a default position of "ignore"
whereby unrecognized options could be safely passed.

> Actually, an (admittedly complex) single SMTP extension to allow for
> envelope elements that can pass through relays that don't understand
> them could be defined now.

...once you upgraded the infrastructure to allow the new extension to work
through all of the hops. :/

> Header signatures and encryption can be accomplished by using nested
> messages.

Completely impractical. I have to send a message to myself (or save it to
the Drafts folder) and then forward the resulting message, which destroys
secondary context such as threading, unless of course I have a tool that
does all of this on my behalf. Even then, somebody can take the embedded
message out of its context and all of the intended features and benefits
will be lost.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Sat, 04 May 2002 15:47:49 -0400
Sender: [email-masked]

> I consider the Relayed action an admission that the infrastructure refuses
> to play along. 

hey, anytime you're upgrading the infrastructure, you have to interface
with legacy components - particularly when you don't control all of that
infrastructure.

> If the first-hop submission server is always generating Relayed messages
> because the firewall doesn't understand the envelope extensions, how
> useful is NOTIFY=SUCCESS really.

hey, if firewall vendors can't implement the specifications properly,
the vendors who build those firewalls deserve to lose.  unfortunately, 
too few folks understand the harm that firewalls do to applications and 
their limitations in providing security.

> > > You don't think UTF-8 in Received header fields would benefit admins
> > > who have to deal with email from International sources?
> > 
> > let's put it this way - I get a lot of spam from international sources,
> > and the domain names in the received headers aren't very useful as it
> > is.  how much more useful will they be in utf-8?
> 
> How much more useful would they be for network admins who don't speak
> english as a primary or secondary language?

they're useless now except as opaque tokens to feed to whois or 
by prepending abuse@ and mailing to them.   putting them in utf-8
won't increase their utility.  at least today, most people have
keyboards that can type in domain names.  that won't be the case if
domains in received fields use utf-8.  actually that would be a good
reason to insist that received fields always use ASCII.

> > that's just great.  you've more than doubled the amount of space that
> > this one header would require
> 
> As opposed to the complexity that the existing spec mandates, which the
> majority (not the plurality, the majority) cannot seem to implement

and exactly how will putting them in XML make the spec clearer?

> > and you've made headers considerably more difficult to read than at
> > present.
> 
> I would suggest that the encoding above is much simpler and cleaner than a
> group address with multiple ACE sequences.

nope, because the ACE parses just like any other domain name.

Keith

Date: Sat, 04 May 2002 14:20:23 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> > Part of my argument is that they were designed that way *partially*
> > because the current envelope-exchange environment requires it.
> 
> I don't really think so.

I consider the Relayed action an admission that the infrastructure refuses
to play along. It is certainly a useful message but one that is required
partially because there is an infrastructure requirement at all.

If the first-hop submission server is always generating Relayed messages
because the firewall doesn't understand the envelope extensions, how
useful is NOTIFY=SUCCESS really.

SMTP is a lot like IP. It even has a miniature ICMP (in the form of
delivery failures messages). Except in the case of envelope extensions,
where the analogy would be that every router has to upgraded for every new
IP option.

> > You don't think UTF-8 in Received header fields would benefit admins
> > who have to deal with email from International sources?
> 
> let's put it this way - I get a lot of spam from international sources,
> and the domain names in the received headers aren't very useful as it
> is.  how much more useful will they be in utf-8?

How much more useful would they be for network admins who don't speak
english as a primary or secondary language? ACE gobbledygook doesn't help
those administrators one iota.

> > Using something like the XML structured approach, it is simple enough
> > to define all recipients as groups:
> >
> >   <PrimaryRecipient>
> >     <DisplayName>Local Admins</DisplayName>
> >     <Mailbox>fred@local</Mailbox>
> >     <Mailbox>joe@remote</Mailbox>
> >   </PrimaryRecipient>
> 
> that's just great.  you've more than doubled the amount of space that
> this one header would require

As opposed to the complexity that the existing spec mandates, which the
majority (not the plurality, the majority) cannot seem to implement

> you haven't added a shred of functionality

Group addresses working for all fields at no additional development cost
isn't an improvement?

> and you've made headers considerably more difficult to read than at
> present.

I would suggest that the encoding above is much simpler and cleaner than a
group address with multiple ACE sequences.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Sat, 04 May 2002 15:18:25 -0400
Sender: [email-masked]

> > > Your own client didn't
> > > even generate valid References headers until I pointed it out.
> >
> > my mail client doesn't generate References fields at all.
> 
> Had to dig through my archives. I meant In-Reply-To. See attached.

it generated valid in-reply-to fields as of the time the MUA was
written.  and 2822 was intended to be compatible with 822, you know,
so 2822 readers should have been able to parse them.

Date: Sat, 04 May 2002 14:10:32 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------5C3A15C10D8C9CE894298E5F
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Keith Moore wrote:

> > Your own client didn't
> > even generate valid References headers until I pointed it out.
> 
> my mail client doesn't generate References fields at all.

Had to dig through my archives. I meant In-Reply-To. See attached.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/
--------------5C3A15C10D8C9CE894298E5F
Content-Type: message/rfc822
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Message-ID: <[email-masked]>
Date: Fri, 01 Feb 2002 13:06:16 -0600
From: "Eric A. Hall" <[email-masked]>
X-Mailer: Mozilla 4.78 [en] (Windows NT 5.0; U)
X-Accept-Language: en-US,en
MIME-Version: 1.0
To: Keith Moore <[email-masked]>
Subject: Re: MUA?
References: <[email-masked]>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Keith Moore wrote:
> 
> > What MUA are you using?
> 
> nmh.  why?

The In-Reply-To header is not in conformance with RFC 2822.

  In-reply-to: Your message of "Fri, 01 Feb 2002 13:02:29 CST."
             <[email-masked]> 

RFC 2822, 3.6.4. Identification fields

  in-reply-to     =       "In-Reply-To:" 1*msg-id CRLF

  msg-id          =       [CFWS] "<" id-left "@" id-right ">" [CFWS]

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

--------------5C3A15C10D8C9CE894298E5F--

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Sat, 04 May 2002 15:03:48 -0400
Sender: [email-masked]

> Keith Moore wrote:
> 
> > that may be the only thing *you* care about, but DSNs were designed to
> > provide a partial response (i.e. the best indication that could
> > be provided) even if the destination mailer doesn't support them.
> 
> Part of my argument is that they were designed that way *partially*
> because the current envelope-exchange environment requires it. 

I don't really think so.  One of the things I (and several other folks) 
didn't like about the return-receipt-to header behavior is that it wasn't 
reliable - you couldn't simply add the header field and expect it to 
give you a notificaiton.   

> > > We need a new message format if we want to allow UTF-8 directly in
> > > the message headers.
> > 
> > provides no benefit to users besides minimal bandwidth savings.
> 
> You don't think UTF-8 in Received header fields would benefit admins who
> have to deal with email from International sources?

let's put it this way - I get a lot of spam from international sources,
and the domain names in the received headers aren't very useful as it
is.  how much more useful will they be in utf-8?  they'll just be more
varied and less likely to be understood.  and the domain names are about
the only parts of a received field that could make use of utf-8.

> Using something like the XML structured approach, it is simple enough to
> define all recipients as groups:
> 
>   <PrimaryRecipient>
>     <DisplayName>Local Admins</DisplayName>
>     <Mailbox>fred@local</Mailbox>
>     <Mailbox>joe@remote</Mailbox>
>   </PrimaryRecipient>

that's just great.  you've more than doubled the amount of space that
this one header would require, you haven't added a shred of functionality,
and you've made headers considerably more difficult to read than at present.

> > > This doesn't even take into consideration the multitude of codecs
> > > and encoding forms that people have to deploy.
> > 
> > if the folks doing audio and video can't agree on a small number of
> 
> I'm talking about header field codecs. 2047, 2231, etc.

I agree that it would be better if message headers had been designed
from the start being able to transmit arbitrary content transparently,
but the number of encoders/decoders that we're likely to see is 
somewhat less than a multitude.  And I'd far rather have to deal with
a small number of these frobs than to replace the entire mail system
and have to suffer through all of this transition pain again.  
(yes, again - some of us have already had to suffer through transitions
from uucp, mail-11, bitnet, multiple flavors of x.400, all-for-nothing, 
notes, etc.)

> > > Flowing and quoting in body parts would be nice, and would be easier
> 
> > first, someone needs to demonstrate that it can be made to work well
> > by defining a new content-type (or a variant of an existing
> > content-type) that does this. once that's done, there's no need for a
> > new message format.
> 
> Whether or not it needs (or benefits from) a new message depends on the
> necessary support services. You can do both of those today with HTML if
> you are willing to live in the current framework.

you can display this stuff with HTML.  however I've yet to see an effective 
interface for replying to an HTML message while editing the previous message 
to remove extraneous parts, adding additional text, and keeping straight 
which parts were from earlier messages.  (granted I don't play with Windows
or Mac software very often, so it's possible that one exists that I haven't
seen)

Keith

Date: Sat, 04 May 2002 13:43:06 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> that may be the only thing *you* care about, but DSNs were designed to
> provide a partial response (i.e. the best indication that could
> be provided) even if the destination mailer doesn't support them.

Part of my argument is that they were designed that way *partially*
because the current envelope-exchange environment requires it. I fully
appreciate the fact that a relay will tell me that the message has made it
as far as X because Y refuses to play along.

> what's the point of having such a facility if you don't have a
> reasonable assurance of getting a notification when you ask for one?

I agree that this is a valid concern and a hard problem. I do not think it
is unsolvable, nor do I think it is so much of a concern that it overrides
all of the other potential benefits.

> > We need a new message format if we want to allow UTF-8 directly in
> > the message headers.
> 
> provides no benefit to users besides minimal bandwidth savings.

You don't think UTF-8 in Received header fields would benefit admins who
have to deal with email from International sources?

> it's unfortunate that groups often aren't implemented correctly.  they
> are very useful, and there would be a need for them in a new format
> also.

Using something like the XML structured approach, it is simple enough to
define all recipients as groups:

  <PrimaryRecipient>
    <DisplayName>Local Admins</DisplayName>
    <Mailbox>fred@local</Mailbox>
    <Mailbox>joe@remote</Mailbox>
  </PrimaryRecipient>

> > This doesn't even take into consideration the multitude of codecs
> > and encoding forms that people have to deploy.
> 
> if the folks doing audio and video can't agree on a small number of

I'm talking about header field codecs. 2047, 2231, etc.

> > Flowing and quoting in body parts would be nice, and would be easier

> first, someone needs to demonstrate that it can be made to work well
> by defining a new content-type (or a variant of an existing
> content-type) that does this. once that's done, there's no need for a
> new message format.

Whether or not it needs (or benefits from) a new message depends on the
necessary support services. You can do both of those today with HTML if
you are willing to live in the current framework.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sat, 04 May 2002 09:24:43 -0700 (PDT)
From: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
To: "Eric A. Hall" <[email-masked]>
Cc: Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

> Keith Moore wrote:

> > but in most cases this is inherent in the nature of the feature, and
> > has nothing whatsoever to do with the message format.   if you could
> > add the feature at all, you could do so without changing the message
> > format and without the disruption that would cause.

> This particular discussion is in relation to envelope and protocol
> extensions, which has nothing to do with the *current* message format. The
> point of this particular discussion is that a message format which
> provided envelope data as a separate blob would allow the envelope
> extensions to survive blackhole hops.

> DSNs with NOTIFY=SUCCESS are an obvious example. The only thing I care
> about is that delivery succeeds, meaning that this is a negotation between
> the originator and the delivery server. The submission server and any
> relays in between are completely irrelevant, yet due to the current
> envelope-exchange design, they dictate the success of that feature.

OK, so let's do the case analysis for this one. Suppose this is done by a new
envelope extension defined after deployment of an infrastructure that doesn't
lose unrecognized extensions along the way. But wait a minute -- if this is a
new extension that's added after initial deployment an old delivery agent won't
recognize it either. So all of a sudden your notification of successful
delivery has become unreliable and dependent on an infrastructure upgrade of
delivery agents. This is worse than the present DSN scheme, where you always
get something back saying what happened.

The way you solve such problems is well known, of course: Criticality flags.
You add a flag to each extension to indicate whether or not recognition of it
is required for proper message handling. But unless you want this to revert to
being equivalent to the old DSN scheme you immediately need multiple
criticality flags -- in this case the extension must be recognized for delivery
and nowhere else. Pretty quickly you arrive at something where you'll have
submission, relay, and delivery criticality at a minimum. This gets to be
complex given that the boundaries between the operations tend to blur over
time.

Regardless, you've just reinvented an aspect of X.400, which had criticality
flags galore. And experience with X.400 taught us some interesting things about
criticality flags: Requirements that say criticality  must be honored tend to
get ignored. Because of this extensions based on them being honored tend to
work quite poorly in practice.

Here's why. Suppose a get an extension I don't recognize that's marked as
critical for whatever operation I'm performing. I can either ignore it or honor
it. If I honor it I have to return an error to somebody somewhere, typically by
bouncing the message. This tends to get people pretty seriously upset. They
don't like it when mail bounces for reasons they don't understand and don't
care about and they don't like it when they get error messages they don't
understand.

The alternative of delivering the message, however, at least gets the mail
through, albeit while violating the standard. Now, it may be encrypted or
improperly formatted or otherwise damaged by the failure to honor the flag, but
this tends to get blamed on the message originator, not on the transport
process.

I'm a many year veteran of X.400 implementation and deployment, and I can tell
you this doesn't play out well at all in practice. As ad-hoc extensions get
developed and deployed (and they will no matter what the rules for such things
are) and cause disruption in message flow implementations will do what is
necessary to succeed. At most implementations will end up having a setting to
operate in standards compliant mode, and it won't be set very often.

And even if this plays out differently than it did in the X.400 space and the
bulk of implementations honor the flags, you still have a problem. A delivery
agent that doesn't recognize your NOTIFY=SUCCESS is still going to return an
error of some sort to someone. It isn't going to return something that's
recognizable as a delivery notification no matter what. So the result is that
the extension is inherently unreliable in a way that the current DSN extension
is not. And the way you fix that is by making the extension critical at every
step so you can detect where the failure is going to occur along the path and
return a sensible indication of what happened. In short, you end up reinventing
the current DSN extension even though your new infrastructure is more capable
in some ways than the present one.

> You are correct in that this feature and others which need to survive
> blackhole hops do not require a new message format by themselves. After
> all, these are envelope extensions, which do not *currently* use the
> message format directly.

Actually, an (admittedly complex) single SMTP extension to allow for envelope
elements that can pass through relays that don't understand them could be
defined now. You don't need a new message format for that. The reason this
hasn't been done is because when you actually start looking at the problem
space closely it hasn't been justifiable to do this.

> However, another critical aspect of what I am talking about is that the
> transer headers be separated from the message headers and moved into the
> envelope blob. That *does* require a new message format. Not only is the
> separation of transfer and message headers needed for basic features (such
> as header encryption)

Header signatures and encryption can be accomplished by using nested messages.
Once again a new message format isn't required.

> but the recombination of transfer headers with
> envelope data is also necessary for other basic features (transfer header
> survivability). So no, I don't think you can add new features without also
> modifying the message format.

This is another place where lessons learned from X.400 tell us the space isn't
shaped the way you think it is. I really don't want to get the reasons why this
is so now, however. I've written too much as it is.

				Ned

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Sat, 04 May 2002 13:06:55 -0400
Sender: [email-masked]

> Keith Moore wrote:
> 
> > but in most cases this is inherent in the nature of the feature, and
> > has nothing whatsoever to do with the message format.   if you could
> > add the feature at all, you could do so without changing the message
> > format and without the disruption that would cause.
> 
> This particular discussion is in relation to envelope and protocol
> extensions, which has nothing to do with the *current* message format. The
> point of this particular discussion is that a message format which
> provided envelope data as a separate blob would allow the envelope
> extensions to survive blackhole hops.

even supposing that it would be useful to change the format of the
envelope, there's no reason to change the format of the message
that's delivered to the UA.

> DSNs with NOTIFY=SUCCESS are an obvious example.  The only thing I care
> about is that delivery succeeds, meaning that this is a negotation between
> the originator and the delivery server. The submission server and any
> relays in between are completely irrelevant, yet due to the current
> envelope-exchange design, they dictate the success of that feature.

that may be the only thing *you* care about, but DSNs were designed to
provide a partial response (i.e. the best indication that could
be provided) even if the destination mailer doesn't support them.
what's the point of having such a facility if you don't have a reasonable
assurance of getting a notification when you ask for one?
and in practice destinations sometimes disable NOTIFY=SUCCESS, even
when the MTA supports it, for security reasons.

> However, another critical aspect of what I am talking about is that the
> transer headers be separated from the message headers and moved into the
> envelope blob. That *does* require a new message format. 

no it doesn't - you could make this work without any change to UAs
at all.  assuming, of course, that there were any benefit to the user
in doing so, which is doubtful.

> > which user-visible features need this kind of change?
> 
> We need a new message format if we want to allow UTF-8 directly in the
> message headers. 

provides no benefit to users besides minimal bandwidth savings.

> We need a new message format if we want to be able to encrypt the full
> message (message headers and body, rather than just the body), since the
> commingling of transfer and message headers prevents that now.

a new content-type would solve this problem.
 
> It would also be really nice if we could simply the header rules as part
> of this effort. The multitude of header rules currently defined are
> obtuse, wildly varied, and prone to significant misinterpretation. 

you don't have to change the message format to fix this.

> I did a
> survey recently of clients that support "Group Names: user1, user2;" and
> only 2 out of 10 clients properly handled them. 

it's unfortunate that groups often aren't implemented correctly.  they 
are very useful, and there would be a need for them in a new format also.

> Your own client didn't
> even generate valid References headers until I pointed it out. 

my mail client doesn't generate References fields at all.  but it's not
reqiured to do so.  if I wanted it to do so, it would be a simple matter
of adding a line to a configuration file.

> This
> doesn't even take into consideration the multitude of codecs and encoding
> forms that people have to deploy. 

if the folks doing audio and video can't agree on a small number of codecs
that solve a wide range of problems, I don't see how mail folks can help them.
still, this is just a matter of content-types, it doesn't require a change
to the message format.

> Flowing and quoting in body parts would be nice, and would be easier with 
> a new message format.

first, someone needs to demonstrate that it can be made to work well
by defining a new content-type (or a variant of an existing content-type)
that does this.  once that's done, there's no need for a new message format.

Keith

Date: Sat, 04 May 2002 10:43:43 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> but in most cases this is inherent in the nature of the feature, and
> has nothing whatsoever to do with the message format.   if you could
> add the feature at all, you could do so without changing the message
> format and without the disruption that would cause.

This particular discussion is in relation to envelope and protocol
extensions, which has nothing to do with the *current* message format. The
point of this particular discussion is that a message format which
provided envelope data as a separate blob would allow the envelope
extensions to survive blackhole hops.

DSNs with NOTIFY=SUCCESS are an obvious example. The only thing I care
about is that delivery succeeds, meaning that this is a negotation between
the originator and the delivery server. The submission server and any
relays in between are completely irrelevant, yet due to the current
envelope-exchange design, they dictate the success of that feature.

You are correct in that this feature and others which need to survive
blackhole hops do not require a new message format by themselves. After
all, these are envelope extensions, which do not *currently* use the
message format directly.

However, another critical aspect of what I am talking about is that the
transer headers be separated from the message headers and moved into the
envelope blob. That *does* require a new message format. Not only is the
separation of transfer and message headers needed for basic features (such
as header encryption) but the recombination of transfer headers with
envelope data is also necessary for other basic features (transfer header
survivability). So no, I don't think you can add new features without also
modifying the message format.

> which user-visible features need this kind of change?

We need a new message format if we want to allow UTF-8 directly in the
message headers. You will probably argue that a UI facade will work just
as well for this, but you will be wrong.

We need a new message format if we want to be able to encrypt the full
message (message headers and body, rather than just the body), since the
commingling of transfer and message headers prevents that now.

It would also be really nice if we could simply the header rules as part
of this effort. The multitude of header rules currently defined are
obtuse, wildly varied, and prone to significant misinterpretation. I did a
survey recently of clients that support "Group Names: user1, user2;" and
only 2 out of 10 clients properly handled them. Your own client didn't
even generate valid References headers until I pointed it out. This
doesn't even take into consideration the multitude of codecs and encoding
forms that people have to deploy. [2]822 is popping at the seams from the
20 lbs of crap getting jammed into this 10 lb sack.

Plenty of others. Flowing and quoting in body parts would be nice, and
would be easier with a new message format.

As single items, none of those warrant a new message format. Collectively
(especially with other benefits that become viable with a new format, as
with the envelope modifications above), they do warrant it.

> > On the other hand, what is it that is not feasible about a new
> > message and transfer service?
> 
> 1. getting users to buy into it when it is disruptive and it doesn't
> provide them with any immediate benefit.

If the features were desirable and it simplified implementation, people
would use it. It wouldn't happen overnight, we can agree on that.

> 2. getting past second-system effect from the large number of people
> who will see it as an opportunity to fix things (many of which aren't
> broken) and to introduce new problems in the process.

You will also note that Brooks provides advice on avoiding it.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: Sam Roberts <[email-masked]>
cc: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Sat, 04 May 2002 11:09:42 -0400
Sender: [email-masked]

> I'm confused. You said that you foresaw an eventual overhaul of the
> message format. If not for the reasons Eric is talking about,
> for what reasons?

I don't know what reasons Eric is talking about, so I can't speak about
those.

When I made my earlier statement, I was thinking that it would take
some really signifcant, generally-usable improvements in the service
provided by the mail system in order to justify the pain of 
transitioning to a completely new system.  Since that time I've 
concluded that none of these improvements require significant changes
to the message format.  They would however require significant changes
to SMTP (though probably doable within the current extension 
mechanism) and significant additional infrastructure.

So now I would say that a significant change to the message format 
does not seem to be technically justified by any improvements to
the mail system that I've thought of or seen proposed. 

Keith

Date: Sat, 4 May 2002 10:19:35 -0400
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Fri, May 03, 2002 at 07:11:03PM -0400:
> > What I am thinking about is that we have an opportunity to do all of this,
> > as well as to fix several other problems. None of these issues alone
> > warrant an overhaul, but cumulatively they certainly do. 
> 
> which issues are the ones that warrant an overhaul of the message format?    
> 
> > Moreover, they are actually feasible as part of an overhaul.
> 
> it's not clear that the overhaul is feasible.

I'm confused. You said that you foresaw an eventual overhaul of the
message format. If not for the reasons Eric is talking about,
for what reasons?

Sam

! Subject: Re: IDN (was Did anyone tell Microsoft yet?)
! From: Keith Moore <[email-masked]>
! Date: Fri, 26 Apr 2002 13:59:24 -0400
! cc: [email-masked]
! To: "Charles Lindsey" <[email-masked]>
! 
! 
! > Has anybody looked into the transitional issues that would arise if Email
! > was to be moved into allowing UTF-8 headers?
! 
! yes, though perhaps not in sufficient detail.
! 
! frankly I think it would be easier to transition to a completely different
! mail format.  that is, I think that having two slightly different
! mail formats (one ascii, another utf-8) is more confusing and more
! likely to lead to interoperability problems than having two obviously
! distinct formats which require explicit translation between the two.
! 
! > Or are you resigned to the fact that we shall still be encoding headers
! > using RFC 2047 in 40 years time even though all transports and operating
! > systems will be 8bit (and even bibary) clean by that time?
! 
! please don't cite 40 years of extrapolation from current practice as
! 'the fact'.
! 
! yes, I expect that there will eventually be a transition to a new message
! format.
! 
! no, I don't think that 8bit transparency of mail transports alone is
! sufficient to warrant the pain of that transition - because zero
! additional functionality is gained beyond a small bandwidth savings,
! and the pain of having to upgrade millions of installations of hundreds
! of different mail programs (MTAs, UAs, list servers, mail filters,
! POP servers, IMAP servers, etc.) is too substantial for that minimal
! gain - which is mostly wanted for the sake of purity.  it doesn't
! lessen complexity, it increases it - because mail handling programs
! would still have to recognize the old format as well as the new one
! for many years.
! 
! what would be worth the pain of transition?  a hypothetical redesign
! of the internet email system that is both reliable (in that you can
! expect messages to be delivered with integrity) and safe to use (in
! that it won't expose your systems to security threats).
! 
! my guess is that this is at least 10-15 years away though.  there's
! just too much investment in the current infrastructure.
! 
! Keith

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Sat, 04 May 2002 09:37:12 -0400
Sender: [email-masked]

> > > the use of an envelope container would certainly "fix" the
> > > problem that we have now with feature blackholes.
> 
> > > Surely you consider these blackholes as "real problems"?
> >
> > no, I don't - because most of the (mail) transport level features I
> > can think of are inherently per-hop and require per-hop negotiation.
> 
> "most" is certainly the right word. If you are arguing that some services
> need to be rejected if they are not understood, then that is certainly a
> valid design point to consider. But as it stands, no new features can be
> deployed between willing participants without upgrading all of the systems
> in between those end-points, and this has to be performed for every new
> extension.

but in most cases this is inherent in the nature of the feature, and 
has nothing whatsoever to do with the message format.   if you could
add the feature at all, you could do so without changing the message
format and without the disruption that would cause.

> > which issues are the ones that warrant an overhaul of the message
> > format?
> 
> The collective lot of them.

I didn't name a single feature that would require any change to the
message format, and I haven't seen any other suggestions for features
that need a change to the message format.  so I ask again, which
user-visible features need this kind of change?

> > > Moreover, they are actually feasible as part of an overhaul.
> >
> > it's not clear that the overhaul is feasible.
> 
> We already know that many of the desirable features are absolutely not
> feasible in the current model, at least not without significant hammering.

no we don't.

> On the other hand, what is it that is not feasible about a new message and
> transfer service? 

1. getting users to buy into it when it is disruptive and it doesn't 
provide them with any immediate benefit. 

2. getting past second-system effect from the large number of people who
will see it as an opportunity to fix things (many of which aren't broken)
and to introduce new problems in the process.

Keith

Date: Sat, 4 May 2002 09:05:23 +0000
From: Alan Barrett <[email-masked]>
To: Ken Hirsch <[email-masked]>
Cc: [email-masked]
Subject: Re: MHonArc support for format=flowed
Sender: [email-masked]

On Wed, 01 May 2002, Ken Hirsch wrote:
> http://www.imc.org/ietf-822/mail-archive/maillist.html
> 
> (As an aside, does anybody else see Chinese characters on that page?
> When I use IE6 on Windows XP, the characters +ietf- display as the
> Unicode character U+89EB, which means "to tremble with fear."  If
> anybody at Microsoft can explain this, I'd really be interested.)

In UTF-7, "+ietf-" decodes to U+89EB followed by a decoding error.  I
have no idea why IE6 is trying to interpret the page as UTF-7.

--apb (Alan Barrett)

Date: Fri, 03 May 2002 19:05:25 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> > the use of an envelope container would certainly "fix" the
> > problem that we have now with feature blackholes.

> > Surely you consider these blackholes as "real problems"?
> 
> no, I don't - because most of the (mail) transport level features I
> can think of are inherently per-hop and require per-hop negotiation.

"most" is certainly the right word. If you are arguing that some services
need to be rejected if they are not understood, then that is certainly a
valid design point to consider. But as it stands, no new features can be
deployed between willing participants without upgrading all of the systems
in between those end-points, and this has to be performed for every new
extension.

> which issues are the ones that warrant an overhaul of the message
> format?

The collective lot of them.

> > Moreover, they are actually feasible as part of an overhaul.
> 
> it's not clear that the overhaul is feasible.

We already know that many of the desirable features are absolutely not
feasible in the current model, at least not without significant hammering.
On the other hand, what is it that is not feasible about a new message and
transfer service? Backwards compatibility with legacy systems and
mailboxes is a migratory issue, and that is absolutely feasible.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, Graham Klyne <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Fri, 03 May 2002 19:11:03 -0400
Sender: [email-masked]

> There are several "real problems" that a new message format could solve.
> For example, the use of an envelope container would certainly "fix" the
> problem that we have now with feature blackholes. Currently, a single
> legacy server between two DSN-aware systems prevents that service from
> functioning, while an envelope block that contained this data as a
> separate unit could survive these kinds of systems (not 821 hosts, but
> hosts which were aware of the new format but which were unaware of a
> particular envelope extension). Surely you consider these blackholes as
> "real problems"?

no, I don't - because most of the (mail) transport level features I can
think of are inherently per-hop and require per-hop negotiation. 

let's put it this way - what real problems would you hope to solve by
having end-to-end conveyance of (mail) transport level features,
and what's preventing you from adding such features to SMTP  (say
by adding a "generic feature" frob to MAIL and/or RCPT) with considerably 
less barrier to deployment and less disruption to the installed base?  

> > If you want to do something useful for email, how about figuring out
> > how to make it really reliable, or spam-proof, or virus proof?
> 
> What I am thinking about is that we have an opportunity to do all of this,
> as well as to fix several other problems. None of these issues alone
> warrant an overhaul, but cumulatively they certainly do. 

which issues are the ones that warrant an overhaul of the message format?    

> Moreover, they are actually feasible as part of an overhaul.

it's not clear that the overhaul is feasible.

Keith

Date: Fri, 03 May 2002 18:02:13 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Graham Klyne <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Keith Moore wrote:

> It seems to me that we should be trying to solve real problems rather
> than trying to invent new things that serve no purpose except to
> disrupt the installed base.

There are several "real problems" that a new message format could solve.
For example, the use of an envelope container would certainly "fix" the
problem that we have now with feature blackholes. Currently, a single
legacy server between two DSN-aware systems prevents that service from
functioning, while an envelope block that contained this data as a
separate unit could survive these kinds of systems (not 821 hosts, but
hosts which were aware of the new format but which were unaware of a
particular envelope extension). Surely you consider these blackholes as
"real problems"?

> If you want to do something useful for email, how about figuring out
> how to make it really reliable, or spam-proof, or virus proof?

What I am thinking about is that we have an opportunity to do all of this,
as well as to fix several other problems. None of these issues alone
warrant an overhaul, but cumulatively they certainly do. Moreover, they
are actually feasible as part of an overhaul.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Graham Klyne <[email-masked]>, [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft ye 
Date: Fri, 03 May 2002 18:33:12 -0400
Sender: [email-masked]

> It seems to me that we should be cataloging the kinds of requirements that
> we want from a new message. 

It seems to me that we should be trying to solve real problems rather than 
trying to invent new things that serve no purpose except to disrupt the 
installed base.

If you want to do something useful for email, how about figuring out how to 
make it really reliable, or spam-proof, or virus proof?

Keith

From: Marc Mutz <[email-masked]>
Organization: KDE
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 3 May 2002 23:44:24 +0200
Content-Description: clearsigned data
Sender: [email-masked]

On Friday 03 May 2002 15:22, Charles Lindsey wrote:
<snip>
> Although actually, simply encoding it with RFC 2047 might be a sensible
> move. It is not strictly allowed, but we have just seen that lots of
> mailers do it,

Correction: Most mailers simply encode the whole addr-spec (some with, some 
sans the enclosing <>'s), only your mailer (dtmail?) did a good job at not 
encoding the @<>'s.

However, the changes to existing mailers would probably be minimal.
Following is the complete patch[1] to make KMail skip the @, so that it's 
output is the same as dtmails:
- -const QString dontQuote = "\"()<>,";
+const QString dontQuote = "\"()<>,@";

> and it would at least ensure the mail for moderators
> arrived at a site which had an interest in disentangling it.

Marc

[1] though KMail has other problems than that w.r.t. addresses: It doesn't 
    (yet) have a rfc822 tokenizer :-(

- -- 
Marc Mutz <[email-masked]>

Date: Fri, 03 May 2002 15:07:23 -0500
From: "Eric A. Hall" <[email-masked]>
To: Graham Klyne <[email-masked]>
CC: [email-masked]
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tellMicrosoft  ye
Sender: [email-masked]

Graham Klyne wrote:

> I can't resist mentioning
> http://www.ietf.org/internet-drafts/draft-klyne-message-rfc822-xml-03.txt

That's a pretty good approach. I like the potential for separating the
transfer headers from the message headers. I also like the potential for a
namespace of fields, since that allows people to develop their own headers
as they wish, without undue fear of conflict. Towards the latter point, I
was considering the use of OIDs for labels, although the ASCII lovers will
probably prefer XML to OIDs.

It seems to me that we should be cataloging the kinds of requirements that
we want from a new message. My bet is that some of these wishes will also
indicate a requirement for a new transfer protocol as well. Some of the
things I would like to see:

 o Direct UTF-8 in the header field values (duh)

 o Bodies that default to flowed line-wrapping, eg without requiring
   fixed line-lengths in the transfer protocol

 o Trace-Level header, allowing me to set "0" for "no trace data",
   "1" for "errors only", "2" for "errors and warnings" through
   "5" for "delivery notifications and everything else"

 o Separate envelope/trace block for end-to-end reliability purposes
   (envelope corruption is a rare problem but still needs fixing)

 o Recursively-signed relay data. EG, the submission server signs
   the envelope/trace block after creating the initial data, the
   next-hop MTA signs the envelope/trace block after it modifies
   the data, etc. The intention here is that the  reverse-path data
   should be able to be validated by any node in the path.

There are some obvious protocol requirements from the above. Some other
ones that would be nice:

 o Need a way to interrupt the message transfer without sending
   <CRLF>.<CRLF> sequence or dropping carrier.

 o New commands and response codes should be configurable via
   namespace identifiers, such as OIDs. Every response associated
   with any command should have a unique identifier specifically
   for that command/response pair.

 o Full-service routing, including remote expansion of local-parts

Some of these are easier than others.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Fri, 03 May 2002 15:53:15 -0400
Sender: [email-masked]

> Actually, it is not as daft as all that. If an MTA needs to know whether
> it is dealing with an "internationalized" MUA, then there may be several
> ways it could be done. Obviously, one is an EHLO on port 25, followed by a
> response "yes, I speak International". But using a different port number
> might be a possible alternative.

this doesn't work with NAPTs.  and port #s are a lousy protocol negotiation
mechanism.

To: [email-masked]
Xref: clerew local.mime:1347
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 3 May 2002 13:16:00 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Paul Smith <[email-masked]> writes:

>It would work better if the latin equivalent was 'china-timber.cn' rather 
>than 'A863B350DHRS-25323-52GH.A4GWT' (or whatever the Unicoded translation 
>into ASCII would be). This would make the job a LOT easier for (almost) 
>everyone involved, as non IDN capable MTAs would only have to do 
>comparisons on ASCII text rather than Unicode text etc, and non-local 
>senders could use the ASCII text version of the domain name.

Well it would be more like zz--sdfmklef-esf.cn, because the IDNA is
applied separately to each component of the domain name. So you at least
know that you are dealing with China.

>You don't need to 'build an entire mail system from scratch'. Simply, for 
>instance, run IDN capable MTAs/MUAs on port 2025 (or whatever), and 
>everyone else continue on port 25, with gateways 'sanitising' things as 
>they pass from the IDN -> non-IDN world. I agree that it's a daft idea, but 
>not TOTALLY ludicrous :-)

Actually, it is not as daft as all that. If an MTA needs to know whether
it is dealing with an "internationalized" MUA, then there may be several
ways it could be done. Obviously, one is an EHLO on port 25, followed by a
response "yes, I speak International". But using a different port number
might be a possible alternative.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1348
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 3 May 2002 13:22:29 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>... This leaves the question of non-ASCII local-parts 
>unaddressed. I'm not sure they need to be.

On the contrary, this whole thread started because we have a problem on
Usefor with certain automatically generated local-parts (mail to
moderators). So a solution is needed in the reasonably near future. But
obviously, it is better to do it in conjunction with whatever solution is
adopted for domain names.

Although actually, simply encoding it with RFC 2047 might be a sensible
move. It is not strictly allowed, but we have just seen that lots of
mailers do it, and it would at least ensure the mail for moderators
arrived at a site which had an interest in disentangling it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1346
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 3 May 2002 13:08:42 GMT
Lines: 33
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> A few well-placed servers that routinely bounced everything that was not
>> standards-compliant would work wonders :-( .

>First of all, this is email, not news. The notion of a "well-placed server"
>is pretty much undefined in email these days.

Yes, well there was a smiley there. I didn't really expect things to change.

>Second, this has been tried quite a few times, and the results of doing it are
>known: There is an immediate groundswell of outrage from the people whosee mail
>no longer gets delivered and who don't give a rat's ass about standards. The
>system is returned to a liberal mode of operation in short order, and the
>people responsible for this reckless and irresponsible act are usually
>disciplined.

But I still think this is a short sighted policy. ISPs and others
essentially have the choice of enforcing the rules, at the expense of
losing a few customers, or or maintaining comprehensive abuse and support
departments at even greater expense. A few of them are cluefull enough to
realise this.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Fri, 03 May 2002 13:52:38 -0400
Sender: [email-masked]

> The agents that do RFC 2047 incorrectly are invariably (almost) MUAs,
> since MTAs have almost no reason to look inside that encoding. But if MUAs
> cannot even get RFC 2047 right, why do you insist on recommending that
> they be given the more demanding (and critical) job of doing IDNA right?

first, because unlike phrases and human-readable text, MUAs are already 
having to identify addresses within headers and treat them specially -
this is already a critical function for them.

second, because the code necessary to do normalization and encoding
will be available in libraries that everyone can use - the MUA vendor
won't have to get that part right.

and of course the normalization would have to be done even with pure UTF-8.
the ASCII encoding step is fairly simple and isn't likely to introduce 
many errors.

that's not to say that everybody will get IDNA exactly right either. but 
the chance of the overall system working reliably when only the MUAs have 
to deal with the new addresses is far greater than the chance that the
overall system will work reliably when several components in the signal
path each have to function properly.  and if IDNA changes are mostly 
isolated in MUAs then it's also far easier to find and fix any bugs 
that result.

Keith

Date: Fri, 3 May 2002 07:28:30 +0200 (CEST)
From: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
To: [email-masked]
Sender: [email-masked]

On  3 May, [email-masked] wrote:
> description of the local-part,  which post RFC 2047 can be expressed
> in a range of alphabets. My expectation is that with the introduction
> of IDNs, the display-name will have to begin describing the
> domain-name as well. This leaves the question of non-ASCII local-parts
> unaddressed. I'm not sure they need to be.

The local parts of the addresses definitely needs to support non
us-ascii characters. Consider the very common naming scheme
[email-masked]. There are a lot of people out there
who have non us-ascii characters in their names.

	/Martin Forssén

To: [email-masked]
Xref: clerew local.mime:1345
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 3 May 2002 13:04:31 GMT
Lines: 81
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> >Such a change inhernetly causes widespread disruption.
>> 
>> Again, you are assuming what you are trying to prove, which is no way to
>> argue.

>This is a decision that has to be based on engineering judgement.  
>There's no way to "prove" whether any particular choice will be 
>successful or not because you can't adequately model the real-world 
>conditions.

>> Ned said, right at the start of this thread, that introducing UTF-8 was
>> not inherently hard or difficult, but that there were a certain number of
>> problems that would need to be tackled, amongst which would likely be a
>> suitable means of falling back to a 7-bit encoding.

>It's not difficult to define how it would need to work, and it's not
>particularly difficult to write the code.  Introducing the code without
>significant disruption is extremely difficult, and the benefit doesn't
>justify the cost.

On the contrary, defining how it would work is the difficult part, because
the definition has to include provision for a smooth transition from the
present system. That will likely include fallbacks to the present
notation, and the hard part is deciding just where those fallbacks should
take place.

>> So yes, it would have to be introduced with care, and tradeoffs between
>> various approaches would have to be looked into.
>> 
>> But the way forward would be to examine the problems to find ways to work
>> around them, rather than to declare in advance, without proof, that is is
>> impossible.

>No, that's not a way forward, it's a way backward.  Because by trying to 
>force a transition to pure utf-8 you end up not only increasing the 
>complexity of MUAs but also of the MTAs that have to negotiate and translate.

But for that reason the utf-8 may turn out to be not so "pure" as you seem
to suppose. Clearly, existing MUAs are going to continue in use for some
considerable time amongst the large number of people who have no
particular need to correspond regularly in chinese. These people may
occasionally see unpronounceable addresses (i.e. post-IDNA ASCII) and will
be well advised to put them in their address books, because typing them by
hand will surely be error-prone :-( .

But people with a regular need to correspond with people in China will
likely be willing to spend money to buy an "internationalized" MUA. Such
agents do not exist yet, so the opportunity now exists to define how they
are to work. That includes deciding whether it is safe to let them do the
IDNA encoding themselves (clearly doing so will be one of the options),
but it also allows the possibility of bringing in utf-8 at the same time.

The interesting question then is whether it will be necessary, useful or
desirable for the MTA to be aware that it is speaking to an
internationalized MUA.

>And by putting this translation at additional places in the signal path
>you decrease the liklihood that the message will arrive intact.  
>Many implementations today can't even encode RFC 2047 correctly.  So how 
>can we expect tomorrow's implementations to translate correctly between 
>UTF-8 and IDNA+2047? 

The agents that do RFC 2047 incorrectly are invariably (almost) MUAs,
since MTAs have almost no reason to look inside that encoding. But if MUAs
cannot even get RFC 2047 right, why do you insist on recommending that
they be given the more demanding (and critical) job of doing IDNA right?
It just ain't going to happen.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: Martin =?ISO-8859-1?Q?Forss=E9n?= <[email-masked]>
Cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
From: Nick Shelness<[email-masked]>
Date: Fri, 3 May 2002 10:04:35 +0100
Sender: [email-masked]

Martin,

> The local parts of the addresses definitely needs to support non
> us-ascii characters. Consider the very common naming scheme
> [email-masked]. There are a lot of people out there
> who have non us-ascii characters in their names.

I fear that we're on the verge of wandering into the swamp of "Names, 
Addresses & Routes -- They're Different". The convention to which you 
refer was promulgated by Henk (I can't remember his last name) of Phillips 
and adopted by the EEMA in the late 80s. It never got any further, which 
is why it's far from "common". Henk was trying to import into everyday 
Internet mail a feature of X.400 O/R Addresses. To whit, that: a) the 
personal-name was structured into distinct (first-name, surname, initials, 
and generational-qualifier) components, and b) under-specified 
personal-names (missing one or more components) that uniquely matched a 
fully specified personal-name were considered a match. Henk's argument was 
that by adopting the X.400 personal-name encoding convention of RFC 987 
for all local-parts, it would be possible to employ similar matching 
conventions for Internet mail, and that this would allow local-parts to be 
guessed with an acceptably high probability of success. In practice, this 
never happened, though some organizations did adopt his convention, among 
a number of others. 

Moving on, there is clearly rough consensus that it makes no sense to 
restrict domain-names (yes, names) to an ASCII alphabet, though clearly a 
pure ASCII encoding is required for backward compatibility.  I think that 
achieving a similar consensus for local-parts will be difficult. I for 
one, will take a lot of convincing. We already have a means of associating 
display-names, expressed in a range of alphabets with mailbox addresses 
(yes, addresses). I also have the sense that people, as distinct from 
corporations, like short local-parts. The last six posting to this list 
(nick, maf, mutz, moore, sroberts, and even ned-ietf-822) would seem to back me up. 

Nick

Nick Shelness
Independent Technology Consultant

Contact Details
   Office Tel: +44 (0) 1828 640 632
   Office Fax: +44 (0) 1828 640 647
   email: [email-masked]

From: Keith Moore <[email-masked]>
To: Paul Smith <[email-masked]>
cc: Keith Moore <[email-masked]>, "Charles Lindsey" <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Thu, 02 May 2002 20:44:21 -0400
Sender: [email-masked]

> At 17:16 02/05/2002 -0400, Keith Moore wrote:
> > > >Second, you're failing to consider that there's essentially no incentive
> > > >for a significant percentage of the world's mail users or MTA operators
> > > >to upgrade.  An approach that relies on MUA upgrades to provide the new
> > > >functionality lets those who benefit from the new functionality - those
> > > >who have an incentive - do the upgrade and get immediate benefit. This
> > > >gets it deployed more quickly for those users who need it.
> > >
> > > I think this is the key issue.
> > >
> > > People in the UK, USA (and a large percentage of Europe) won't want to
> > > fiddle with things that ain't broke just so that some people can send them
> > > emails that they can't understand... This means that the majority of
> > > Internet users  will be running mail systems which people using IDN
> > > wouldn't be able to contact in the near/mid future.
> >
> >actually, they'll be able to contact each other.  it's just that the IDNs
> >will be displayed in encoded form on the systems that haven't been upgraded,
> >and the user with the non-upgraded MUA will have to type in the ugly encoded
> >form of the address (or copy-and-paste that address into his address book).
> 
> Assuming the MTAs & MUAs can handle it properly.

since IDNA is syntactically compatible with existing ascii domain names,
it's likely that they will handle it properly.  after all, that's how
IDNA was designed to work.

> >if you substitute "ascii equivalent" for "latin equivalent", that's
> >essentially what IDNA does.
> 
> It would work better if the latin equivalent was 'china-timber.cn' rather 
> than 'A863B350DHRS-25323-52GH.A4GWT' (or whatever the Unicoded translation 
> into ASCII would be). 

people can still set up their own human-friendly ASCII aliases if they want.
IDNA doesn't preclude this.

> >arrgh!  sure, let's build an entire mail system from scratch  and run two
> >incompatible mail systems just so that half of the users can have one
> >additional feature.  makes *lots* of sense...
> 
> If the alternative is to break everything else in the world, then it COULD 
> make a lot of sense. (If it won't break anything else, then what's this 
> discussion all about then :-) )

IDNA seems to be a solution with minimum breakage and minimum deployment
effort.  It works with existing MTAs, MUAs, whatever without creating a 
need to run new servers and agents, and without causing the existing 
servers and agents to fail.  Others have proposed solutions which appear
to have considerably more breakage and more deployment cost in exchange
for a very marginal performance gain.

> EG, you could have modifications to the RCPT and MAIL commands so that 
> messages are transferred (in the IDN 'world') as something like:
> 
> RCPT TO: <[email-masked]> ASCII=[email-masked]

no, because it wouldn't solve any problems.  users don't generally care 
about what happens at the SMTP level, so they don't care whether the 
addresses at that level are encoded in ASCII or not.

> Obviously, people in IDN 'worlds' would have to have two 'names' (one 
> 'local', one 'ASCII') for each user if they are going to be communicated to 
> from other countries, but that would probably make sense anyway. (If there 
> isn't an ASCII= extension, you could either bounce the message when it goes 
> out of the IDN 'world', or you could send it unchanged and hope for the 
> best :-) )
> 
> If it worked this way, then the only people who would need to worry about 
> IDN would be people who wanted to use it - everyone else would keep things 
> EXACTLY as they are now.

the IDNA solution already has this property, and requires fewer changes 
than what you are proposing.

Keith

To: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
From: [email-masked]
Date: Fri, 3 May 2002 00:01:54 +0100
Sender: [email-masked]

My sense is that the demand for IDNs arose primarily in the web space. It 
is entirely reasonable that businesses with well known, 
non-ASCII-expressable, names would want to employ them directly in URLs. 
Clearly once IDNs are out there, they will start leaching into mailbox 
addresses. Displayed directly in their ASCII form, they may be 
unintelligible, but so too will be their decoded UNICODE equivalent to 
many foreigners. Those of us with a long history in email remember the 
days of totally unintelligible local parts (i.e. UX1234), and even today, 
who the hell is JimB? In these cases, we depend on RFC 2822 display-names 
to represent a meaningful description of the local-part,  which post RFC 
2047 can be expressed in a range of alphabets. My expectation is that with 
the introduction of IDNs, the display-name will have to begin describing 
the domain-name as well. This leaves the question of non-ASCII local-parts 
unaddressed. I'm not sure they need to be.

Nick

Nick Shelness
Independent Technology Consultant

Contact Details
   Office Tel: +44 (0) 1828 640 632
   Office Fax: +44 (0) 1828 640 647
   email: [email-masked]

Date: Thu, 02 May 2002 23:18:22 +0100
To: Keith Moore <[email-masked]>
From: Paul Smith <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Cc: Keith Moore <[email-masked]>, "Charles Lindsey"   <[email-masked]>, [email-masked]
Sender: [email-masked]

At 17:16 02/05/2002 -0400, Keith Moore wrote:
> > >Second, you're failing to consider that there's essentially no incentive
> > >for a significant percentage of the world's mail users or MTA operators
> > >to upgrade.  An approach that relies on MUA upgrades to provide the new
> > >functionality lets those who benefit from the new functionality - those
> > >who have an incentive - do the upgrade and get immediate benefit. This
> > >gets it deployed more quickly for those users who need it.
> >
> > I think this is the key issue.
> >
> > People in the UK, USA (and a large percentage of Europe) won't want to
> > fiddle with things that ain't broke just so that some people can send them
> > emails that they can't understand... This means that the majority of
> > Internet users  will be running mail systems which people using IDN
> > wouldn't be able to contact in the near/mid future.
>
>actually, they'll be able to contact each other.  it's just that the IDNs
>will be displayed in encoded form on the systems that haven't been upgraded,
>and the user with the non-upgraded MUA will have to type in the ugly encoded
>form of the address (or copy-and-paste that address into his address book).

Assuming the MTAs & MUAs can handle it properly.

>if you substitute "ascii equivalent" for "latin equivalent", that's
>essentially what IDNA does.

It would work better if the latin equivalent was 'china-timber.cn' rather 
than 'A863B350DHRS-25323-52GH.A4GWT' (or whatever the Unicoded translation 
into ASCII would be). This would make the job a LOT easier for (almost) 
everyone involved, as non IDN capable MTAs would only have to do 
comparisons on ASCII text rather than Unicode text etc, and non-local 
senders could use the ASCII text version of the domain name.

>Frankly, I don't think the demand for non-ascii addresses is from folks
>who want to use such addresses to communicate with speakers of other
>languages - I think it's from people who want to use them when
>communicating with speakers of their own language.

Agreed - thus...

> > If you want Chinese (for instance) emails just to have a scope of
> > China, then why not have a new set of protocols ('UDNS', 'USMTP' etc) 
> which
> > handle Unicode addressing, with gateways run by ISPs in those countries to
> > convert 'USMTP' to SMTP to go to the rest of the world.
>
>arrgh!  sure, let's build an entire mail system from scratch  and run two
>incompatible mail systems just so that half of the users can have one
>additional feature.  makes *lots* of sense...

If the alternative is to break everything else in the world, then it COULD 
make a lot of sense. (If it won't break anything else, then what's this 
discussion all about then :-) )

You don't need to 'build an entire mail system from scratch'. Simply, for 
instance, run IDN capable MTAs/MUAs on port 2025 (or whatever), and 
everyone else continue on port 25, with gateways 'sanitising' things as 
they pass from the IDN -> non-IDN world. I agree that it's a daft idea, but 
not TOTALLY ludicrous :-)

Alternatively, you could have part of the 'IDN MTA' standard to have a new 
ESMTP capability, and if an IDN capable MTA talks to a non-IDN capable MTA, 
the IDN capable MTA has to 'sanitise' the data in a carefully prescribed 
manner before sending it on. (This is probably a lot less daft than my 
previous suggestion :-) )

EG, you could have modifications to the RCPT and MAIL commands so that 
messages are transferred (in the IDN 'world') as something like:

RCPT TO: <[email-masked]> ASCII=[email-masked]

Then, when an IDN MTA talks to a non-IDN MTA, it simply replaces the RCPT 
data with the 'ASCII=' text.

Obviously, people in IDN 'worlds' would have to have two 'names' (one 
'local', one 'ASCII') for each user if they are going to be communicated to 
from other countries, but that would probably make sense anyway. (If there 
isn't an ASCII= extension, you could either bounce the message when it goes 
out of the IDN 'world', or you could send it unchanged and hope for the 
best :-) )

If it worked this way, then the only people who would need to worry about 
IDN would be people who wanted to use it - everyone else would keep things 
EXACTLY as they are now.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

From: Keith Moore <[email-masked]>
To: Paul Smith <[email-masked]>
cc: Keith Moore <[email-masked]>, "Charles Lindsey" <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Thu, 02 May 2002 17:16:08 -0400
Sender: [email-masked]

> >Second, you're failing to consider that there's essentially no incentive
> >for a significant percentage of the world's mail users or MTA operators
> >to upgrade.  An approach that relies on MUA upgrades to provide the new
> >functionality lets those who benefit from the new functionality - those
> >who have an incentive - do the upgrade and get immediate benefit. This
> >gets it deployed more quickly for those users who need it.
> 
> I think this is the key issue.
> 
> People in the UK, USA (and a large percentage of Europe) won't want to 
> fiddle with things that ain't broke just so that some people can send them 
> emails that they can't understand... This means that the majority of 
> Internet users  will be running mail systems which people using IDN 
> wouldn't be able to contact in the near/mid future.

actually, they'll be able to contact each other.  it's just that the IDNs 
will be displayed in encoded form on the systems that haven't been upgraded,
and the user with the non-upgraded MUA will have to type in the ugly encoded
form of the address (or copy-and-paste that address into his address book).

(as a practical matter, folks who often correspond with speakers of
other languages will probably want to have an easy-to-type email
address - e.g. one consisting of a short sequence of ASCII-only characters.)

> All non-latin email addresses/domains would either have to have a latin 
> equivalent, or would just have a scope of the country of origin. If all 
> addresses have to have a latin equivalent, then, simply, why not use that 
> for all the 'important' stuff, and just use the local script for display 
> purposes. 

if you substitute "ascii equivalent" for "latin equivalent", that's 
essentially what IDNA does.  

Frankly, I don't think the demand for non-ascii addresses is from folks 
who want to use such addresses to communicate with speakers of other 
languages - I think it's from people who want to use them when 
communicating with speakers of their own language.

> If you want Chinese (for instance) emails just to have a scope of 
> China, then why not have a new set of protocols ('UDNS', 'USMTP' etc) which 
> handle Unicode addressing, with gateways run by ISPs in those countries to 
> convert 'USMTP' to SMTP to go to the rest of the world.

arrgh!  sure, let's build an entire mail system from scratch  and run two
incompatible mail systems just so that half of the users can have one 
additional feature.  makes *lots* of sense...

Keith

Date: Thu, 02 May 2002 22:05:55 +0100
To: Keith Moore <[email-masked]>, "Charles Lindsey" <[email-masked]>
From: Paul Smith <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Cc: [email-masked]
Sender: [email-masked]

> > The view I have been trying to put across is that the point where
> > normalization occurs is the weakest link in the system, and that
> > therefore such points should be concentrated so that they are few
> > in number and easily fixed. From that POV, User Agents fail on both counts.
>
>The view is over-simplistic for at least two reasons.  First is that
>you're comparing systems of very different total complexity -
>you will not rid UAs of the burden of supporting unicode (including
>normalization) but you are adding complexity to parts of the system
>that don't need to support unicode, thus adding more opportunities
>for failure.  In other words, your proposal is adding more weak links
>without strengthening the weak links that you're worried about.
>
>Second, you're failing to consider that there's essentially no incentive
>for a significant percentage of the world's mail users or MTA operators
>to upgrade.  An approach that relies on MUA upgrades to provide the new
>functionality lets those who benefit from the new functionality - those
>who have an incentive - do the upgrade and get immediate benefit. This
>gets it deployed more quickly for those users who need it.

I think this is the key issue.

People in the UK, USA (and a large percentage of Europe) won't want to 
fiddle with things that ain't broke just so that some people can send them 
emails that they can't understand... This means that the majority of 
Internet users  will be running mail systems which people using IDN 
wouldn't be able to contact in the near/mid future.

IMHO, some people seem to be missing a key problem. If people start having 
(for instance) email addresses which can ONLY be shown using Chinese 
characters, how on earth is the average westerner supposed to send them an 
email message (ie they won't be able to type in the email address)? All 
non-latin email addresses/domains would either have to have a latin 
equivalent, or would just have a scope of the country of origin. If all 
addresses have to have a latin equivalent, then, simply, why not use that 
for all the 'important' stuff, and just use the local script for display 
purposes. If you want Chinese (for instance) emails just to have a scope of 
China, then why not have a new set of protocols ('UDNS', 'USMTP' etc) which 
handle Unicode addressing, with gateways run by ISPs in those countries to 
convert 'USMTP' to SMTP to go to the rest of the world.

(Just to throw another daft idea into the ring :-) )

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Thu, 02 May 2002 13:10:31 -0400
Sender: [email-masked]

> >this problem isn't specific to IDNA. it exists with any use of Unicode
> >to represent strings that need to be compared for equivalence.
> 
> Not necessarily. We are dealing with twp forms of normalization in
> Nameprep. One to Normalization Form C or KC, and one from upper case to
> lower case. Generally speaking, there are two schools of thought as to
> when Normalization should be done: Early and Late.
> 
> The official advice of the Unicode people is to do it Early (see
> draft-duerst-i18n-norm-04.txt). My view is that that should be as early as
> the keyboard driver in the operating system, so that applications can
> assume, without further testing, that textual data is already normalized.
> However, that is not appropriate for upper to lower case normalization.
> 
> The view of the IDNA people seems to be to do it Late, i.e. just before
> the call of gethostbyname, or even inside gethostbyname eventually.

I don't think that's a fair representation.  You certainly have to do 
the normalization before encoding in ASCII, so if the normalization wasn't 
done before calling gethostbyname, it had better be done there.  It
won't hurt for gethostbyname to normalize a string that's already
normalized.  This doesn't preclude the app normalizing the string earlier
than that, and in many cases there are good reasons for doing so.

> The point is that "ordinary applications" are not supposed to care. 

another point is that apps that aren't IDN-aware are not supposed to care.

> The view I have been trying to put across is that the point where 
> normalization occurs is the weakest link in the system, and that 
> therefore such points should be concentrated so that they are few 
> in number and easily fixed. From that POV, User Agents fail on both counts.

The view is over-simplistic for at least two reasons.  First is that
you're comparing systems of very different total complexity - 
you will not rid UAs of the burden of supporting unicode (including 
normalization) but you are adding complexity to parts of the system 
that don't need to support unicode, thus adding more opportunities 
for failure.  In other words, your proposal is adding more weak links 
without strengthening the weak links that you're worried about.  

Second, you're failing to consider that there's essentially no incentive 
for a significant percentage of the world's mail users or MTA operators
to upgrade.  An approach that relies on MUA upgrades to provide the new 
functionality lets those who benefit from the new functionality - those 
who have an incentive - do the upgrade and get immediate benefit. This 
gets it deployed more quickly for those users who need it.

Keith

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Thu, 02 May 2002 12:57:03 -0400
Sender: [email-masked]

> A few well-placed servers that routinely bounced everything that was not
> standards-compliant would work wonders :-( .

not necessarily.  a significant percentage of servers won't accept bounces
or don't supply a valid return address.

Date: Thu, 02 May 2002 09:42:00 -0700 (PDT)
From: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> [email-masked] writes:

> >> I tend to think that agents that try to correct "apparent" errors do more
> >> harm than good, though they certainly seem to be the norm in the email
> >> world.

> >This is a lovely theorectical discussion of how things could work. But before I
> >bother responding with my own take on this please show me the magic wand you
> >intend to wave that will undo 25 years and countless millions of lines of code
> >in common use today that don't work the way you think things should.

> A few well-placed servers that routinely bounced everything that was not
> standards-compliant would work wonders :-( .

First of all, this is email, not news. The notion of a "well-placed server"
is pretty much undefined in email these days.

Second, this has been tried quite a few times, and the results of doing it are
known: There is an immediate groundswell of outrage from the people whosee mail
no longer gets delivered and who don't give a rat's ass about standards. The
system is returned to a liberal mode of operation in short order, and the
people responsible for this reckless and irresponsible act are usually
disciplined.

I've dealt with situations where sales were lost over this issue.

				Ned

To: [email-masked]
Xref: clerew local.mime:1331
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Thu, 2 May 2002 10:01:40 GMT
Lines: 44
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> The chief problem with using IDNA in any User Agent is the Nameprep
>> operation (see my message to this list a couple of days ago). It requires
>> the presence of large tables of Unicode transformations and forbidden
>> characters. These tables may well change over time as Unicode evolves. 

>this problem isn't specific to IDNA. it exists with any use of Unicode 
>to represent strings that need to be compared for equivalence.

Not necessarily. We are dealing with twp forms of normalization in
Nameprep. One to Normalization Form C or KC, and one from upper case to
lower case. Generally speaking, there are two schools of thought as to
when Normalization should be done: Early and Late.

The official advice of the Unicode people is to do it Early (see
draft-duerst-i18n-norm-04.txt). My view is that that should be as early as
the keyboard driver in the operating system, so that applications can
assume, without further testing, that textual data is already normalized.
However, that is not appropriate for upper to lower case normalization.

The view of the IDNA people seems to be to do it Late, i.e. just before
the call of gethostbyname, or even inside gethostbyname eventually.

The point is that "ordinary applications" are not supposed to care. IMO
User Agents are ordinary applications. However, that may me too simplistic,
but the only way to move forward is to look at particular cases and see
which approaches appear to have the best chance of success. The view I
have been trying to put across is that the point where normalization
occurs is the weakest link in the system, and that therefore such points
should be concentrated so that they are few in number and easily fixed.
>From that POV, User Agents fail on both counts.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1332
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Thu, 2 May 2002 10:03:59 GMT
Lines: 25
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> I tend to think that agents that try to correct "apparent" errors do more
>> harm than good, though they certainly seem to be the norm in the email
>> world.

>This is a lovely theorectical discussion of how things could work. But before I
>bother responding with my own take on this please show me the magic wand you
>intend to wave that will undo 25 years and countless millions of lines of code
>in common use today that don't work the way you think things should. 

A few well-placed servers that routinely bounced everything that was not
standards-compliant would work wonders :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 02 May 2002 08:21:58 -0700 (PDT)
From: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
To: Keith Moore <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Sender: [email-masked]

> > >Such a change inhernetly causes widespread disruption.
> >
> > Again, you are assuming what you are trying to prove, which is no way to
> > argue.

> This is a decision that has to be based on engineering judgement.
> There's no way to "prove" whether any particular choice will be
> successful or not because you can't adequately model the real-world
> conditions.

> > Ned said, right at the start of this thread, that introducing UTF-8 was
> > not inherently hard or difficult, but that there were a certain number of
> > problems that would need to be tackled, amongst which would likely be a
> > suitable means of falling back to a 7-bit encoding.

Well, as long as we're complaining about argument techniques, what I said
was that it isn't hard to introduce but I'd yet to see evidence that the
benefits outweight the costs. You're quoting out of context.

> It's not difficult to define how it would need to work, and it's not
> particularly difficult to write the code.  Introducing the code without
> significant disruption is extremely difficult, and the benefit doesn't
> justify the cost.

Exactly. There's a huge difference between defining the mechanism and
writing the code and actually suffering through the deployment. I probably
see sllightly less disruption in store than Keith does, but I probably also
see less overall benefit. So it's a wash -- I agree with Keith's assessment
of this.

This is what has stopped me writing the relatively trivial documents needed to
specify a mechanism for this. Getting the mechanism right isn't exactly rocket
science, you know.

				Ned

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Thu, 02 May 2002 09:28:58 -0400
Sender: [email-masked]

> >Such a change inhernetly causes widespread disruption.
> 
> Again, you are assuming what you are trying to prove, which is no way to
> argue.

This is a decision that has to be based on engineering judgement.  
There's no way to "prove" whether any particular choice will be 
successful or not because you can't adequately model the real-world 
conditions.

> Ned said, right at the start of this thread, that introducing UTF-8 was
> not inherently hard or difficult, but that there were a certain number of
> problems that would need to be tackled, amongst which would likely be a
> suitable means of falling back to a 7-bit encoding.

It's not difficult to define how it would need to work, and it's not
particularly difficult to write the code.  Introducing the code without
significant disruption is extremely difficult, and the benefit doesn't
justify the cost.

> So yes, it would have to be introduced with care, and tradeoffs between
> various approaches would have to be looked into.
> 
> But the way forward would be to examine the problems to find ways to work
> around them, rather than to declare in advance, without proof, that is is
> impossible.

No, that's not a way forward, it's a way backward.  Because by trying to 
force a transition to pure utf-8 you end up not only increasing the 
complexity of MUAs but also of the MTAs that have to negotiate and translate.
And by putting this translation at additional places in the signal path
you decrease the liklihood that the message will arrive intact.  
Many implementations today can't even encode RFC 2047 correctly.  So how 
can we expect tomorrow's implementations to translate correctly between 
UTF-8 and IDNA+2047? 

Keith 

Date: Thu, 02 May 2002 13:32:00 +0200
From: Marc Mutz <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
To: "Eric A. Hall" <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Organization: KDE
Content-description: clearsigned data
Sender: [email-masked]

On Wednesday 01 May 2002 00:28, Eric A. Hall wrote:
> Marc Mutz wrote:
> > The main problem is therefore the domain part. I must admit that I
> > haven't yet read the IDN/punycode draft(s?), but what is the problem
> > with using that for the domain?
>
> The biggest problem is that 2047 would not produce labels that conform to
> the historically significant subset of letters-digits-hyphen (LDH) which
> constitute the hostname syntax used by most applications.
<snip>

You misunderstood me here. I meant: "what's the problem with using punycode 
for the domain part". rfc2047 was re: local-part.

Marc

- -- 
Marc Mutz <[email-masked]>

To: [email-masked]
Xref: clerew local.mime:1330
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Thu, 2 May 2002 09:21:17 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> >who is going to pay for such a change?  who is going to want to deal
>> >with the disruption that it would cause?
>> 
>> Nobody, because nobody (except you apparently) imagines that it would be
>> done in a manner which caused such widespread disruption.

>Such a change inhernetly causes widespread disruption.

Again, you are assuming what you are trying to prove, which is no way to
argue.

Ned said, right at the start of this thread, that introducing UTF-8 was
not inherently hard or difficult, but that there were a certain number of
problems that would need to be tackled, amongst which would likely be a
suitable means of falling back to a 7-bit encoding.

So yes, it would have to be introduced with care, and tradeoffs between
various approaches would have to be looked into.

But the way forward would be to examine the problems to find ways to work
around them, rather than to declare in advance, without proof, that is is
impossible.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: "Ken Hirsch" <[email-masked]>
To: <[email-masked]>
Subject: MHonArc support for format=flowed
Date: Wed, 1 May 2002 23:17:10 -0400
Sender: [email-masked]

I've created an experimental version of MHonArc with support for format=flowed.
I've submitted it for comment on the MHonArc mailing list, and thought I should get
some feedback here.  For anybody that doesn't know, MHonArc is the program that
creates web archives of mailing lists, including
http://www.imc.org/ietf-822/mail-archive/maillist.html

(As an aside, does anybody else see Chinese characters on that page?  When I use IE6
on Windows XP, the characters +ietf- display as the Unicode character U+89EB, which
means "to tremble with fear."  If anybody at Microsoft can explain this, I'd really
be interested.)

I have a demonstration of the output at:
http://personal.rdu.bellsouth.net/~kahirsch/maillist.html, using just the
format=flowed messages from the ietf-822 mailing list.

You can download the Perl source for the revised module at
http://personal.rdu.bellsouth.net/~kahirsch/mhtxtplain.pl

Here are the highlights, for those who know Perl:
Here are the highlights:
(1) define the HTML formatting for flowed quotes
(line 45 in revised mhtxtplain.pl)
$StartFlowedQuote = '<blockquote style=' .
 '"border-left: #0000ff solid 2px; margin-left: 5px; padding-left: 5px">' .
  "\n";
$EndFlowedQuote   = "</blockquote>\n";

(2) process the body
(line 303)
  if ($textformat eq 'flowed') {
    # experimental support for format=flowed Ken Hirsch (May 2002)
    # text/plain; format=flowed defined in RFC2646
    my $currdepth = 0;
    my $ret='';
    s!^</?x-flowed>\r?\n>!!mg; # we don't know why Eudora puts these in
    while (length($$data)) {
      $$data =~ /^((?:&gt;)*)/;
      my $qd = $1;
      if ($$data =~ s/^(.*(?:\n$qd(?!&gt;).*)*\n?)//) {
          # divide message into chunks by "quote-depth",
          # which is the number of leading > signs
          my $chunk = $1;
          $chunk =~ s/^$qd ?//mg; # N.B. also takes care of space-stuffing
          $chunk =~ s/^-- $/--/mg; # special case for '-- '
          if ($chunk =~ / \r?\n/) {
            # Treat this chunk as format=flowed
            # Lines that end with spaces are
            # considered to have soft line breaks.
            # Lines that end with no spaces are
            # considered to have hard line breaks.
            $chunk =~ s/(?<! )(\r?\n|\Z)/<br>$1/g;

          } else {
            # Treat this chunk as format=fixed
            if ($nonfixed) {
                $chunk =~ s/(\r?\n)/<br>$1/g;
                if ($keepspace) {
                    $chunk =~ s/^(.*)$/&preserve_space($1)/gem;
                }
            } else {
                $chunk = "<pre>\n" . $chunk . "</pre>\n";
            }
          }
          my $newdepth = length($qd)/length('&gt;');
          if ($currdepth < $newdepth) {
            $chunk = $StartFlowedQuote x ($newdepth - $currdepth) .  $chunk;
          } elsif ($currdepth > $newdepth) {
            $chunk = $EndFlowedQuote x ($currdepth - $newdepth) . $chunk;
          }
          $currdepth = $newdepth;
          $ret .= $chunk;
      } else {
        # I think the above regex will always match, but
        # I put this in here to catch any weird cases
        # so there's no infinite loop
        print STDERR "DEQUOTINGERROR($$data)\n";
        $ret .= $$data;
        last;
      }
    }
    if ($currdepth > 0) {
      $ret .= $EndFlowedQuote x $currdepth;
    }
    $$data = $ret;
  } else {

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Wed, 01 May 2002 12:56:01 -0400
Sender: [email-masked]

> The chief problem with using IDNA in any User Agent is the Nameprep
> operation (see my message to this list a couple of days ago). It requires
> the presence of large tables of Unicode transformations and forbidden
> characters. These tables may well change over time as Unicode evolves. 

this problem isn't specific to IDNA. it exists with any use of Unicode 
to represent strings that need to be compared for equivalence.

Keith

Date: Wed, 01 May 2002 17:27:09 +0100
To: [email-masked] (Kai Henningsen)
From: Graham Klyne <[email-masked]>
Subject: Re: Dreaming about replacements (was IDN (was Did anyone tell Microsoft ye
Cc: [email-masked]
Sender: [email-masked]

At 11:57 AM 5/1/02 +0200, Kai Henningsen wrote:
>While not universally loved, that immediately brings to mind XML. Which
>conveniently also has an existing solution to the character set problem.
>AND many existing implementations.
>
>I assume data would still have MIME types, the parameters would just be
>encoded differently - I see no reason to invent a different data typing
>scheme here.

I can't resist mentioning 
http://www.ietf.org/internet-drafts/draft-klyne-message-rfc822-xml-03.txt

>Possibly the correct thing is to follow one XML blob (with everything that
>in a MIME message is in headers, including internal ones in multiparts)
>with a number of binary blob, possibly with a HTTP-1.1-like minimal length
>encoding ... hmm. Can't say I like that a lot ... but MIME-header-like
>versions don't really seem to be any better either.

The above proposal uses MIME multipart/related with a cid: URI reference 
from the XML header section.  For the (common?) case of pure text or XML 
payload, it can be embedded in the header, and a local fragment identifier 
used as the target reference.

#g

-------------------
Graham Klyne
<[email-masked]>

To: [email-masked]
Xref: clerew local.mime:1322
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Wed, 1 May 2002 11:19:11 GMT
Lines: 75
Sender: [email-masked]

In <[email-masked]> Marc Mutz <[email-masked]> writes:

>> To: Jürgen Schmidt <[email-masked]>

>It might be interesting to know how existing mailers handle this. Typing the 
>above into the TO: fields of various mailers gives:

<Interesting examples snipped>

Dtmail (as supplies with Solaris 7)

To: =?ISO-8859-1?Q?J=FCrgen?= Schmidt
        <=?ISO-8859-1?Q?j=FCrgen?=@=?ISO-8859-1?Q?tu-m=FCnchen?=.de>

But, to my surprise, it then sent it to localhost via SMTP, and localhost
invoked sendmail, which accepted it without demur and queued it for later
delivery. Here are some interesting extracts from the queue:

RPFD:<=?ISO-8859-1?Q?j=FCrgen?=@=?ISO-8859-1?Q?tu-m=FCnchen?=.de>
HReceived: from localhost (localhost [127.0.0.1])
        by clw.cs.man.ac.uk (8.9.1b+Sun/8.9.1) with SMTP id KAA16419
        for <=?ISO-8859-1?Q?j=FCrgen?=@=?ISO-8859-1?Q?tu-m=FCnchen?=.de>;
        Wed, 1 May 2002 10:08:29 +0100 (BST)
HX-Mailer: dtmail 1.3.0 CDE Version 1.3 SunOS 5.7 sun4m sparc

I am curious as to what bounces I shall get when the queue is run. I shall
report is there is anything interesting.

OK, the queue was run as I typed this, and my sendmail reported

501 <=?ISO-8859-1?Q?j=FCrgen?=@=?ISO-8859-1?Q?tu-m=FCnchen?=.de>: domain
missing or malformed

which seems fair enough. I am just surprised that it got as far as it did.

>From this, one might conclude that simply extending rfc2047 might be the 
>easiest upgrade path ;-)

It might well be a useful notation for the interface between the User
Agent and the MTA (but not on any wire).

>I think that the local-part is the smallest of the problems. local-parts are 
>uninterpreted and can thus carry any encoding that would be a valid 
>local-part in rfc2822, e.g. rfc2047. The burden to make sense out of this is 
>on the receiving end: If someone allows mailboxes with non-US-ASCII 
>characters on her machine, then she needs to make sure that she installs 
>tools that can handle the new encoding. For everyone else, the local-part 
>just looks funny.

>The main problem is therefore the domain part. I must admit that I haven't yet 
>read the IDN/punycode draft(s?), but what is the problem with using that for 
>the domain?

The chief problem with using IDNA in any User Agent is the Nameprep
operation (see my message to this list a couple of days ago). It requires
the presence of large tables of Unicode transformations and forbidden
characters. These tables may well change over time as Unicode evolves. I
just don't trust implementors of User Agents to get it right (and
especially I don't trust Bill Gates to get it right). There is a much
better chance of getting it right if it is done in the transport agents.
For example, if sendmail was REQUIRED by some standard to translate any
name-addr containing RFC 2047 encodings into IDNA, then my message to
Jürgen above might well get delivered.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 01 May 2002 08:59:30 -0700 (PDT)
From: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
To: Keith Moore <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Sender: [email-masked]

> > >who is going to pay for such a change?  who is going to want to deal
> > >with the disruption that it would cause?
> >
> > Nobody, because nobody (except you apparently) imagines that it would be
> > done in a manner which caused such widespread disruption.

> Such a change inhernetly causes widespread disruption.

You betcha.

				Ned

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Wed, 01 May 2002 10:43:01 -0400
Sender: [email-masked]

> >who is going to pay for such a change?  who is going to want to deal
> >with the disruption that it would cause?
> 
> Nobody, because nobody (except you apparently) imagines that it would be
> done in a manner which caused such widespread disruption.

Such a change inhernetly causes widespread disruption.

Keith

Date: Wed, 01 May 2002 07:05:38 -0700 (PDT)
From: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> Keith Moore <[email-masked]> writes:

> >In practice, things that handle mail (like mailing list software,
> >gateways, virus filters,  spam filters, etc. - certainly not just
> >MTAs) parse most of the structured fields for one reason or another.
> >Several of them also try to rewrite such fields or to correct
> >apparent errors in such fields - for instance, by taking strings
> >of illegal characters and quoting them according to RFC 2047
> >(regardless of where those characters appear in the header).

> I tend to think that agents that try to correct "apparent" errors do more
> harm than good, though they certainly seem to be the norm in the email
> world.

> On the Web, browsers that try to interpret malformed HTML have done untold
> harm, because they simply encourage sloppily constructed web sites. Hence
> the plethora of sites which say "this site can only be viewd in Internet
> Explorer", or Netscape, or whatever. And decent browsers like Opera get
> shunned because they refuse to play those silly games.

> Usefor is quite strict on this issue. If any transport receives malformed
> articles, either it passes them on *untouched* (most likely because it did
> not bother to check for that particular abnormality), or else it drops
> them on the floor *silently*. This is to ensure that every copy of a given
> article that is found on the network is identical (a few "variable"
> headers excepted). It was also done that way because transorts that tried
> to "correct" things usually did so incorrectly.

This is a lovely theorectical discussion of how things could work. But before I
bother responding with my own take on this please show me the magic wand you
intend to wave that will undo 25 years and countless millions of lines of code
in common use today that don't work the way you think things should. 

Without being able to effect such a change this sort of discussion is moot.

				Ned

To: [email-masked]
Xref: clerew local.mime:1321
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Wed, 1 May 2002 10:49:52 GMT
Lines: 46
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>yes, but the cost of persuading the whole email community to implement
>a change to UTF-8 in any timeframe significantly shorter than 20 years
>is so large that it seems totally infeasible.

Yes, but the purpose of this present discussion is to investigate whether
a change to UTF-8 is feasible without undue pain at the present time. So
asserting that it is is impossible with 20 years and than using that
assertion to prove that it is impossible is just a circular argument.

The real question is whether the change could be introduced in such a way
that that the whole email community would not be required to change
overnight. So one needs to consider:

1. Whether facilities for downgrading need to be specified, and if so how,
and where they should be applied.  The availability of RFC 2047 and RFC
2231 are a good start for that purpose, if needed, but other possibilities
would likely have to be looked at.

2. The syntactic contexts in which UTF-8 characters would be allowed. One
should start with the presently defined syntax (which allows them nowhere)
and define the relaxations on a case by case basis.

It is likely that the cases where RFC 2047 is allowed would all be safe.
Also quoted-strings in the value parts of parameters (as envisaged in RFC
2231). Also most other quoted-strings (but not all of them). Things that
remained in strict ASCII would likely include msg-ids, tokens and header
names (indeed, anything used as a keyword).

>who is going to pay for such a change?  who is going to want to deal
>with the disruption that it would cause? 

Nobody, because nobody (except you apparently) imagines that it would be
done in a manner which caused such widespread disruption.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1320
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Wed, 1 May 2002 10:23:24 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>In practice, things that handle mail (like mailing list software,
>gateways, virus filters,  spam filters, etc. - certainly not just 
>MTAs) parse most of the structured fields for one reason or another.  
>Several of them also try to rewrite such fields or to correct 
>apparent errors in such fields - for instance, by taking strings 
>of illegal characters and quoting them according to RFC 2047
>(regardless of where those characters appear in the header).

I tend to think that agents that try to correct "apparent" errors do more
harm than good, though they certainly seem to be the norm in the email
world.

On the Web, browsers that try to interpret malformed HTML have done untold
harm, because they simply encourage sloppily constructed web sites. Hence
the plethora of sites which say "this site can only be viewd in Internet
Explorer", or Netscape, or whatever. And decent browsers like Opera get
shunned because they refuse to play those silly games.

Usefor is quite strict on this issue. If any transport receives malformed
articles, either it passes them on *untouched* (most likely because it did
not bother to check for that particular abnormality), or else it drops
them on the floor *silently*. This is to ensure that every copy of a given
article that is found on the network is identical (a few "variable"
headers excepted). It was also done that way because transorts that tried
to "correct" things usually did so incorrectly.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 01 May 2002 11:57:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Dreaming about replacements (was IDN (was Did anyone tell Microsoft ye
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Keith Moore)  wrote on 30.04.02 in <[email-masked]>:

> and there's so little benefit in "pure UTF-8" that it's conceivable
> that it would never be worth the trouble to transition to it.  for
> example, "pure UTF-8" wouldn't rid the user agent of the need to
> parse message header fields and to treat different parts of a
> structured field in different ways.

Hmm. So *if* the real solution is a completely new format, that should  
address these things, too.

So what we would need - at least for headers - is some format that has a  
consistent scheme for structured information.

While not universally loved, that immediately brings to mind XML. Which  
conveniently also has an existing solution to the character set problem.  
AND many existing implementations.

I assume data would still have MIME types, the parameters would just be  
encoded differently - I see no reason to invent a different data typing  
scheme here.

Hmm. But the right way to combine possibly binary blobs with these headers  
isn't at all clear. We'd certainly like to avoid yet again ASCII-encoding  
binary data - that's one of the things we'd like to get away from, after  
all! So no gigantic CDATA sections.

Possibly the correct thing is to follow one XML blob (with everything that  
in a MIME message is in headers, including internal ones in multiparts)  
with a number of binary blob, possibly with a HTTP-1.1-like minimal length  
encoding ... hmm. Can't say I like that a lot ... but MIME-header-like  
versions don't really seem to be any better either.

Comments?

MfG Kai

Date: 01 May 2002 11:35:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Keith Moore)  wrote on 26.04.02 in <[email-masked]>:

> , there was
> significant objection at the time to *ever* expecting systems
> to be 8-bit clean,

There was *WHAT*?!

Now if I've ever heard of an offense in  protocol design worth shooting,  
this certainly qualifies.

That is absolutely inexcusable.

And I thought the original 7 bit decision was stupid. That was still far  
saner than *this*!

MfG Kai