Subject: [ietf-smtp] Is this a new bad i18n idea?

From: "John Levine" <[email-masked]>
Date: 21 May 2014 21:06:55 -0000

Over in DNSOP the usual arguments about CNAME and domain name variants
are being rerun.  (Short summary: too many people believe that
variants can be made "the same" purely within the DNS, because they
don't understand the problem.)

One of the harder issues is application configuration.  For some older
protocols like telnet, if you can resolve a name to an A or AAAA
you're done, but for interesting ones like mail and http, the server
also needs to know what domains it handles and what to do with them.
It's long been clear that any real solution to "the same" will involve
some way for application servers to configure themselves automatically
to match what the DNS expects them to handle.

So is this a new idea? Let's say your mail server gets a RCPT TO
for a domain it doesn't recognize, e.g. RCPT TO:<[email-masked]>.
Before it rejects it, it does a DNS lookup, and let's say it
finds this CNAME:
 
	fóó.com. IN CNAME foo.com.

It treats the message as though it were to [email-masked].* Poof, you
just set up the CNAMEs or DNAMEs and the mail server handles them
automagically.

I can think of a variety of reasons this would be a bad idea,
but before I start enumerating them, has someone done it before?

R's,
John

* - This is an example. I know you can't put a CNAME at the zone apex.
You can pretend it was fóó.xn--kprw13d. and the CNAME was synthesized
from the DNAME.

---

From: Ned Freed <[email-masked]>
Date: Thu, 22 May 2014 08:05:46 -0700 (PDT)

> So is this a new idea?

For i18n? Maybe. For email? No. If memory servers, ancient versions of sendmail
supported this and actually rewrote such aliases. 

It doesn't work well at all, in part because the DNS is a lousy choice for such
information operationally, but mostly because in practice domains require much
richer semantics: Sometimes it's a true alias, sometimes it's an overlay,
sometimes you want rewriting, sometimes you don't, sometimes you want it to
apply to arbitrary subdomains, sometimes you don't, etc. etc. etc.

The DNS just isn't flexible enough. Which means that you also have to store
information about the alias elsewhere, and ... remind me again why we're
using the DNS for this?

> Let's say your mail server gets a RCPT TO

> but before I start enumerating them, has someone done it before?

I've never seen a writeup of the issues anywhere.

> You can pretend it was fóó.xn--kprw13d. and the CNAME was synthesized

Understood.

				Ned

---

From: S Moonesamy <[email-masked]>
Date: Thu, 22 May 2014 08:38:42 -0700

Hi Ned,
>supported this and actually rewrote such aliases.

This is a configurable option in sendmail to do the rewrite.

>apply to arbitrary subdomains, sometimes you don't, etc. etc. etc.

Yes.

Regards,
S. Moonesamy

---

From: John C Klensin <[email-masked]>
Date: Thu, 22 May 2014 12:04:22 -0400

--On Thursday, 22 May, 2014 08:05 -0700 Ned Freed
<[email-masked]> wrote:

>...
>> for a domain it doesn't recognize, e.g. RCPT TO:<bob@f?om>.
>> 	f?om. IN CNAME foo.com.
>> It treats the message as though it were to [email-masked].*
>> Poof, you just set up the CNAMEs or DNAMEs and the mail
>> server handles them automagically.

> I've never seen a writeup of the issues anywhere.

I haven't either, except for 

(1) one case mentioned fairly often (and coincidentally
explained in a lecture today that I'll eventually get online):
it won't work on any predictable way, resulting in user
astonishment.  

(2) The example, as written, is simply invalid.  SMTP does not
allow single-labels domains.  They are explicitly prohibited in
5321 and its predecessors.  We've had several conversations
about doing that and concluded not to.   So, if the example SMTP
server does that DNS lookup, it in violation of the spec and
nothing meaningful can be said about what is done with it.

For the first case and more generally, the problem is that these
examples keep getting written the way this one was, as bob@f?om
or bob@b?ar.f?om.  If f?om and b?ar are intended to represent
IDN labels, the first case is invalid (see above) and the second
one involves the "you only get one shot at an alias" problem,
e.g., if 
   f?om. IN CNAME foo.com.
exists, then there can be no b?ar.f?om because "f?om" usually
cannot have both a CNAME and a delegation record.   b?ar\.f?om)
is another matter, but let's not go there and there is some
question about what happens when a zone break does not occur,
adding up to a situation in which very similar FQDNs get very
different treatment.

If they do not represent, both cases are invalid because 5321
does not allow "?" in a domain name (remember that the DNS specs
cite SMTP as the reason for the "preferred syntax" (aka "LDH")
recommendation).  

FWIW, while we removed the restriction that required canonical
names, a discussion with Jon Postel prior to the DRUMS
conclusions identified little issues like the above as part of
the reason the prohibition existed initially, along with the
observation that one could get entirely different behavior out
of the (entirely flat) host table.

    john

---

From: Keith Moore <[email-masked]>
Date: Thu, 22 May 2014 12:10:36 -0400

> to match what the DNS expects them to handle.
No, that's broken.   DNS should never dictate how an application server 
behaves.  If anything, the application should be able to configure DNS.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Thu, 22 May 2014 09:11:33 -0700

On 5/22/2014 8:05 AM, Ned Freed wrote:
>> One of the harder issues is application configuration. 
...
>> So is this a new idea?
> 
> For i18n? Maybe. For email?
...
> richer semantics: 
...
> The DNS just isn't flexible enough.
...
> I've never seen a writeup of the issues anywhere.

John's opening classes this as a generic problem.

Seems like what is needed is a careful casting of the kinds of
functional requirements people often make, here, and then a careful
explanation of the reasons the DNS is a poor venue for satisfying them.

So, help the read to "understand the problem" and then to understand how
the "DNS just isn't flexible enough".

"DNS Name-Mapping Considered Harmful?" (No, I'm don't think that's the
right title; just trying to vector the direction.)

For FUSSP, we have the benefit of Cory Doctorow's squib.  Given it's
form, it might be odd that it's useful, but still, it helps.

Sounds like we need a similar place to point to, for mis-applications of
the DNS.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: "John R Levine" <[email-masked]>
Date: 22 May 2014 12:33:36 -0400

> behaves.  If anything, the application should be able to configure DNS.

Hmmn.  Tell us about http virtual domains.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

---

From: Ned Freed <[email-masked]>
Date: Thu, 22 May 2014 09:19:08 -0700 (PDT)

> explanation of the reasons the DNS is a poor venue for satisfying them.

I provided a list in my original response that demonstrates that a simple name
mapping is in no way sufficient. So if you're going to do this with the DNS,
you're talking about defining some new, and very complex, records. Which AFAIK
isn't anything anyone wants to do.

FWIW, we typically store domain information in LDAP along with user entries.
The records we use are shared by a bunch of different applications besides
email. I'd venture that there are >50 defined attributes in various object
classes, with ~25 of them having some relevance to email. And setups with many
(as in tens of thousands) of domains are not uncommon.

But really, Keith's answer is better. Applications should be provisioning the
DNS, not the other way around. In practice mail systems deal with large amounts
of configuration and account information. Are we going to stuff all of that in
the DNS? And if not, why complicate applications with having to get their
configuration from an additional place?

> right title; just trying to vector the direction.)

Am I missing something? Is there a serious proposal being worked on to try and
specify this nonsense? If not, why should we bother?

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 22 May 2014 12:46:04 -0400

> allow single-labels domains.

Your MUA seems to have smashed my UTF-8.  That was fXX.com where the X's 
were o's with accents, really xn--f-vgaa.com so it was

xn--f-vgaa.com CNAME foo.com

> cannot have both a CNAME and a delegation record.

Let's say we wave our hands here and say it was a DNAME at a higher level, 
like the one in Taiwan, or maybe someone does BNAME or CLONE.

Also, with respect to the sendmail thing, that was back before there was 
much of a line between submission and relay, and I believe that was 
intended as a submission fixup to implement the old rule that CNAMEs have 
to be resolved before you send the mail.

> Am I missing something? Is there a serious proposal being worked on to 
> try and specify this nonsense? If not, why should we bother?

No, I just wanted to be sure I hadn't missed anything interesting.

Regards,

---

From: Dave Crocker <[email-masked]>
Date: Thu, 22 May 2014 09:46:38 -0700

> specify this nonsense? If not, why should we bother?

As John noted, these sorts of issues keep coming up.  My thought is that
it could be helpful to formulate a document that deals with the more
common problematic suggestions and then be able to just point to it.

d/

---

From: Keith Moore <[email-masked]>
Date: Thu, 22 May 2014 12:46:46 -0400

On 05/22/2014 12:33 PM, John R Levine wrote:
>> No, that's broken.   DNS should never dictate how an application 
>> server behaves.  If anything, the application should be able to 
>> configure DNS.
>

What is there to tell?    The HTTP server has to be explicitly 
configured to recognize the virtual domain anyway, otherwise it has no 
idea where to find the content that goes with that domain. That's how it 
is with most applications - the application server needs to know more 
than could reasonably be included in the DNS anyway.   Add to that that 
it's pretty common for servers to handle multiple DNS domains - so how 
is the server to know which DNS domains it should trust?  Trusting every 
DNS domain that references the server makes no sense because anybody can 
reference your server in their DNS domain.   So at a minimum the 
application server needs to know which DNS domains to trust.  Finally, 
there's the problem that even for "legitimate" DNS references to your 
server, the people who run the DNS servers are often out of touch with 
reality.

Executive summary:  DNS should never dictate how an application server 
behaves.  If anything, the application should be able to configure DNS.

Keith

---

From: "John R Levine" <[email-masked]>
Date: 22 May 2014 12:50:21 -0400

> so how is the server to know which DNS domains it should trust? 
> Trusting every DNS domain that references the server makes no sense 
> because anybody can reference your server in their DNS domain.  So at a 
> minimum the application server needs to know which DNS domains to trust.

Yeah, that's one of the reasons I think it's a bad idea.  If we had 
something like CLONE that flipped the direction of the pointer so the 
canonical name controled what its aliases were, it might be a slightly 
less bad idea.

Regards,

---

From: Keith Moore <[email-masked]>
Date: Thu, 22 May 2014 12:56:08 -0400

On 05/22/2014 12:50 PM, John R Levine wrote:
>> because anybody can reference your server in their DNS domain.  So at 
>> a minimum the application server needs to know which DNS domains to 
>> trust.
>

Having DNS dictate the behavior of applications is one of those 
perpetually recurring Bad Ideas that needs to be soundly defeated. I 
wish I could understand why anyone thinks it makes sense.

(Maybe there's some psychology in effect here - the DNS records are 
simple and publicly visible, the application configuration is rarely 
simple and not publicly visible, so whenever people see that the two are 
out-of-sync, people somehow assume that the DNS is right?   As far as I 
have been able to tell, it's actually slightly more probable that it's 
the DNS that is wrong - again, because the DNS administrators tend to be 
at some distance from the people who actually run the applications.)

I do think it makes sense for applications to be able to check DNS to 
make sure that the DNS configuration matches the application 
configuration.   Even better would be to let the application update DNS 
- but that would require DNS servers to have fine-grained authentication 
and ACLs.

Keith

---

From: John C Klensin <[email-masked]>
Date: Thu, 22 May 2014 13:07:44 -0400

--On Thursday, 22 May, 2014 12:46 -0400 John R Levine
<[email-masked]> wrote:

>...
> Your MUA seems to have smashed my UTF-8.  That was fXX.com
> where the X's were o's with accents, really xn--f-vgaa.com so
> it was
> 

While those lines were definitely smashed, I receive a lot of
UTF-8 traffic and that traffic doesn't get smashed in either my
MTA or MUA.   In particular, I receive messages with Chinese
coded in UTF-8 on an almost daily basis.   It would be very
interesting to track down where the problem is occurring, but I
don't have time this week.

regards from (because it is slightly relevant) Beijing,
     john

---

From: Ned Freed <[email-masked]>
Date: Thu, 22 May 2014 11:13:27 -0700 (PDT)

> at some distance from the people who actually run the applications.)

This, if anything, understates the problem. It's actually pretty common
for the DNS admins to be in a different part of the org and for coordination
processes to be overly onerous, usually due to political turf-grabbing.

You'd think that such ring-tailed idiocy would not be tolerated when there
are lots of domains involved, but it often is.

You also run into other types of name-related lunacy like, "Due to the
tremendous and ongoing shortage of alphanumeric labels, you must place all your
host names in the same namespace as your external domains." Or, "I read the
Earthsea trilogy while stoned and therefore believe that knowledge of a name
confers power over what it names, so none of these names can be exposed in
message header."

> configuration.

Yep. Sites with lots of domains do this routinely.

> Even better would be to let the application update DNS

In my experence it's sometimes done by generating sections of zone files from
the directory or whatever, which then get fed through some sort of automated
update process.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Fri, 23 May 2014 12:20:39 +0100

John C Klensin <[email-masked]> wrote:
>
> The example, as written, is simply invalid.  SMTP does not allow
> single-labels domains.  They are explicitly prohibited in 5321 and its
> predecessors.

That is incorrect. The syntax in RFC 5321 is

   Domain         = sub-domain *("." sub-domain)

Whereas in RFC 2821 it is

      Domain = (sub-domain 1*("." sub-domain)) / address-literal

And in RFC 821 it is

            <domain> ::=  <element> | <element> "." <domain>

It was generally agreed that the 1* in RFC 2821 was an editorial mistake
which was fixed in RFC 5321.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Irish Sea: Northeast veering east later, 5 to 7, occasionally gale 8. Slight
or moderate, occasionally rough. Rain or showers. Good, occasionally poor.

---

From: Tony Finch <[email-masked]>
Date: Fri, 23 May 2014 12:21:39 +0100

John R Levine <[email-masked]> wrote:
>
> Your MUA seems to have smashed my UTF-8.

Your message was smashed before it got to me.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
West Bailey: Variable 4. Slight or moderate. Showers. Moderate or good.

---

From: Tony Finch <[email-masked]>
Date: Fri, 23 May 2014 12:31:19 +0100

S Moonesamy <[email-masked]> wrote:
> > If memory servers, ancient versions of sendmail supported this and
> > actually rewrote such aliases.
>

I was looking at this yesterday. Sendmail has a DontExpandCnames option
(off by default so it does CNAME-based rewriting by default) but as I
understand it this controls the routing of outgoing mail and does not
affect which domains Sendmail will consider to be local and which it will
accept in non-relay RCPT commands.

The key thing about John's suggestion is that an MX MTA would accept mail
to domains for which it does not have any explicit configuration but which
have a CNAME pointing to an explicitly configured domain.

(Aside: The DontExpandCnames option has amusingly backwards documentation
which seems to date from the late 1990s: "This currently violates the
published standards, but the IETF seems to be moving toward legalizing
this.")

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Plymouth: Cyclonic, mainly west or northwest 5 or 6. Moderate. Rain or
thundery showers. Good, occasionally poor.

---

From: Tony Finch <[email-masked]>
Date: Fri, 23 May 2014 12:46:55 +0100

Keith Moore <[email-masked]> wrote:
>
> Even better would be to let the application update DNS - but that would
> require DNS servers to have fine-grained authentication and ACLs.

DNS servers do have fine-grained authentication and ACLs. (Though that
doesn't really solve the provisioning problem: it just moves it from
provisioning application records directly, to provisioning application
ACLs.) The problem is usually that the DNS is populated from some
configuration database which does not have such flexible access control.

http://ftp.isc.org/isc/bind9/9.10.0-P1/doc/arm/Bv9ARM.ch06.html#dynamic_update_policies

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Southwest German Bight, Humber, Thames: Variable 4, becoming cyclonic, mainly
southeasterly 5 to 7 later. Slight or moderate. Thundery showers. Good,
occasionally poor.

---

From: Tony Finch <[email-masked]>
Date: Fri, 23 May 2014 13:15:40 +0100

Keith Moore <[email-masked]> wrote:

> The HTTP server has to be explicitly configured to recognize the virtual
> domain anyway, otherwise it has no idea where to find the content that
> goes with that domain.

It doesn't need to be explicit. Here's something I wrote 15 years ago...

http://httpd.apache.org/docs/1.3/vhosts/mass.html

> so how is the server to know which DNS domains it should trust?

The domains that the server doesn't know about will not work because the
content isn't where the server expects :-)

I am less negative about John's CNAME idea than Keith or Ned. Yes of
course as Ned said it cannot possibly cope with all the wild and wooly
mail configurations that people want, but it is sufficient for simple
cases. My servers have some simple domain-level aliases which could be
supported just fine by this feature.

A significant caveat is that some senders (such as Sendmail with
DontExpandCnames off) will take it upon themselves to rewrite based
on the CNAME and others will not, and you can't control this.

You could in principle do something similar in a web server, e.g. have a
catch-all virtual host which sends responses that do CNAME-based domain
redirects. (That should keep the DNS lookup away from latency-sensitive
paths.)

The key question is whether this kind of dumb DNS-based provisioning is
useful enough to satisfy people who want easier IDN setup, or whether
their requirements are diverse enough that there's no point trying.
The people who might be interested in this kind of thing are the big
hosting providers, but they already have full-fat provisioning systems
which ought to be able to cope with IDN aliases, so I expect they would
see no benefit from using the DNS in this way.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Trafalgar: North or northwest 4 or 5, increasing 6 at times. Rough in north,
otherwise moderate. Showers. Good.

---

From: Ned Freed <[email-masked]>
Date: Fri, 23 May 2014 06:46:06 -0700 (PDT)

> published standards, but the IETF seems to be moving toward legalizing

Well, I stand corrected. Here I thought this nonsense had faded into
nonuse, and apparently it's still the default. Sigh.

				Ned

---

From: S Moonesamy <[email-masked]>
Date: Fri, 23 May 2014 07:18:43 -0700

Hi Tony,

Yes.

>accept in non-relay RCPT commands.

The problem I ran across is that I was using a CNAME and there was a 
rewrite of the address in the header.  I did not want that behavior; 
the reason is vanity.  The domains to be considered as local was not 
a problem as I could easily configure that.

Yes.

>published standards, but the IETF seems to be moving toward legalizing

The comment has been there since a very long time. :-)

Regards,
S. Moonesamy

---

From: S Moonesamy <[email-masked]>
Date: Fri, 23 May 2014 06:36:42 -0700

Hi Tony,

Yes.

That's the dotless [1] discussion. :-)

Regards,
S. Moonesamy

1. http://tools.ietf.org/html/draft-moonesamy-dotless-domains-00

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Fri, 23 May 2014 16:12:12 +0100

> accept in non-relay RCPT commands.

From memory, ‘DontExpandCnames’ simply tells Sendmail to first perform CNAME expansion in the SMTP client.  Canonicalisation of the form described in [1] is done inside the rewrite rules, by using the results of $[ … $] expansions, and is controlled (if using the mc files to build your configurations) with FEATURE(`nocanonify').  Finally the list of acceptable domains is controlled by a class and has nothing to do with client rewrites; we suppose that the destination of CNAMEs will be listed to accept mail from obsolete Sendmail, but it should also include every other DNS name by which the system is known locally.  Furthermore there are various hardcoded corner cases, which support your theory that it’s obsolete, EG it still wants to canonify single-label host names, and it leaves the DNS resolver unconfigured unless you specifically set options (EG to suppress the dotless search).

So to summarise: I think sendmail users should all be using FEATURE(`nocanonify’) and should have their ‘w’ class correctly listing all of the names that they are known by, and they should have “DontExpandCnames=F” (off), and have their resolver set up not to search.

Or you could just upgrade to Postfix. :)

> published standards, but the IETF seems to be moving toward legalizing

Exactly, that described in [1].

Cheers,
Sabahattin

[1] http://cr.yp.to/im/cname.html

---

From: Tony Finch <[email-masked]>
Date: Fri, 23 May 2014 17:22:31 +0100

Sabahattin Gucukoglu <[email-masked]> wrote:

Thanks for the extra Sendmail knowledge.

> http://cr.yp.to/im/cname.html

Don't believe everything on that page. Sendmail does not do ANY queries,
and qmail's canonicalization code is bad in several ways.
http://fanf.livejournal.com/122220.html

He says: "If you set up www.your.site with a CNAME for your.site, mail to
www.your.site will automatically be accepted by your.site's mailer. Many
sites rely on this feature."

My mailers have never (well, going back to 1997 at least) done CNAME
canonicalization and this hasn't caused the interop problems you might
expect from many sites relying on this feature.

His comment about a 2001 target for deprecating the feature is almost as
amusing as the Sendmail documentation...

Tony.

---

From: "John R Levine" <[email-masked]>
Date: 23 May 2014 12:26:06 -0400

> http://fanf.livejournal.com/122220.html

qmail used to do ANY queries, but everyone I know who uses qmail has 
undone that particular hack

> sites rely on this feature."

I wonder if that still works.  Probably not, since most of us have 
replaced the SMTP daemon with something else.

Regards,

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Fri, 23 May 2014 17:56:04 +0100

> Thanks for the extra Sendmail knowledge.

I was curious to see how my memory stacked up so I just grabbed the sources for 8.14.4.  The option is still unset by default, and the generated cf files are still using canonicalisation.  This is all very silly. :(

The magic is all happening inside the DNS map, so all you really have to do to turn this behaviour off is use FEATURE(`nocanonify’), which stops map lookups.  I hope that every Sendmail site does this.  As a bonus your performance will shoot through the roof, because you are turning off many DNS queries, even those that do not result in CNAME expansion.  The check is being done for every RCPT command so if you want to send mailing list mail this is definitely advised.

> expect from many sites relying on this feature.

It didn’t make sense to me when I first learned about this particular “Feature” of Sendmail, when I was first setting it up in 2004.  Thankfully, before and since, it still doesn’t.  I well imagine that it’s just another piece of prehistory that has got muddied over the years, but for my part I reject the notion of application awareness of DNS, in this and other cases.  CNAMEs are just aliases and have no significance; in 2001 RFC 2821 was clear that this was the case for SMTP and I configured my Sendmail with DontExpandCnames=True.

> amusing as the Sendmail documentation…

:)

Cheers,
Sabahattin

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 23 May 2014 19:55:28 +0200

On Thu 22/May/2014 19:07:44 +0200 John C Klensin wrote: > --On
> don't have time this week.

Assuming both Johns set their MUAs correctly, it must be the list
receiver.  What looks strange is that the message has two MIME
entities, both text/plain, one with charset=utf-8 and the other with
charset="us-ascii" for the footer.  However, fóó.com was encoded in
latin1 (66 f3 f3 2e 63 6f 6d).  DKIM verified, so that must have
been the outgoing format.  (你可以阅读本)

Looking at the source, I found out there is a new header field:

Archived-At:
http://mailarchive.ietf.org/arch/msg/ietf-smtp/VSm3BgQxO8fA2C0sbTqsZFeJonc
Here, fóó.com is encoded in utf-8 (66 c3  b3 c3 b3 2e 63 6f 6d)
as advertised correctly in both Apache's header and the file's
meta http-equiv="Content-type".

Ale

---

From: "John Levine" <[email-masked]>
Date: 23 May 2014 18:00:32 -0000

>>> where the X's were o's with accents, really xn--f-vgaa.com so

No, it was a shell script here that didn't prime my editor to store
stuff in UTF-8.  "Never mind".

R's,
John