Subject: Re: [ietf-smtp] Address transformations

From: Tony Finch <[email-masked]>
Date: Mon, 1 Aug 2016 13:55:03 +0100

John R Levine <[email-masked]> wrote:
>
> So my reading is that while 5321 tells me that it's OK to use a CNAME in MAIL
> FROM/RCPT TO, 1123 still tells me that it's better to resolve those CNAMEs
> first.
>
> I hope that nobody who matters believes this, but if you look at the two
> documents it's not a totally ridiculous way to reconcile them.

Sendmail still does RFC 1123 mail domain canonicalization unless you use
FEATURE(`nocanonify`)

http://www.sendmail.com/sm/open_source/docs/configuration_readme/

Qmail does it unconditionally (and uses DNS ANY queries in the process).

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Shannon: Cyclonic becoming southwesterly 5 or 6, occasionally 7 in south,
decreasing 4 at times later. Moderate or rough. Rain, then drizzle later, fog
patches. Moderate or poor, occasionally very poor.

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Aug 2016 09:08:40 -0400

--On Sunday, July 31, 2016 19:25 -0700 Sean Leonard
<[email-masked]> wrote:

> (RFC 5321)
> All this stuff brings up a very interesting point about
> deliverable email addresses in draft-seantek-mail-regexen:
> 
> What are the domain part limitations on deliverable email
> addresses, that are not encapsulated by the RFC 5321 ABNF?
>...
> The string "1.2.3.4" is a valid domain production.
> However, it is not (or *should not*) be considered a
> deliverable email address, because when passed to the famous
> function "gethostbyname", that function will certainly
> return 1.2.3.4; it will not perform a lookup of the domain
> record with a top-level label of "4". 
>...

The "fame" of gethostbyname doesn't make it a standard or any
type.  In fact, it has been widely suggested on IETF and other
lists that it is broken.

>... 
> I have tried "[email-masked]" -- Windows and Unix/Linux
> stacks will try to query the DNS for that string and not parse
> it as IPv4. So, it is syntactically a valid, deliverable email
> address.

It is syntactically a valid email address.  It is unlikely to be
deliverable.  And what Windows and Unix/Linux do is not
sufficient to justify "So".

> But foo@411 will get IPv4-parsed to 0.0.1.155, which
> means that "411" should not be considered a valid domain
> name for a deliverable email address.

Actually, if foo@411 is treated as containing an address
literal, rather than a domain name, that is an error
(independent of what were done with the address literal once
that determination was made.  It is, of course, not a valid
domain name either, and that is independent of whether it is
invalid because it is all digits, because it is numeric and at
the root, or simply because it is not a label that appears in
the root zone.  Of course, it could also be a local name that
should really be interpreted as [email-masked],
[email-masked], etc., depending on local configuration
setup in, e.g., the submission server.

>...

More generally, it seems to me that, if you take the questions
up a level, the either the answers to your several questions
become either clear or the questions are trivial.

First, my recollection is that there was a strong belief when
RFC 821 was written that the distinction between "address
literal" and "domain" in the domain part was to be made by
obvious, no-lookahead, syntax, not by either heuristics or "is
this a number" tests.  So, as far as SMTP is concerned,
foo@[1.2.3.4] is a mailbox containing an address literal while
[email-masked] is a mailbox containing a domain.  I/we carried the
view forward when we had to deal with IPv6 addresses.  That led
to the decision to explicitly designate the address family
associated with an address literal rather than leaving it to
heuristics on, e.g., what syntax is used in the address.
Whether something that SMTP identifies lexically as an address
literal is valid or not (and whether a particular SMTP client or
server knows what to do with it) is largely not an SMTP problem.
Similarly, whether something that SMTP lexically identifies as a
domain is valid (or can ever be valid) is another question and
not really an SMTP issue except that...

	(i)  There is a long history of SMTP (and other email)
	clients (including submission servers) trying to do
	sanity checks on putative domain names.  Some of that
	history is rooted in systems that were not directly
	connected to the Internet, and hence unable to do DNS
	lookups themselves, wanting to eliminate (reject,
	bounce, or, in the case of submission servers and
	gateways, fix) messages with bogus mailbox addresses as
	early in the transmission processes as possible.  At one
	time, it was very common for SMTP clients (and even some
	MUAs) to have lists of valid TLD labels or valid TLD
	labels of other than two characters in length and/or for
	them to "know" that there was only one four-letter TLD
	label and no TLD label longer than four characters long.
	Some of those tests became much more fragile when ICANN
	started introducing gTLDs at a relatively high rate
	including many with labels other than three characters
	in length and were further complicated by IDN TLDs and
	are usually considered poor practices, especially for
	hosts directly connected to the Internet, today.
	
	(ii) RFC 1123 (in a section not affected by the
	"replacing the mail transport..." comment in 5321) says
	that TLD names must be alphabetic.  At least so far and
	with the exception for the form of IDNA labels, ICANN
	has not violated that rule.  So, 1.2.3.123 is still
	clearly invalid as a domain name.

In both of the above cases, the issues are DNS ones, or ones in
the interface between an SMTP system and the DNS, not SMTP
issues.

Second, coming back to address literals, if SMTP identifies
something as an address literal (i.e., the string "@[" appears)
then, while syntax is given in 5321 for IPv4 address literals,
all of the others (present and future) are ultimately 
   Tag ":" String

For IPv6, the Tag is "IPv6" and the string specified by
standards for presentation forms of IPv6 addresses, not SMTP.
For anything else, the Tag is required to be standardized and
that standard or the related one is expected to specify the
syntax of the String (but there are few character restrictions
on it).    It was, and is, quite intentional that, if some set
of systems, by prior agreement, invented the "foozy" address
format and associated protocols but did not standardize it
through the IETF, 

    usermailbox@[Foozy:666]
could work perfectly well within their collective environment
without causing interoperability problems with
standards-conforming systems.

Finally, while the email specs were quite careful to make a
lexical distinction between a domain and an address literal in a
mailbox name, USLs/URIs did not follow that lead and appear to
rely on heuristics or a subtle understanding of the strings
involved instead.  That is one of many cases in which an
identifier that looks more or less like an email address may not
actually be an email address and certainly doesn't imply
constraints on email addresses or email systems.  If you want to
discuss such identifiers, this is almost certainly not the right
list.

best,
    john

---

From: "John R Levine" <[email-masked]>
Date: 1 Aug 2016 11:34:54 -0400

> Qmail does it unconditionally (and uses DNS ANY queries in the process).

Qmail's been abandonware since 1998.  Everyone who still uses it has 
patched it extensively, and we got rid of the ANY query ages ago.  I don't 
think it ever rewrote CNAMEs in message headers.  I don't see code to do 
it in either qmail-inject or its slightly newer replacement new-inject.

If I were doing an update to 5321 my non-negotiable preconditions would be 
that changes are only allowed to describe existing practice in non-trivial 
amounts of mail, that it obsoletes every previous definitions of SMTP, and 
that it's published as a full standard.  Once we spend a year agreeing to 
that, the rest is easy.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

---

From: Sean Leonard <[email-masked]>
Date: Mon, 1 Aug 2016 15:11:02 -0700

> On Aug 1, 2016, at 6:08 AM, John C Klensin <[email-masked]> wrote:
> 

Actually, getaddrinfo (the successor to gethostbyname) *is* in the POSIX standard: IEEE Std 1003.1, 2013 Edition. <http://pubs.opengroup.org/onlinepubs/9699919799/> And it does specifically call out support for IPv4 and IPv6 address literals.

That being said, I do not wish to devolve into a standards-baiting discussion about that API call. I am interested however how mail systems actually implement the parsing of the domain production. If you give any number of popular, widely implemented SMTP servers RCPT TO:<[email-masked]>, will it query DNS for “11.22.33.44”, or will it attempt to contact the server at the IPv4 address 11.22.33.44 ? Implementers can answer this question...

> setup in, e.g., the submission server.

Ok. However, Section 2.3.5 of RFC 5321 says:
   The domain name, as described in this document and in RFC 1035 [2],
   is the entire, fully-qualified name (often referred to as an "FQDN").
   A domain name that is not in FQDN form is no more than a local alias.
   Local aliases MUST NOT appear in any SMTP transaction.

The ABNF of RFC 5321 shows that you cannot have a single bare label
with a trailing dot. It seems to me that a single bare label
indicates that the item has to be a FQDN, and never a local alias;
therefore, it must not be all-numeric.

> 	clearly invalid as a domain name.

Ok, I found the text: Section 2.1 of RFC 1123. Thanks.

This can be implemented by checking that the final sub-domain is not
all-numeric:

((?&sub_domain)\.)+(?![0-9]+[^\-A-Za-z])(?&sub_domain)

This will permit the following:

[email-masked]

which getaddrinfo will consume as an IPv4 address literal.
However, as long as we at the IETF are okay with that (or don’t care),
then we are all good.

> the interface between an SMTP system and the DNS, not SMTP

Yes. The patterns for *deliverable* email addresses
are intended to incorporate DNS rules. Generic email addresses
(RFC 5322) do not need to be deliverable; therefore, they
do not necessarily need to incorporate DNS rules.

There is probably a way to express the “last sub-domain cannot be all-numeric” in ABNF,
and therefore, to translate that to an integrated regular expression (as opposed to a regular expression with a negative-lookahead). I am not sure if it would be more efficient.

I think it would be:

Domain = *(sub-domain ".") final-sub-domain

sub-domain     = Let-dig [Ldh-str]

AH = ALPHA / "-"

final-sub-domain = ALPHA [Ldh-str] /
             DIGIT *(ALPHA / DIGIT / "-") ALPHA /
             DIGIT 1*(*DIGIT AH *DIGIT) DIGIT

> standards-conforming systems.

I want to be sure about this. If someone enters an email address of

user@[23.34.45.56]

into an input field (e.g., web browser or mail client) that *purports to accept all deliverable email addresses*, shall it be accepted as valid input, or not?

Section 2.3.4 of RFC 5321 says:
   Hosts are known by names
   (see the next section); they SHOULD NOT be identified by numerical
   addresses, i.e., by address literals as described in Section 4.1.2.

Well that is is a SHOULD NOT, not a MUST NOT. I just want to be sure that we are collectively sure that an input field that *purports to accept all deliverable email addresses*, *needs* to accept address-literal syntax.

If the answer is yes, it needs to accept address-literal syntax, shall the input field accept General-address-literal? Shall it accept IPv6-address-literal (which, by the way, will always get matched as General-address-literal, and therefore, is superfluous if we permit General-address-literal)?

My current position is that if address-literal syntax is needed, it should only accept IPv4 and IPv6 syntaxes; general address literals are not “deliverable” using the modern Internet & SMTP infrastructure. If a new address literal becomes globally routable on the public Internet and is standardized by the IETF, then the regular expression should be updated (along with billions of instances of email and Internet software). I do not have a position on whether address-literal syntax itself is needed, but I am a little skeptical that such an email address fits a working engineering definition of “deliverable” using the modern Internet & SMTP infrastructure.

> discuss such identifiers, this is almost certainly not the right

Yep, not talking about URIs here.

Regards,

Sean

---

From: Michael Peddemors <[email-masked]>
Date: Mon, 1 Aug 2016 15:46:26 -0700

> That being said, I do not wish to devolve into a standards-baiting discussion about that API call. I am interested however how mail systems actually implement the parsing of the domain production.…

And what I would more like to know, is there ANY use cases for still 
having foo@IPv4address any more?

That was for a time so long forgotten wasn't it?

I know we stopped accepting it, since only the spammers used it any more.

This would be a great time, if no one has a legitimate use case for 
that, to maybe deprecate the concept of 'user' emailing 'user' at a 
physical IP location, and start considering that SMTP should be 
standardized for a different use case.

If someone wants to update the 'draft' to supercede the original RFC, 
which was for a time so long ago... maybe it is time for a 'modern' 
definition of the SMTP protocol that no longer allows for that.

While backwards compatibility would make sense if there was anything 
left that uses it, if in practice it is never used, should it be not put 
to end-of-life at some point?

-- 
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
------------------------------------------------------------------------
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

---

From: "John Levine" <[email-masked]>
Date: 1 Aug 2016 22:46:29 -0000

>mail systems actually implement the parsing of the domain production. If you give any number of popular, widely implemented
>SMTP servers RCPT TO:<[email-masked]>, will it query DNS for “11.22.33.44”, or will it attempt to contact the server at
>the IPv4 address 11.22.33.44 ? Implementers can answer this question...

Given that the chances of either being a useful address rounds to
zero, why is this question worth answering?  RFC 5321 is quite clear
that this is not an address literal, since it's not in brackets, and
there will never be an all-numeric TLD in the public Internet.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 1 Aug 2016 23:07:02 -0000

>having foo@IPv4address any more?

postmaster@[127.0.0.1] might be useful for panic reports on your server
but that's about it.

I agree that if we deprecated it in the future, that's no great loss.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Aug 2016 19:51:34 -0400

--On Monday, August 01, 2016 15:11 -0700 Sean Leonard
<[email-masked]> wrote:

>...
> The ABNF of RFC 5321 shows that you cannot have a single bare
> label with a trailing dot. It seems to me that a single bare
> label indicates that the item has to be a FQDN, and never a
> local alias; therefore, it must not be all-numeric.

Sean,

There is a philosophical question as to whether authoritative
limitations on DNS labels in various positions should be
reflected in application standards that happen to use or
reference DNS names.  The argument for doing so is that it makes
life easier for implementers and makes it easier for those
implementers to design implementations that catch nonsense
early.   The argument against it is that the restrictions that
might be thereby imposed might later be relaxed elsewhere and
that could cause a mess.  IIR, DRUMS rather explicitly decided
on the latter model.  In retrospect, it is a good thing that it
did.

Two examples:

> This can be implemented by checking that the final sub-domain
> is not all-numeric:
> 

That example is actually another reason why there is reluctance
to incorporate DNS label allocation policy rules into, e.g.,
SMTP.  What you write above doesn't reflect the rule in 1123.
That rule says "will be alphabetic" and meant it [1], so, as a
TLD label, "0xe" is invalid.  So, and this is important, is
"exam-ple".

Now, suppose someone had chosen to incorporate your regular
expression into an application.  No problem wrt, e.g., "oxe"
because, wrt alphabetics, your rule is simply too permissive
because it allows numerics.  

However, a few years ago (long after 1123 and, for that matter,
5321), ICANN decided to start allowing IDNs at the top level.
Their initial rule was that only letters would be allowed in TLD
labels (in U-label form), which is true to the spirit of that
rule from 1123.  However, what goes into the DNS isn't U-labels
but A-labels.  A typical IDNA A-label contains two consecutive
hyphens (and usually more or them) and one or more digits, so,
while the conceptual rule is retained, any regular expressions
that prohibit either hyphens or digits (as a strict reading of
1123 would do) would end up rejecting perfectly valid domain
names because the TLD label did not match an obsolete
restriction (one over which, at least in the case of IDNs and
TLD label lengths, IETF demonstrably has little control).

> into an input field (e.g., web browser or mail client) that
> *purports to accept all deliverable email addresses*, shall it
> be accepted as valid input, or not?
> 
>    (see the next section); they SHOULD NOT be identified by
> numerical    addresses, i.e., by address literals as described
> in Section 4.1.2.
> 
> Well that is is a SHOULD NOT, not a MUST NOT. I just want to
> be sure that we are collectively sure that an input field that
> *purports to accept all deliverable email addresses*, *needs*
> to accept address-literal syntax.

What you have been hearing, and what the text you refer to in
5321bis was trying to say, is that the consensus of the email
community is that address literals have served their purpose and
become more of a problem, and more risk, than they are worth
_but_ SMTP implementations should still support them (and
probably allow configuration to reject them).  But SMTP is not
about either mail clients (i.e., MUAs) or web broswers -- it is
about the transport of email.  If you are looking for a Grand
Unified Theory of email addresses, you need to look elsewhere.

>...
> My current position is that if address-literal syntax is
> needed, it should only accept IPv4 and IPv6 syntaxes; general
> address literals are not "deliverable" using the modern
> Internet & SMTP infrastructure. 

The generalization was never intended to apply to the
contemporary Internet and what I assume you mean by "SMTP
infrastructure".  It was particularly intended to lay down a
syntax template for whatever comes after IPv6  and, to a lesser
extent, to point the way to a way of doing things on private
networks that used different protocols without causing problems
if (when) the inevitable "leak" occurred.  

     john