Subject: [ietf-smtp] ietf.org SMTP server "RFC2821 Violation" for EHLO ip-literal.

From: Hector Santos <[email-masked]>
Date: Sun, 01 Dec 2019 14:12:13 -0500

Can anyone confirm why the following session would be an "RFC2821 
Violation" as issued by the ietf.org SMTP server?

220 ietfa.amsl.com ESMTP
EHLO [A.B.C.D]
250-ietfa.amsl.com
250-PIPELINING
250-SIZE 67108864
250-ETRN
250-STARTTLS
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250 8BITMIME
MAIL FROM:<>
250 2.1.0 Ok
RCPT TO:<[email-masked]>
550 5.7.1 <[A.B.C.D]>: Helo command rejected: RFC2821 violation
QUIT
221 2.0.0 Bye

The client connection IP is A.B.C.D. It matches the EHLO [A.B.C.D] 
ip-literal input.  PTR records exist for the IP A.B.C.D.  In this mail 
app, using the IP literal is preferred.

So what RFC2821 violation could it be complaining about?

Both RFC 2821/5321 clearly indicate an verification fail MUST NOT be a 
reason for rejection. Yet as we know, the PTR requirement has been 
increasingly enforced at various sites.

Since the server used RFC2821 as a reference, I see these two 
paragraphs in conflict:

    RFC2821 Section 4.1.4 Order of Commands

    The SMTP client MUST, if possible, ensure that the domain parameter
    to the EHLO command is a valid principal host name (not a CNAME or MX
    name) for its host.  If this is not possible (e.g., when the client's
    address is dynamically assigned and the client does not have an
    obvious name), an address literal SHOULD be substituted for the
    domain name and supplemental information provided that will assist in
    identifying the client.

    An SMTP server MAY verify that the domain name parameter in the EHLO
    command actually corresponds to the IP address of the client.
    However, the server MUST NOT refuse to accept a message for this
    reason if the verification fails: the information about verification
    failure is for logging and tracing only.

While RFC 5321 has revised the two paragraphs, the protocol semantics, 
I believe, remain the same.

In the 1st paragraph, there is a MUST for using a host name, if 
possible. This means if one or more PTR records exist for the 
connecting IP, one of host records returned MUST be used for the EHLO 
command.

In the 2nd paragraph, there is a MUST NOT refuse for verification 
fails.  This has been the standard expectation for SMTP servers for 
many years.

Obviously, this would be a candidate for a RFC5321bis revision where 
it would say "the server MAY refuse."   But there are a number of mail 
app scenarios where the IP literal is legitimately preferred even when 
PTR record(s) exist.

Comment??

-- 
HLS

---

From: John C Klensin <[email-masked]>
Date: Sun, 01 Dec 2019 14:56:55 -0500

Hector,

AFAICT, unless what you have written as "[A.B.C.D]" is not
actually a valid domain literal, it is broken and in more than
one way.  Section 2.3.5 of RFC 5321 seems quite specific that an
address literal in that syntax is allowed.  It says:

	o The domain name given in the EHLO command MUST be either
		a primary host name (a domain name that resolves to an
		address RR) or, if the host has no name, an address
		literal, as described in Section 4.1.3 and discussed
		further in the EHLO discussion of Section 4.1.4.

Similar text appears in RFC 2821, but there is no excuse for a
modern SMTP server to be reporting or rejecting (correctly or
not) on the basis of what 2821 says (or is imagined to say).  It
is disturbing that anything the IETF is running is referring to
2821 and not only because it may reinforce my impression,
mentioned in a different thread, that it is unlikely that anyone
will pay any more attention to a revision of 5321 than they did
to 5321 itself.

Two things are not clear from that text: (i) while the language
describing a domain name requires that the domain be resolvable
with the target RRSet containing an address RR, there does not
seem to be a requirement that whatever is at those address(es)
support an SMTP client or server.  (ii) Similarly, there does
not appear to be a requirement that, if an address literal is
used, that it refer to a system that can be accessed from the
public Internet or that an SMTP client and/or server be
supported there.   Given that EHLO and its arguments are sent
from an SMTP client, I don't see those additional potential
tests (much less clarification(s) that would require them) as
important or even helpful but, if anyone disagrees and we move
forward with 5321bis, please raise the issue then (while noting
that any stronger requirement than a SHOULD would impose a
restriction that was not there before).

If you have seen this as a problem in the wild, please write to
[email-masked], supply documentation the form of logs that
show the actual transaction, and ask that it be fixed.

best,
   john

--On Sunday, December 1, 2019 14:12 -0500 Hector Santos
<hsantos=[email-masked]> wrote:

> Can anyone confirm why the following session would be an
> "RFC2821 Violation" as issued by the ietf.org SMTP server?
> 
> 220 ietfa.amsl.com ESMTP
> EHLO [A.B.C.D]
> 250-ietfa.amsl.com
> 250-PIPELINING
> 250-SIZE 67108864
> 250-ETRN
> 250-STARTTLS
> 250-AUTH PLAIN LOGIN
> 250-AUTH=PLAIN LOGIN
> 250-ENHANCEDSTATUSCODES
> 250 8BITMIME
> MAIL FROM:<>
> 250 2.1.0 Ok
> The client connection IP is A.B.C.D. It matches the EHLO
> [A.B.C.D] ip-literal input.  PTR records exist for the IP
> A.B.C.D.  In this mail app, using the IP literal is preferred.
> 
> Both RFC 2821/5321 clearly indicate an verification fail MUST
> NOT be a reason for rejection. Yet as we know, the PTR
> requirement has been increasingly enforced at various sites.
> 
> While RFC 5321 has revised the two paragraphs, the protocol
> semantics, I believe, remain the same.
> 
> In the 1st paragraph, there is a MUST for using a host name,
> if possible. This means if one or more PTR records exist for
> the connecting IP, one of host records returned MUST be used
> for the EHLO command.
> 
> In the 2nd paragraph, there is a MUST NOT refuse for
> verification fails.  This has been the standard expectation
> for SMTP servers for many years.
> 
> Obviously, this would be a candidate for a RFC5321bis revision
> where it would say "the server MAY refuse."   But there are a
> number of mail app scenarios where the IP literal is
> legitimately preferred even when PTR record(s) exist.
> 
> Comment??

---

From: Keith Moore <[email-masked]>
Date: Sun, 1 Dec 2019 15:21:32 -0500

On 12/1/19 2:56 PM, John C Klensin wrote:

> support an SMTP client or server.

My take on this is that the SMTP server has an implementation bug, 
likely a failure to really upgrade to RFC2821 from RFC821. (I'm tempted 
to cite laziness but I must also admit that the RFCs are hard to read 
and the protocols have lots of corner cases that are rarely exercised - 
so I'm not surprised if implementors are impatient and cut corners.   
I've done that myself.)

Keith

---

From: John C Klensin <[email-masked]>
Date: Sun, 01 Dec 2019 15:41:39 -0500

--On Sunday, December 1, 2019 15:21 -0500 Keith Moore
<[email-masked]> wrote:

>> Two things are not clear from that text: (i) while the
>> language describing a domain name requires that the domain be
>> resolvable with the target RRSet containing an address RR,
>> there does not seem to be a requirement that whatever is at
>> those address(es) support an SMTP client or server.
> 
> My take on this is that the SMTP server has an implementation
> bug, likely a failure to really upgrade to RFC2821 from
> RFC821. (I'm tempted to cite laziness but I must also admit
> that the RFCs are hard to read and the protocols have lots of
> corner cases that are rarely exercised - so I'm not surprised
> if implementors are impatient and cut corners.   I've done
> that myself.)

Actually, that explanation doesn't quite work either.  I just
went back and looked at 821, and it rather clearly says 

    HELO <SP> <domain> <CRLF>

and then

	 <domain> ::=  <element> | <element> "." <domain>

	 <element> ::= <name> | "#" <number> | "[" <dotnum> "]"

So 
   HELO [1.2.3.4]

Would be perfectly valid syntax under 821.  So, of course, would
some combinations we'd find abhorrent today and that 2821/5321
prevent, like
   HELO [1.2.3.4.5.6]
or 
   HELO [1.2.3.4].example.com
(I haven't checked closely enough to see whether there are
prohibitions in the text for either of those, but they aren't in
the syntax.)

So, if all someone did in "upgrading" to RFC 2821 from 821 was
to treat "EHLO" as a synonym for "HELO", the example Hector gave
would still be invalid.  More important, it appears that they
actually made a serious effort to support EHLO because the
response is a multi-line one with options that an 821
implementation could not produce (and they are supporting
extended reply codes too).  So, unless what was actually sent in
the example was a clearly invalid IPv4 address, with something
like

   EHLO [500.600.700.800]
as an example candidate, I don't see any good basis for
rejecting on the basis of a 2821 (or 821 or 5321) restriction
about EHLO arguments.  Nor do I see any justification for an
IETF system reporting anything in terms of 2821 rather than
5321.  Unless we want to encourage others to disregard out
standards, the "eat one's own dogfood" principle should apply
here.  
 
best,
   john

---

From: Keith Moore <[email-masked]>
Date: Sun, 1 Dec 2019 17:10:20 -0500

On 12/1/19 3:41 PM, John C Klensin wrote:

> So
>     HELO [1.2.3.4]
>
> Would be perfectly valid syntax under 821.

Ah, good notice.   (I reread the relevant portions of 2821 but neglected 
to reread 821.)

So basically it's just an implementation bug.

Keith

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sun, 01 Dec 2019 17:21:19 -0500

> So basically it's just an implementation bug.

Unfortunately, it looks to me like a "brown M&M at a Van Halen concert"
sort of bug.

For those who aren't into the music scene, David Lee Roth explains:

https://www.youtube.com/watch?v=_IxqdAgNJck (about 5 minutes)

The tl;dr: Yes, it really *was* in their concert contract rider, and no, it wasn't
just rock stars being jerks.  And David Lee Roth is smarter than he looks. :)

---

From: Keith Moore <[email-masked]>
Date: Sun, 1 Dec 2019 17:33:29 -0500

> Unfortunately, it looks to me like a "brown M&M at a Van Halen concert"

In the sense that if the implementation couldn't get that detail right, 
they probably botched up a lot more than that?

Keith

p.s. I do agree that it's important that SMTP implementations permit IP 
addresses in HELO/EHLO.    An SMTP server is free to reject content 
based on server policy, but not to complain about syntax errors that 
don't actually exist.    In other news, I keep wanting to write an I-D 
(and see if I can get it published) entitled "DNS Is Not, And Cannot Be, 
Ubiquitous".

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sun, 01 Dec 2019 18:34:38 -0500

> they probably botched up a lot more than that?

Exactly. 

The relevant Received: lines indicate that ietf.org is using Postfix. The good
news is that the problem is *probably* just a bad regexp in one of the local
config files, as some googling doesn't find anybody else tripping over this,
which would be expected for something used as much as Postfix.

(Am placing bets that somebody was trying to prevent <127.0.0.1> literals
and forgot to allow it with brackets)

And something that will warm some hearts: http://www.postfix.org/postconf.5.html
makes numerous references to RFC5321 and RFC5322 :)

---

From: Hector Santos <[email-masked]>
Date: Sun, 01 Dec 2019 19:09:08 -0500

On 12/1/2019 3:41 PM, John C Klensin wrote:

> would still be invalid.

In the trace provided, the [A.B.C.D] was my winserver.com IP address. 
I should of just shown the actual I. It was legit.

821 did support ip literals and it was used.  Since 2003, my server 
did an client connection IP check against the bracketed ip-literal 
provided. The brackets is the trigger for a valid dotted IP address 
check rule.

> More important, it appears that they
> about EHLO arguments.

Right, it was legit.  I tried to just state the case generalizing the IP.

> Nor do I see any justification for an

The trace shows the server delayed the response until RCPT TO was 
provided.  This is most likely to collect all four envelope process 
parameters for logging and I will presume it was part of a SPF 
processor.   So what is this implementation at ietf.org doing?

Given:

response := MY_SPF_CHECK_HOST(CIP, CDN, RP, FP)

where the SMTP process (envelope) parameters are:

CIP  Client IP
CDN  Client Domain Name (HELO/EHLO host name)
RP   Return Path (MAIL FROM)
FP   Forwarding Path (RCPT TO)

The following is all possible:

Was it trying to do a SPF check on the CDN?  This is not always 
enabled since it considered unreliable and not always defined. As an 
exception, did it do a CDN check because the RP was NULL?  If so, 
before calling DNS to query the CDN SPF record, a syntax check is 
done.  It didn't like the bracketed IP address and the rule was 
enforced because PTR Records do exist for the CIP?

Who knows, but it could of read the 1st paragraph MUST use a host 
name, since one existed for the CIP, as the justification to ignore 
the 2nd paragraph "MUST NOT refuse" requirement.

Since 2003, there has been much in the area of trying to justify and 
legitimize the machine name and IP association in some protocol or 
filter, but the false positives are just still too high to enforce it.

-- 
HLS

---

From: John C Klensin <[email-masked]>
Date: Sun, 01 Dec 2019 19:57:39 -0500

Hector,

There are, IMO, two rather separate issues here.  One is whether
that specific IP address should have been rejected and how that
gets fixed.  That is a question you would need to take up with
the Secretariat.  The other is whether the error report is
correct and sensible.  It clearly is not.

Even if this was an SPF check or something equivalent, it would
justify neither the claim that the problem is with the EHLO
command nor the allegation that some 2821 restriction was not
met.    So, while we can speculate endlessly on what,
specifically, the bug is and how it came be be, the answer to
your original question is that this is fairly clearly a bug (or
two) and should be reported appropriately and fixed... and that
there is little or nothing that can be done  with 5321 (much
less 2821) or on this list that would contribute to getting that
done (unless, and this would surprise me, the Secretariat claims
the behavior is correct, in which case we could all try singing
"no, it isn't", in unison :-( ).

best,
   john

--On Sunday, December 1, 2019 19:09 -0500 Hector Santos
<hsantos=[email-masked]> wrote:

>> Would be perfectly valid syntax under 821.  So, of course,
>> would some combinations we'd find abhorrent today and that
>> 2821/5321 prevent, like
>>     HELO [1.2.3.4.5.6]
>> or
>> prohibitions in the text for either of those, but they aren't
>> in the syntax.)
>> 
>> to treat "EHLO" as a synonym for "HELO", the example Hector
>> gave would still be invalid.
> 
> In the trace provided, the [A.B.C.D] was my winserver.com IP
> address. I should of just shown the actual I. It was legit.
> 
> 821 did support ip literals and it was used.  Since 2003, my
> server did an client connection IP check against the bracketed
> ip-literal provided. The brackets is the trigger for a valid
> dotted IP address check rule.
> 
>> extended reply codes too).  So, unless what was actually sent
>> in the example was a clearly invalid IPv4 address, with
>> something like
>> 
>>     EHLO [500.600.700.800]
> The trace shows the server delayed the response until RCPT TO
> was provided.  This is most likely to collect all four
> envelope process parameters for logging and I will presume it
> was part of a SPF processor.   So what is this implementation
> at ietf.org doing?
> 
> Given:
> 
> Was it trying to do a SPF check on the CDN?  This is not
> always enabled since it considered unreliable and not always
> defined. As an exception, did it do a CDN check because the RP
> was NULL?  If so, before calling DNS to query the CDN SPF
> record, a syntax check is done.  It didn't like the bracketed
> IP address and the rule was enforced because PTR Records do
> exist for the CIP?
> 
> Who knows, but it could of read the 1st paragraph MUST use a
> host name, since one existed for the CIP, as the justification
> to ignore the 2nd paragraph "MUST NOT refuse" requirement.
> 
> Since 2003, there has been much in the area of trying to
> justify and legitimize the machine name and IP association in
> some protocol or filter, but the false positives are just
> still too high to enforce it.

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 14 Dec 2019 15:45:52 -0500

> On Dec 1, 2019, at 6:34 PM, Valdis Klētnieks <[email-masked]> wrote:
> 

What's been lost in this discussion, is that plainly the MTA is NOT reporting a
syntax error.  Yes the MTA is Postfix, and no Postfix does not reject address
literals as a built-in feature, that sort of access policy is something the
administrator would have to craft for himself.  The apparent syntax error is
just an illusion suffered by speakers of English.  A dog[1] would arrive at a
more correct interpretation:

  550 5.7.1 <[A.B.C.D]> blah blah Hello command rejected blah blah...

  [1] https://www.flickr.com/photos/sluggerotoole/153603564/

Syntax errors in SMTP are reported with 50x basic error codes, not 55x
error codes.  And https://tools.ietf.org/html/rfc3463#section-3.8
defines 5.7.1 as:

      X.7.1   Delivery not authorized, message refused

         The sender is not authorized to send to the destination.  This
         can be the result of per-host or per-recipient filtering.  This
         memo does not discuss the merits of any such filtering, but
         provides a mechanism to report such.  This is useful only as a
         permanent error.

The fact that a system administrator happened to decorate the policy
rule with misleading English text, does not fundamentally alter the
nature of the response.

Sadly, on the Internet, if botnets are found to use HELO address
literals with non-trivial frequency, but "legitimate" MTAs almost
never do, and are often held to "a higher standard" (e.g. Google
placing stringent requirement on relaying via IPv6), then sysops
are unsurprisingly implementing various stop-gap measures to
reduce the quantity of junk accepted by their systems.

The text messages that follow the numeric codes are sometimes even
deliberately non-specific.  That's life.  This particular case was
actually pretty usable, clearly the receiving MTA does not accept
address literals in the HELO, whether or not that's required by
some RFC the administrator may have been too busy to read.

And that's fine, provided on balance rejecting address literals
sufficiently reduces junk in IETF lists, with little collateral
damage.

-- 
	Viktor.

---

From: John C Klensin <[email-masked]>
Date: Sat, 14 Dec 2019 16:48:57 -0500

Victor,

>From my point of view, there are only two flaws in your
reasoning, one minor and one major:

Minor: No matter what is going on, where, and why, I can see
little or no reason for blaming this on RFC 2821. 5321 was
published in October 2008, over eleven years ago and 2821 only
seven years before that.  If someone claimed that this was due
to the much-older RFC 821, I'd believe them, but then they would
presumably be treating EHLO as a syntax error and not returning
the type of response specified in 2821 (or 1869 and its
predecessors).

Major: There is a good deal of language in 5321 (most of it
carried forward exactly from 2821) that says, in essence, that
if following particular provisions of the standard causes Bad
Things to happen operationally, then it it ok to deviate from
those provisions for self-protection. The standard deliberately
does not say so, but the WG very much had attacks by spammers in
mind when those exceptions were specified.  However, there is no
such exception for address literals: 5321 says they MUST NOT be
rejected, at least as syntax errors.  I think I can read 5321 as
allowing rejection because Bad Things have come from particular
IP addresses, or even some closely-related family of them, just
as it is reasonable to reject and EHLO or MAIL command
containing and argument of, e.g., malware-are-us.example.  But,
"if it is an address literal, reject it" is clearly prohibited.
Now, since you mentioned dogs in your explanation, there is a
longstanding principle that, if the IETF wants its standards to
be taken seriously, we are obligated to eat our own dog food.  I
think that dog food consumption principle, viewed very
pragmatically and in terms of the IETF's desire to have its work
taken seriously, suggests that, even if every other mail server
on the Internet is rejecting address literals, we must not do
so.  For us, the proper remedy if a provision of a standard has
become unworkable is an I-D that changes that provision and gets
IETF consensus for doing so.  I haven't seen an I-D containing
that proposal yet whether from you, from the members of the IETF
leadership who decided that address literals be rejected, or
from anyone else.

    best,
     john

--On Saturday, December 14, 2019 15:45 -0500 Viktor Dukhovni
<[email-masked]> wrote:

> What's been lost in this discussion, is that plainly the MTA
> is NOT reporting a syntax error.  Yes the MTA is Postfix, and
> no Postfix does not reject address literals as a built-in
> feature, that sort of access policy is something the
> administrator would have to craft for himself.  The apparent
> syntax error is just an illusion suffered by speakers of
> English.  A dog[1] would arrive at a more correct
> interpretation:
> 
> Syntax errors in SMTP are reported with 50x basic error codes,
> not 55x error codes.  And
> https://tools.ietf.org/html/rfc3463#section-3.8 defines 5.7.1
> as:
> 
>          The sender is not authorized to send to the
> destination.  This          can be the result of per-host or
> per-recipient filtering.  This          memo does not discuss
> the merits of any such filtering, but          provides a
> mechanism to report such.  This is useful only as a
> permanent error.
> 
> The fact that a system administrator happened to decorate the
> policy rule with misleading English text, does not
> fundamentally alter the nature of the response.
> 
> Sadly, on the Internet, if botnets are found to use HELO
> address literals with non-trivial frequency, but "legitimate"
> MTAs almost never do, and are often held to "a higher
> standard" (e.g. Google placing stringent requirement on
> relaying via IPv6), then sysops are unsurprisingly
> implementing various stop-gap measures to reduce the quantity
> of junk accepted by their systems.
> 
> The text messages that follow the numeric codes are sometimes
> even deliberately non-specific.  That's life.  This particular
> case was actually pretty usable, clearly the receiving MTA
> does not accept address literals in the HELO, whether or not
> that's required by some RFC the administrator may have been
> too busy to read.
>

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 14 Dec 2019 17:27:52 -0500

> On Dec 14, 2019, at 4:48 PM, John C Klensin <[email-masked]> wrote:
> 

Well, my point (perhaps not sufficiently clear) was that the
allusion to RFC2821 was the "blah blah" part of the response
and should not carry any weight.  The error is "550 5.7.1"
indicating a policy that rejects EHLO address literals.  The
RFC allusion is a distraction.

That leaves two questions.  The first is:

   * Does the community wish to hold the MTAs hosting IETF lists
     to higher standards of accuracy in the free-form text portion
     of policy-based failure responses.

If so, someone could ask the AMSL team to update the message text.
I'm not too hung up on the message text, but in this case the
misleading text is unlikely to keep the bad guys from guessing
what they need to do to get around the filter.  So in this case
there's little reason to give a more accurate text message.

> Major: [...] The standard deliberately
> rejected, at least as syntax errors.

Well again, my point was that they are not being rejected as
syntax errors.  They are rejected as a matter of access policy.

> But, "if it is an address literal, reject it" is clearly prohibited.

That battle is lost, address literals, usually accepted from
authenticated submission users or from clients on local networks
are widely rejected from strangers by inbound MTAs.  If the
collateral damage is sufficiently low, and enough junk is stopped,
the tactic will get (and is being) used.

> be taken seriously, we are obligated to eat our own dog food.

Which brings us to the second, and more important question:

  * To what extent is the community willing to stand fast in
    the defense of principled interoperability requirements,
    even when essentially the only parties benefiting from
    this are abusing access?

> I think that dog food consumption principle, viewed very
> so.

I sympathise with this point of view, but also keep in mind
that there is no RFC police, and RFCs specify what must do
in order to interoperate, but ultimately do not coerce
interoperability.

In practice some aspects of some RFCs have proved unworkable,
or the situation on the ground has changed over time, and
actual systems deviate from the RFCs to achieve the interoperability
they need "in practice".

I don't believe that deviating in some small details from
an 11-year old document (that on the whole has aged quite
well), which in turn is a conservative revision of an
18-year old document, ... amounts to disrespect of the
document as a whole.  Things change...

> For us, the proper remedy if a provision of a standard has
> IETF consensus for doing so.

I'm sure you know that doing that often requires a great deal
of effort that may not be commensurate with the reward.

> I haven't seen an I-D containing that proposal yet whether
> from you, from the members of the IETF leadership who decided
> that address literals be rejected, or from anyone else.

Well, you're working on 5321bis, would you be willing to add
text that states that use of address literals by relays is
now strongly discouraged, because they are not infrequently
rejected by the receiving SMTP server?

I would support such text, it represents operational reality.
It could go in "operational considerations".

-- 
	Viktor.

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sat, 14 Dec 2019 20:18:01 -0500

> RFC allusion is a distraction.

I'm pretty sure that everybody agreed on that 2 weeks ago.

The problem is that the IETF mail server shouldn't be actively
misleading non-spammers about the reason for their mail rejection.

Did anybody ever get the attention of the people the email support
is outsourced to?

---

From: John C Klensin <[email-masked]>
Date: Sat, 14 Dec 2019 20:42:06 -0500

> I'm pretty sure that everybody agreed on that 2 weeks ago.

I thought so.
 
> Did anybody ever get the attention of the people the email
> support is outsourced to?

Yes.  Hector and I did.  The response was, in essence, that the
IETF leadership told them to do it.  I've been too busy to
follow up, but watch for a note within the next 24 hours or so
-- I'll copy this list.

best,
  john

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 15 Dec 2019 12:42:29 +0100

> rejected by the receiving SMTP server?

I don't know how good a tactic it is to reject address literals.  Its rationale
makes little sense.  How about a response like this:

550 5.7.1 <[A.B.C.D]> Please write "foo.example" instead of "[A.B.C.D]"

Too tutorial?

> It could go in "operational considerations".

If we reject [A.B.C.D], why don't we also reject foo.example?

The reason why SMTP requires to accept such stuff, AFAIUI, is to allow a broken
server to still be able to do its job.  Is it so?  Compare that to, e.g., DANE,
where a certificate error can shut a server up.

Best
Ale
--

---

From: Jeremy Harris <[email-masked]>
Date: Sun, 15 Dec 2019 12:17:54 +0000

> where a certificate error can shut a server up.

It was so, back in the days when mailservers were few and trustworthy
and before spammers and marketers arrived.

Now, it's a flag for a system that the operator cannot be bothered to
set up properly, or was never supposed to be sending mail at all
(and has been hijacked).

-- 
Cheers,
  Jeremy

---

From: Keith Moore <[email-masked]>
Date: Sun, 15 Dec 2019 11:13:38 -0500

> where a certificate error can shut a server up.

IMO, hosts should not be required as a matter of SMTP protocol to use 
DNS names, not even to send mail.   DNS is simply a convenient way of 
finding IP addresses and allowing applications to continue to operate 
across infrequent changes in IP addresses.   It is not "a control plane 
for the Internet" (as some are now saying), it is not a call setup 
protocol, and it is nowhere nearly universal.

There are very many IP networks that do not use DNS but which still use 
SMTP to send mail, even if only to a forwarder. Before web browsers were 
commonplace, many networks did not use DNS except to forward email to 
the Internet.   The popularity of web browsers has resulted in much 
wider support for DNS, but there are still many networks that aren't 
intended to support humans using web browsers.   For some of those 
networks, DNS is not a feature, it is a liability - it adds nothing of 
value for them and is something that can break and cause (say) assembly 
lines to fail.

Of course if an organization wants to refuse mail based on a MAIL FROM 
address containing an IP address literal, it is free to do so.   But 
there is no particular reason I can see that an outgoing mail relay 
needs to know its own DNS name, or even to have one.

Keith

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 15 Dec 2019 18:14:30 +0100

On Sun 15/Dec/2019 17:13:38 +0100 Keith Moore wrote:
>> The reason why SMTP requires to accept such stuff, AFAIUI, is to 
>> allow a broken server to still be able to do its job.  Is it so?
>> Compare that to, e.g., DANE, where a certificate error can shut a
>> server up.
> 
> DNS names, not even to send mail.

In a server to server connection, the HELO name can be used to provide
SPF authentication.

> DNS is simply a convenient way of finding IP addresses and allowing
> applications to continue to operate across infrequent changes in IP
> addresses.

To that end, host files would suffice.  In addition to associating an
IP number to a name, DNS establishes a delegation hierarchy.

> It is not "a control plane for the Internet" (as some are now
> saying), it is not a call setup protocol, and it is nowhere nearly
> universal.

Hmm... not universal.  However, the global Internet, given that you
spell it with a capital I, is, well, global.  Multiple, non-global
DNSes were one of the doom future scenarios ISOC hypothesized a while
ago.

> There are very many IP networks that do not use DNS but which still
> use SMTP to send mail, even if only to a forwarder. Before web
> browsers were commonplace, many networks did not use DNS except to
> forward email to the Internet.   The popularity of web browsers has
> resulted in much wider support for DNS, but there are still many
> networks that aren't intended to support humans using web browsers.  
> For some of those networks, DNS is not a feature, it is a liability -
> it adds nothing of value for them and is something that can break and
> cause (say) assembly lines to fail.

In a walled garden it makes sense to accept mail from anyone who is
able to connect.  To do so globally has proven to be unaffordable.
Anti-spam techniques vary from dropping messages at random to
assessing senders reputation, each has its FPs.

> needs to know its own DNS name, or even to have one.

The reason to have a name is that you can port it, along with your
reputation, across IP address changes.

Best
Ale
--

---

From: S Moonesamy <[email-masked]>
Date: Sun, 15 Dec 2019 09:25:46 -0800

Hi Alessandro,
>550 5.7.1 <[A.B.C.D]> Please write "foo.example" instead of "[A.B.C.D]"

Yes.

There was a specification which was published 18 years ago [1].  It 
was recommended to use four small decimal integers separated by dots 
and enclosed by brackets if the host is not known to the domain name 
system and communication is blocked.  Is there any information to 
show that the recommendation was followed?

Regards,
S. Moonesamy

1. Whether the IETF is able adhere to its standards is a different matter.

---

From: John C Klensin <[email-masked]>
Date: Sun, 15 Dec 2019 13:37:33 -0500

--On Saturday, December 14, 2019 17:27 -0500 Viktor Dukhovni
<[email-masked]> wrote:

> actual systems deviate from the RFCs to achieve the
> interoperability they need "in practice".

Sure.

> document as a whole.  Things change...

Yes... and no.
If you decide to commit those deviations in your server, I
wouldn't be inclined to say a word, instead attributing it a
local operational decision and/or an invocation and extension of
the principle that a mail server may refuse to handle any
message for any reason, both of which are called out in the
spec.  However, two distinctions are, I think, worth making
(without quibbling about the response code or error message
text):

(1) Relative what the spec says, you would be on much more firm
ground rejecting 
   MAIL FROM:<[email-masked]>
then
   EHLO [127.0.0.5]   or 
   EHLO maleware-are-us.example
In the first instance, you'd be rejecting the message on the
basis of its sender, which 5321 very explicitly allows.  In the
second pair, you'd be rejecting any message from a point of
origin, perhaps on the basis of a domain name that does not
identify a host or an IP address that doesn't point anywhere
useful.  What 5321 says about those things is that you can
check, you can use what you find in logging or reporting, but
you MUST NOT reject on that basis.

(2) We are talking about the ietf.org servers here.  I think
that the IETF, quietly and without public explanation or
evidence of community consensus, ignoring the requirements of
its own specifications is bad news.  That a specification is
eleven years old does not seem to me to be relevant and no more
appropriate than it would be to argue that some provisions of
RFC 791 that have not bee updated or replaced by later specs
should now be ignored because it is 38 years old.  On our own
systems, we should follow our own specs until we are ready to
change them and have community consensus for doing so.
 
>> become unworkable is an I-D that changes that provision and
>> gets IETF consensus for doing so.
> 

Well, I think I know two things about that.  One is that
proposals that are tightly focused and that have clear consensus
usually go through the system rather quickly.  The second is
that, when the first is not the case, something is badly broken
to the point that there should be appeals, explanations to the
Nomcom, and/or efforts to fix the recall procedures so that they
are usable.

In general, if there is consensus around there being a problem
with a spec, an I-D specifying a change does not go through
quickly, and none of those actions are taken, it is probabl7 an
indication that no one really cares. 

> It could go in "operational considerations".

Short answer, as editor, no, in part because 5321 already
discourages the use of address literals unless they are
necessary and because the discussion so far does not lead me to
conclude that there is strong consensus for making a change.
Note in particular that prohibiting the us of IP literals (which
you did not suggest) would be equivalent to requiring that all
SMTP clients and servers have domain names, a requirement that
821 -> 2821 -> 5321 rather carefully avoid.  

More on my general editing rules/ assumptions and policies in a
separate note, forthcoming.

    john

---

From: Keith Moore <[email-masked]>
Date: Sun, 15 Dec 2019 15:09:02 -0500

> In a server to server connection, the HELO name can be used to provide

Don't assume that the mail is going to arbitrary destinations. In many 
cases SMTP is used for status and error reporting.   As long as the 
devices sending such mail are sending it through relays that are 
themselves trusted by the destinations of these messages (whether via 
SPF or whatever other means), the system works.

More generally, we should not assume that everybody uses Internet 
technology in the same way.   The Internet (by which I mean all networks 
using IP) is incredibly diverse and there is no usage scenario that is 
representative of all or even most cases.   We should keep that in mind 
when writing protocol specifications.

> IP number to a name, DNS establishes a delegation hierarchy.

Even host files are not necessary in many use cases.   Many useful 
applications using IP do not need to use names at all.

> DNSes were one of the doom future scenarios ISOC hypothesized a while

I still think there should only be one DNS root and one delegation tree.

(Which also means I think that two-faced DNS is a protocol and 
architecture violation if it is used to provide lookups for names that 
aren't delegated.)

> able to connect.  To do so globally has proven to be unaffordable.

I agree with both of these statements.

It's important to understand that private, isolated, and intermittently 
connected networks are all useful.   "Walled garden" has somewhat of a 
pejorative sense, as a way to lock in customers, but there are isolated 
networks that aren't isolated for that reason.  There are many useful 
(often industrial) IP based networks that have no need for connect to 
external networks, perhaps not even via NATs or application level 
gateways.   Those networks still use Internet protocols, and we do our 
users a disservice if we don't consider their needs when writing and 
updating our protocol standards.

> reputation, across IP address changes.

That only matters if the devices using those forwarders to send mail, 
use names as a means to learn how to connect to them.  In many cases, 
they do not use names, and should not be required to do so.

Keith

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sun, 15 Dec 2019 15:35:52 -0500

> assessing senders reputation, each has its FPs.

However, consider this:

If we write a standard that says address literals aren't permitted, then the
writers of MTAs will be totally in their rights to refuse to accept mail with
an address literal.

At which point it doesn't matter if it makes sense in a walled garden, because
a device in the garden won't find anybody to talk to.

I once had to deal with the loss of a 17 terabyte LUN in a very large
filesystem - made even more troublesome by the fact that many large files ended
up with holes in the middle because they were striped across multiple LUNs.  We
ended up having to recover over 100T of data.  All caused by a vendor microcode
update borking alert e-mails from a large storage array - so we stopped getting
alerts that drives were failing. We lost 3 drives in an 8+2 RAID6, plus another
2 drives in other raidsets in a sudden flurry of failures before we got
suspicious that no drives had failed - a monthly occurrence when you have many
42U racks full of drive trays.

(For the record, the storage arrays didn't have a concept of a FQDN, and
lived in a 10.X/26 subnet. We could SSH in via a jump host, and relay SMTP
out. Although the devices *had* SNMP traps, our SNMP monitoring system
couldn't deal with a relayed trap where the SNMP packet showed up with
an IP address other than what the trap data said (our network engineering
guys got told "Fixed in the next release".  So we relied on email alerts, which
*had* been totally reliable for the first few years of the arrays.....)

Not that I'm bitter or anything.... :)

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 15 Dec 2019 16:18:11 -0500

> On Dec 15, 2019, at 3:35 PM, Valdis Klētnieks <[email-masked]> wrote:
> 

As John Klensin observed, I did not suggest (nor did anyone else in this thread)
that address literals should be prohibited.  They are fine to use in private
networks, and "mostly" work even on the public Internet, but (hearsay) over the
years I've seen repeated posts on e.g. the Postfix users list, where sysops report
using regexp HELO name filters that would block address literals.

I do not encourage such filters, they probably don't generally catch junk to
warrant their use, but it is my (admittedly unscientific) impression that
such filters are not uncommon.  It would be good to have real data on
this, but getting it would be a non-trivial exercise.

So the question to AMSL would be whether the rules were adopted second-hand,
and perhaps don't specifically address abusive traffic directed at ietf.org.

As to the standards landscape, address literals should remain part of the
SMTP grammar, but prudent relays should avoid using them outbound.  That's
almost what the document says already.  The only thing I'd consider changing
is adding operational guidance that if you do use them, you may encounter
difficulties delivering to outside organizations across the public Internet.

Now that advice is (admittedly) based on impressions formed by reading various
posts from email administrators over the years, and could be heavily biased.
It is weakly supported by the fact that even AMSL saw fit to implement such
filters, but perhaps they are one of the exceptions that prove the rule[1].

-- 
	Viktor.

[1] Here I'm going with the archaic meaning of "prove" meaning "test".

https://en.wikipedia.org/wiki/Exception_that_proves_the_rule

[...] The alternative origin given is that the word "prove" is used in the
archaic sense of "test". In this sense, the phrase does not mean that an
exception demonstrates a rule to be true or to exist, but that it tests the
rule, thereby proving its value. There is little evidence of the phrase being
used in this second way.

[ Recursion warning: My use of the idiom is apparently an exception. :-) ]

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sun, 15 Dec 2019 17:34:25 -0500

> using regexp HELO name filters that would block address literals.

I admit not knowing how to express in RFC language the concept that the MTA
code MUST (2119 sense) allow address literals, while email admins be advised
that accepting them on the open Internet is, as a practical matter, somewhere
between MAY NOT and SHOULD NOT (but not  rising to MUST NOT).  In three decades
of dealing with RFCs, I can't recall a prior instance where we've ended up
mandating that an implementation support something that's usually a bad
idea except in specialized corner cases...

That's why I let John Klensin write them. :)

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 15 Dec 2019 18:11:12 -0500

> 
> 
> On Dec 15, 2019, at 5:34 PM, Valdis Klētnieks <[email-masked]> wrote:
> 

Note that I've at no point advocated that receiving MTAs SHOULD NOT
(or MAY choose not to) accept address literals.  They definitely
SHOULD accept address literals.  I'm only saying that, despite the
foregoing, in practice they "may" (lower case) accept only FQDN forms,
and so a sending MTA is perhaps well advised to avoid their use.

> That's why I let John Klensin write them. :)

Indeed. :-)

-- 
	Viktor.

---

From: S Moonesamy <[email-masked]>
Date: Sun, 15 Dec 2019 16:48:39 -0800

Hello,

I tried the following [1]:

   C:EHLO [192.0.2.1]
   S:250-ietfa.amsl.com
   S:250-PIPELINING
   S:250-SIZE 67108864
   S:250-ETRN
   S:250-STARTTLS
   S:250-AUTH PLAIN LOGIN
   S:250-AUTH=PLAIN LOGIN
   S:250-ENHANCEDSTATUSCODES
   S:250 8BITMIME
   C:MAIL FROM:<>
   S:250 2.1.0 Ok
   C:RCPT TO:<[email-masked]>
   S:550 5.7.1 <[192.0.2.1]>: Helo command rejected: RFC2821 violation

As a comment to John, Victor explained when the EHLO check is being 
done in the message at 
https://mailarchive.ietf.org/arch/msg/ietf/F6FfNvkLeQGcE9RdBkHWFsaTEjQ

I doubt that such filtering would, in general, catch much junk 
mail.  There might be a reason why it was implemented in this 
case.  It may not be a good time to ask about this as there was an 
announcement about some upgrades to ietf.org servers.

The specification would be more lengthy if operational guidance about 
such cases is included.  Anyway, it would take more than that to fix 
the incorrect recommendations which are available online.

Regards,
S. Moonesamy

1. The client IP address was used for the test.

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 16 Dec 2019 10:59:39 +0100

On Sun 15/Dec/2019 21:09:02 +0100 Keith Moore wrote:
> Don't assume that the mail is going to arbitrary destinations. In many cases
> SMTP is used for status and error reporting.   As long as the devices sending
> such mail are sending it through relays that are themselves trusted by the
> destinations of these messages (whether via SPF or whatever other means), the
> system works.

Right.  Let's not forget that we also have a submission protocol.  I think most
servers accept submissions on port 25 as well, providing per-IP authentication,
e.g. granting relay privileges to 127.0.0.1.  That way, bare bones clients
won't actually distinguish SMTP from submission and will continue to work.

> More generally, we should not assume that everybody uses Internet technology in
> the same way.   The Internet (by which I mean all networks using IP) is
> incredibly diverse and there is no usage scenario that is representative of all
> or even most cases.   We should keep that in mind when writing protocol
> specifications.

Agreed.  However, an IETF public mailing list should be allowed to make some
assumptions about the kind of clients it allows.  While syntactically
interchangeable, SMTP and submission protocols are semantically different.  If
SMTP stated that message transactions should provide some form of
authentication —helo or sender names, DKIM, whatever—  there wouldn't be
questions about why [192.0.2.1] is rejected.

> That only matters if the devices using those forwarders to send mail, use names
> as a means to learn how to connect to them.  In many cases, they do not use
> names, and should not be required to do so.

Consider a server who accepted an anonymous message from the disk driver, using
submission.  It can deliver it to root, or it can relay it.  In the latter case
it has to use SMTP.  I think it may make sense to require the message to be
deanonymized even in this use case.

Best
Ale
--

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Mon, 16 Dec 2019 05:16:19 -0500

> the incorrect recommendations which are available online.

I have to wonder if we shouldn't go in the opposite direction - having a
protocol standard that's quit nailed down so it can move to Full Standard, and
an "operational guidance" RFC/BCP that would be easier to update/replace every
5-10 years as the landscape changed....

---

From: Ned Freed <[email-masked]>
Date: Mon, 16 Dec 2019 13:49:41 -0800 (PST)

> writers of MTAs will be totally in their rights to refuse to accept mail with

The reason we didn't drop address literal support sooner is that there are lots
of low end clients which for various reasons don't have a name they can use.
They also typically don't have support for SSL/TLS or SASL.

Now, an argument can be made that these are submission clients, and that
any relay client should be configured with and use a name. But if we
push these limited clients into the submission space, we'd need to loosen
the crypto requirements there. Does anyone really want to go there?

> a device in the garden won't find anybody to talk to.

FWIW, I checked my home logs for the past year, and not only do I see a number
of IP literals, they are without exception associated with legitimate mail.

Mileage varies.

				Ned

---

From: Michael Peddemors <[email-masked]>
Date: Mon, 16 Dec 2019 14:41:51 -0800

On 2019-12-16 1:49 p.m., Ned Freed wrote:
>>>> forward email to the Internet.�� The popularity of web browsers has
>>>> networks that aren't intended to support humans using web browsers. �

Interesting, especially given the large 'botnet' contingent that still 
uses IP Address literals, eg the one currently going around (13k+ IP(s) 
reported over night) that send using [127.0.0.1]  in the EHLO, typically 
used during dictionary/weak pass discovery..

Or the other couple of bots that use random generated dotted quads in 
the reserved IP space..

Or the many IoT bots that if they can't figure out what the external PTR 
record is for the device, will default to the IP Address of the external 
interface.

While almost every MTA can handle IP literals, I am sure that MOST tend 
to look unfavorably when doing spam filtering for instance, but of 
course there are many 'a.b.c.d' address formats that will have the same 
problem.. eg 'localhost.localdomain', or the infamous naming 
convention.. *ahem*

35.186.158.237	x4	237.158.186.35.bc.googleusercontent.com
35.247.162.235	x4	235.162.247.35.bc.googleusercontent.com

Well, tongue in check, but you get the point.  Nothing wrong with IP 
literals, but you can expect to have a more 'challenging' time getting 
email accepted from many email operators around the world.

-- 
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

---

From: Keith Moore <[email-masked]>
Date: Mon, 16 Dec 2019 18:02:16 -0500

On 12/16/19 5:41 PM, Michael Peddemors wrote:

> uses IP Address literals, eg the one currently going around (13k+ 
> IP(s) reported over night) that send using [127.0.0.1]  in the EHLO, 
> typically used during dictionary/weak pass discovery..

But that's a very interesting special case.   If you're operating an MTA 
on a server and you know for a fact that nothing on that server send 
SMTPs traffic to localhost, you can safely reject every session using 
HELO or EHLO [127.0.0.1].   Anything that presents that address in 
HELO/EHLO is lying about its address, and that's a perfectly valid 
reason to reject that session.

But if any significant fraction of EHLO [some-ip-address] is valid mail, 
use of an IP address literal in EHLO is NOT, by itself, a valid reason 
to reject the mail.   And if you uniformly reject traffic that starts 
with EHLO [some-ip-address] you have no idea how much valid mail is 
being rejected, because you never even see that mail.

Also, these things change over time.   Just because there was once some 
bot that was sending out IP addresses in EHLO, doesn't mean it's a valid 
filtering criterion today.

Keith