Subject: Re: [ietf-smtp] IETF Policy on dogfood consumption or avoidance - SMTP version

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 17 Dec 2019 14:24:43 -0500

> On Dec 17, 2019, at 2:02 PM, Hector Santos <hsantos=[email-masked]> wrote:
> 
> But here is I see it:
> 
> 1) Yes, everyone agree the response text needs to be fixed up, but
> 
> 2) It is in fact a violation of RFC2821/5321 specification when a rejection is applied by a server to a perfectly valid ip-literal per specification, and

It is not in fact.  A receiving MTA can refuse your email for any reason.
As a matter of RFC-compliance it MUST recognize address literals as
valid syntax (which it did by returning a 550 rather than 500 or 501),
but is then free to reject them on policy grounds.

> 3) It lacks consistency in its operational decision on what Client Mail/Host Names are rejected or accepted.

This is also not true.  It consistently rejects address literals because
doing so carries little risk of false positives (as already explained on
the ietf-smtp list, where the more technical discussion belongs).  "Real"
MTAs use domain names.  It is as simple as that.

> If a rejection is going to apply to ip-literals, hence enforcing a FQDN, then at the very least, it should validate the FQDN.

No, because enough "Real" MTAs use HELO domain names that don't map to
their own IP address, or any address at all.  So the risk of FPs is
too high.

There is no a priori discrimination here, it is all just based on what
one can get away with to reduce spam without blocking a non-trivial
volume of legitimate email.

> The mail.ietg.org servers appears to accept any FQDN including a existing
> FQDN which does not match the connecting IP address and a non-existing FQDNs:

As they must for operational reasons.

> Yet, it does not validate the FQDN. Why?

Because, much as one might want to, too many "Real" MTAs (sending
legitimate traffic) have FQDNs that would fail verification.

-- 
	Viktor.

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Dec 2019 18:04:23 -0500

Viktor,

I believe we are in agreement with the same principles. But maybe not. 
Not sure. I already implied local policy always prevails.

That said, I believe the IETF "operational decision" was incorrect and 
inconsistent. To honor it, it may set a negative precedent for future 
implementer trust and confidence in protocol design.  Is it advocating 
others to follow? After all, an SDO did it, so why not across all 
systems?

We may have to explicitly embed this clause in the protocol:

      A "MUST NOT" protocol element MAY be overridden with "550" Policies.

That is the implication we have here.  Keeping in mind that these new 
550 policies are generally based on some "Bad Guy" presumptions. It 
did not explicitly exist before RFC5321. Previously, 550 was like a 
"Catch all" reply code.   Since 2003, with GreyListing, SPF, DKIM/SSP, 
DKIM/ADSP, DKIM/DMARC, etc, the beauty of these new protocols to help 
deal with "bad guys," they were ALL consistent with the written SMTP 
specifications.  They did not change the SMTP protocol except for one 
thing.  I was instrumental during RFC2821BIS in proposing and Klensin 
adding section 7.9. "Scope of Operation of SMTP Servers."   With the 
advent of the new DNS TXT-based applications, we needed new SMTP 
justifications for rejections.  The bar was raised for SMTP compliancy 
and correction.

    7.9.  Scope of Operation of SMTP Servers

    It is a well-established principle that an SMTP server may refuse to
    accept mail for any operational or technical reason that makes sense
    to the site providing the server.  However, cooperation among sites
    and installations makes the Internet possible.  If sites take
    excessive advantage of the right to reject traffic, the ubiquity of
    email availability (one of the strengths of the Internet) will be
    threatened; considerable care should be taken and balance maintained
    if a site decides to be selective about the traffic it will accept
    and process.

    In recent years, use of the relay function through arbitrary sites
    has been used as part of hostile efforts to hide the actual origins
    of mail.  Some sites have decided to limit the use of the relay
    function to known or identifiable sources, and implementations SHOULD
    provide the capability to perform this type of filtering.  When mail
    is rejected for these or other policy reasons, a 550 code SHOULD be
    used in response to EHLO (or HELO), MAIL, or RCPT as appropriate.

What it needs to update here is to include DATA because of ADSP and 
DMARC.

Again, now of the new augmented protocols did not change SMTP.

For me, being consistent allows for raising the bar.  In regards to 
the new ietf mail policy, if it was consistent with this new 550 
policy to reject legitimate ip literals for a yet to be explained 
non-reason, it would be more consistent and acceptable,  if it also 
consistently enforced a correct FQDN using a new 550 policy 
justification to reject IP::DOMAIN mismatches.  After all a "real" MTA 
is expected to use rDNS to obtain the correct FQDN for the sender 
machine.  No?  That is not always optimal. Small lite weight SMTP 
clients exist.  PTR slows them down.  What if there is no PTR record? 
   After all, again, more inconsistency, are we promoting PTR records 
now? For a certain period there, we were discouraging them, in fact, I 
think SPF tried to deprecate it. I am not going to bother to confirm 
but I recall the debates.

Consistency with everyone "expected" to play by the same rules is 
paramount for maximum interoperability and with minimum support. IMO, 
this mail.ietf.org odd "operational decision" could drastically change 
things because now others will follow.

I am still trying to understand why the legit IP-literal is being 
rejected in the first place.  I don't see any legit reason for it.  Do 
you?

Thanks

-- 
HLS

-- 
HLS

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 17 Dec 2019 18:35:15 -0500

[ Replying to just ietf-smtp, I hope that's OK. ]

> On Dec 17, 2019, at 6:04 PM, Hector Santos <hsantos=[email-masked]> wrote:
> 
> For me, being consistent allows for raising the bar.  In regards to the new ietf mail policy, if it was consistent with this new 550 policy to reject legitimate ip literals for a yet to be explained non-reason, it would be more consistent and acceptable,  if it also consistently enforced a correct FQDN using a new 550 policy justification to reject IP::DOMAIN mismatches.  After all a "real" MTA is expected to use rDNS to obtain the correct FQDN for the sender machine.  No?  That is not always optimal. Small lite weight SMTP clients exist.  PTR slows them down.  What if there is no PTR record?   After all, again, more inconsistency, are we promoting PTR records now? For a certain period there, we were discouraging them, in fact, I think SPF tried to deprecate it. I am not going to bother to confirm but I recall the debates.

Your insistence on "consistency" is misplaced.  The literals are
rejected because they are almost never used by legitimate MTAs,
and sufficiently often used by abusive clients.

Nobody is attempting to "validate" the HELO name.  The policy goal
is to block as much junk as possible at least cost, without incurring
too many FPs.  Blocking address-literals *from strangers* (i.e. not
submission or fax-to-email machines on the local network, etc.) appears
to work well enough.

Validation of HELO FQDNs does not work well.  The policy is not trying
to enforce any sort of RFC correctness, it is just a (presumably)
modestly effective anti-spam measure.  That's all.

> Consistency with everyone "expected" to play by the same rules is paramount for maximum interoperability and with minimum support. IMO, this mail.ietf.org odd "operational decision" could drastically change things because now others will follow.

Everybody is expected to use an FQDN.

> I am still trying to understand why the legit IP-literal is being rejected in the first place.  I don't see any legit reason for it.  Do you?

Because (once again) their use by legitimate MTAs is minuscule,
but their use by sources of abuse is not.

-- 
	Viktor.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 18 Dec 2019 00:40:44 +0000

>          A "MUST NOT" protocol element MAY be overridden with "550" Policies.
    
Bleh. We often say "we're not the protocol police."

I think it's always clear that local policy can override.

Is the IETF going to come and arrest it's own website?

---

From: Randy Bush <[email-masked]>
Date: Tue, 17 Dec 2019 19:59:20 -0800

> I think it's always clear that local policy can override.

that's a SHOULD NOT

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Dec 2019 23:23:02 -0500

> Is the IETF going to come and arrest it's own website?

No. It should keep it simple - follow the standard protocol.

The bottom line is this breaks code, 37+ years of consistent protocol 
SMTP state machine logic, its not a bad vs good thing, regardless of 
the frequency of usage, regardless of scale, even the hobbyist, the 
entire community, in my long pre-internet protocol design and 
engineering world, a MUST concept is burned in, SHOULDs and MAYs are 
optional features provided to sysops/customers, your switches.

Do it this way, and the network/system homogeneous or heterogeneous, 
it all works.   Due to inconsistent application of some unknown rule, 
it broke a SMTP "standard" protocol.   In lieu of a specific black 
listing/block "trained thing," my concern this gem will most 
definitely begin to be copied and soon enough it will trap more 
clients using ip literals.

Its wrong. its that simple.  I am the OP. I filed the action item.  I 
only discovered it because the long time whitelisted mail.ietf.org 
server changed it's whitelisted ip address.  So a unsolicited mail 
sender CBV (Call Back Verification) was triggered. It failed due to 
the mail.ietf.org server violated the RFC2821 protocol. It had no 
legit reason for the rejection.

For the record, a CBV is the most effective spoof/fake reverse path 
filter method that exist.  I have 16 uses of daily statistics to prove 
it, yield 50-80% rejection rates near near-zero (hate to say none) 
false/true positive/negatives.

http://www.winserver.com/public/spamstats.wct

I will follow the cogs with this. Just do the right thing. I am doing 
whats prudent and practice for my compliant SMTP package ALWAYS. In 
fact, we filter on violations of the ip-literal mismatches. I am not 
going to reject legitimate SMTP compliant machine HELO/EHLO 
ip-literals. If my sysops and admins want to do that, thats up to them 
but its not coming from me, I am not going to encourage it because 
there is absolute no legitimate reason to do so and I think the 
mail.ietf.org admins, whoever, are wrong about this. It is unfair to 
Klensin and the mail community who work hard to get these things right.

I'm done.

--
HLS

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Dec 2019 23:23:45 -0500

>> I think it's always clear that local policy can override.

+1, exactly.

-- 
HLS

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 18 Dec 2019 15:46:53 +0000

I overstated, sorry.

On 12/17/19, 11:23 PM, "Hector Santos" <[email-masked]> wrote:

    +1, exactly.

    -- 
    HLS

---

From: S Moonesamy <[email-masked]>
Date: Wed, 18 Dec 2019 10:32:26 -0800

Hi Eliot,
At 12:12 AM 18-12-2019, Eliot Lear wrote:
>If one looks at RFC 5321, there are three reasons to think that 
>really the standard does not prohibit the behavior being 
>described.  First, the text in 4.1.4 doesn't actually prohibit the 
>rejecting literals.  Here is what is said:
>
>    If the EHLO command is not acceptable to the SMTP server, 501, 500,
>    502, or 550 failure replies MUST be returned as appropriate.
>
>At this point in the transaction, it may not be readily apparent 
>that the EHLO indeed is unacceptable.  That's because the server may 
>need to gather more information, such as the recipient, in order to 
>check against SPF records or other inputs delivered later to make a decision.

The text in Section 4.1.4 is about the restrictions on the order in 
which commands may be used.  The is also a bug fix for STD 10 in that 
section.  As a nit, the SPF verification is done on the "MAIL FROM" 
identity (RFC 7208).

>    An SMTP server MAY verify that the domain name argument in the EHLO
>    command actually corresponds to the IP address of the client.
>    However, if the verification fails, the server MUST NOT refuse to
>    accept a message on that basis.
>
>That "MUST NOT" conflicts with the most important principle we have 
>to go with in order to defend against spam and malware, as clearly 
>stated in Section 7.9:

There was a time when not adhering with the above requirement would 
cause rejection of (valid) mail messages from a well-known mail service.

>    to the site providing the server.
>
>And so the server chose to reject a message.  I would add that an 
>erratum should be filed to resolve this conflict.

There is the following sentence after the one which you quoted: 
"However, cooperation among sites and installations makes the 
Internet possible".  That sentence, and the sentence which follows 
it, serves as an explanation for the recommendation at the end of the 
paragraph.

What is the purpose for the (suggested) erratum?  Is it to fix a 
technical defect in the specification?

Regards,
S. Moonesamy