Subject: Re: [ietf-smtp] [dmarc-ietf] Fwd: RFC 7372 on Email Authentication Status Codes

From: Ned Freed <[email-masked]>
Date: Wed, 17 Sep 2014 13:41:30 -0700 (PDT)

> Hi,

> On 09/16/14 05:08, Murray S. Kucherawy wrote:
> >          URL: https://www.rfc-editor.org/rfc/rfc7372.txt
> >
> > This document registers code points to allow status codes to be
> > returned to an email client to indicate that a message is being
> > rejected or deferred specifically because of email authentication
> > failures.
> >
> > This document updates RFC 7208, since some of the code points
> > registered replace the ones recommended for use in that document.

> I hope this is not off topic here:

(chair hat on)

It is. I've cc'ed and directed replies to the ietf-smtp list, where 
this sort of discussion belongs.

(chair hat off)

> RFC 7372 proposes to use a 550 response code for reverse DNS auth
> failures, see section 3.3.

Correct.

> Reverse DNS checks are usually done early in the connection (like IP
> blocks) in the connection establishment stage of the SMTP dialog.

> RFC 5321 allows only a 554 error response there, see section 4.3.2.

You're misreading RFC 5321. Nowhere does it say that its lists of possible
responses are exhaustive. So it is perfectly permissible for RFC 7372 to
specify additional possible responses as long as they fit into the overal
theory of reply codes.

This sort of thing has been done many times.

> So, shouldn't a 554 code be used here? Or does RFC 5321 need an update?

Neither. See above.

				Ned

---

From: Christoph Kluge <[email-masked]>
Date: Thu, 18 Sep 2014 20:24:38 +0200

On 17 Sep 2014, at 22:41, <[email-masked]> <[email-masked]> wrote:

> It is. I've cc'ed and directed replies to the ietf-smtp list, where this sort of discussion belongs.
> 
> (chair hat off)
> 

It doesn’t say that (well-known) reply codes can be chosen arbitrarily in
whatever SMTP stage as long as they fit some formal schema either.

What theory of reply codes?  THE theory of reply codes goes back to FTP
conventions and the 70s which were adapted to SMTP and HTTP and
other transfer protocols.   5yz is the well-known code for a permanent negative
completion, and x5z, the category code, that meant “protocol defined” in FTP
includes “policy defined” in SMTP.

So suggesting code 550 with its semantics is perfectly okay for any SMTP
stage but CONNECT because RFC 5321 and its ancestors fortunately are pretty
clear when discussing reply codes and sequencing.

But that isn’t the point.  The question is whether SMTP clients SHOULD also
expect a “policy code” in CONNECT and MAY act accordingly, and how this
is covered by said RFC.  My concern aren’t MUAs who might wonder what
colour to select for some doesn’t-work-popup but border MTAs where
semantics and interoperability matters.  Different from the reply message, the reply
code is not just informal because in the real world, semantics come with system
behaviour.  We are fighting the battle of fending off bad reputations while maintaining
ours every day.

> This sort of thing has been done many times.

Certainly, including your own RFCs for some SMTP extensions.  But they didn’t
without also discussing implications.

>> So, shouldn't a 554 code be used here? Or does RFC 5321 need an update?

Christoph

---

From: Ned Freed <[email-masked]>
Date: Thu, 18 Sep 2014 15:44:38 -0700 (PDT)

> whatever SMTP stage as long as they fit some formal schema either.

The publication of an RFC specifying a new code or a new usage of an old code
is hardly arbitrary.

> includes “policy defined” in SMTP.

Exactly.

> semantics and interoperability matters.

If the server returned a 550 error on CONNECT, the client has three options:

(1) Immediately bounce the message, which is what it should do.
(2) In violation of protocol and common sense, treat this as a temporary
    error and retry until the message times out.
(3) Something else even more bogus.

So where's the problem? It seems the worst case scenario is that the client
retries unnecessarily and continues to fail to get through until the message
times out. And even this seems very unlikely.

>  Different from the reply message, the reply
> code is not just informal because in the real world, semantics come with
> system behaviour.  We are fighting the battle of fending off bad reputations
> while maintaining ours every day.

Well, if we're going to talk about the real world, do you have any evidence
of clients that misbehave when they receive a 550 response in a banner
as opposed to some other 5yz response? Indeed, do you have evidence of
any client that looks at anything other than the first digit in a banner
response?

I'll also note that if we had wanted to stick to previously specified usage, we
would have been stuck with 554 or 521, both of which carry the semantic of "no
SMTP service here", which to the extent that second and third digit choices
actually matter, seems far more likely to be problematic.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 19 Sep 2014 09:28:39 +0200

On Thu 18/Sep/2014 20:24:38 +0200 Christoph Kluge wrote:
> is covered by said RFC.

Every upcoming authentication method has at some point considered the
need to color its rejections with its own reply code.  For an older
snippet on 550 vs 554, see:

   John C Klensin, [yam] Ambiguities in RFC 5321, 21 Nov 2009
   http://www.ietf.org/mail-archive/web/yam/current/msg00186.html

My understanding is that SMTP reply codes serve to guide protocol
behavior rather than to ruminate deliverability statistics after the
fact.  The latter is confined to text strings, which may include
status codes.

Ale

---

From: Franck Martin <[email-masked]>
Date: Fri, 19 Sep 2014 02:35:27 -0500 (CDT)

I thought we had extended SMTP codes for that. To bring meaning to the rejection. The only thing the client MTA needs to know is what to do with the message when it is rejected.

I think that's what the xxx should express:
- don't try again
- try again in same or different session
- disconnect and try again later
- disconnect and try again on another MTA immediately
- ok
- ok continue
- ....

Toute connaissance est une réponse à une question.

> On Sep 19, 2014, at 9:28, Alessandro Vesely <[email-masked]> wrote:
> 
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: Christoph Kluge <[email-masked]>
Date: Sun, 21 Sep 2014 19:58:24 +0200

> times out. And even this seems very unlikely.

Is that all you consider when defining a new reply code or redefining an old
code for a new purpose?  If a client in its protocol engine suddenly starts
misbehaving then someone has to hire a better software engineer.  Please,
let’s not discuss trivia.  Mail systems are more than having a mail bounced,
deferred, or seeing it delivered happily, since 20 years and more.

Why do you say that reply codes in the first, second, and third digit actually
matter and at the same time that clients won’t look at the last two?  Of
course they do, all over.  No mail system beyond a certain size can do
without looking at reply codes directly or indirectly.  Reply codes can be fed
into reputations systems to execute anti-spam, anti-abuse measures in
automatic, semi-automatic or manual ways.  Reply codes are used to
monitor the reputation of outbound IPs.  What happens if the grey IPs
turn black because someone has missed to register a new reply code?
He can watch his customer care department blow up: all legitimate mails
are bounced.  And what if white IPs turn black?  He needs a good
lawyer because bounces mustn't be bounced and so he suffers serious
mail loss.  I’d call that an interoperability issue.  Shop systems are
trying to figure out what exactly happened to their billing mails, bulk
mailers are using them, etc.  Heck, people even have been parsing
the reply messages themselves, because neither base code nor
extended code gave them enough information.  You’ve no idea, what
those people have been costing us in customer care and postmaster
support.  There are weird and not so weird application of reply codes.
Personally, I’d advice anyone against using reply codes directly, and to
always only use what’s absolutely necessary, but sometimes it’s all people
have got, and their use is legitimate, they are part of the interface!

Interfaces are to join two sides neatly at the protocol level, and in
between is the machinery that makes the system tick.  That machinery
is simple and negligible if you only always think of the cute little
sendmail installations of the '80s with their 1000 mailboxes; it is
complicated and vast in the large mail systems like ours that serve many
millions of domains: information flows both ways from the interfaces into the
machinery and back from there controlling the interfaces.  Information
goes from MTAs and MSAs into the system out to the border MDAs
and vice versa.  It’s not that _we_ depend on reply codes within the
machinery but others might!   SMTP, the protocol, just is the (inter-)face
of it all, and reply codes are its eyes.  So whenever you do something
to the base RFC that _is_ the interface specification, directly or indirectly,
you have to tell people about it.  And the only way you can do this is
to put that information in the RFC itself, RFC 7372 in this case, rather
than making it a “(mailing list ) eyes only” affair that only popped up here
by chance.

> actually matter, seems far more likely to be problematic.

As for facts rather than evidence, see above.

As for code 550 specifically, no, I don’t have evidence.  No one _is_ using
it in the banner.  People have adopted 554 base code with or without
5.7.1 extended code many years ago, and so have we in the ‘90s without
issues ever since.

As for code 554: Am I the only one who doesn’t have that much an issue with it?
It doesn’t just say “No Service”, being a “policy code” it says, “No service for you
unless you come back under different conditions on your the sender’s side."  That’s
how those codes are read.  Now what can those conditions be, in the banner?
The IP itself or anything associated with the IP, a bad DNS PTR RR, an RBL
entry, etc.  I could also live with any other code "55z Go Away You Stinks” in the
greeting were it properly specified.   On the other hand I wouldn’t want see
“45z You Stinks But I Might Change My Mind”;  that seems not to be useful
as the reputation of an IP does't change by retrying with the same IP.  A few
unfortunately very large mail service providers are using 421 in banner for these
rejects.  It is non-sense and backfires, filling our queues by the terabyte and
so 45z would have an interoperability issue also.

Back to RFC 7372.

Should we choose not to implement code 550 in CONNECT, would we be
violating RFC 7372, and by reversal conclusion, per your argument also
RFC 5321?  I don’t think so.

(1) Other than changing the base RFC itself there have been only two options
so far to get it updated: (a) specifying an SMTP extension, or (b) registering
new extended reply codes with the IANA database.  RFC 7372 obviously isn’t
an SMTP extension that needs a new EHLO keyword; it does the latter and
nothing else.  That’s fine.  So no change to the base RFC beyond the  extended
reply codes.

(2) Nowhere it says that 550 should also apply to the banner stage directly or
indirectly.  Or do you mean form actually here also is function?  Do I have to
read the section heading as the precondition that leads to the extended code
and then to the “associated” base code?  So “associated” actually means
“mandatory”?  That’s interesting.

(3) Nowhere it says that 550 in the banner should have the same special
sequencing rule like 554 the only other policy code specified by RFC 5321.
If that doesn’t matter, why have it a MUST in RFC 5321?

By not making an interface specification clear you won’t get it implemented
the way it was intended.  You are risking interoperability issues and you are
weakening earlier RFCs and the overall system.  People again will be
inventing their own codes or use existing codes in whatever stage they see
fit _beyond_ RFCs with the exact single argument that is the golden rule of
the theory of reply codes.  That won’t do.  We’ve been there.  No, thanks.

I have talked a lot about reply codes;  I should have talked more about
interfaces and their meaning because the real issue isn’t even the reply code
itself, interoperability or not, but the way you handle that interface change!

I appreciate the efforts of RFC 7372.  It is a good RFC but it cannot be
understood per your intention.  We are implementing those RFCs,
we are operating the systems, we are the backbone to the net, we have
to know about interface changes that _might_ affect the critical parts of the
system so it keeps on ticking 24/7. 

Christoph

---

From: Ned Freed <[email-masked]>
Date: Sun, 21 Sep 2014 19:34:12 -0700 (PDT)

> ...

> code for a new purpose?

Of course it isn't. And wasn't. If you want to review the discussions leading
up to the publication of RFC 7372, you'll find the discussions leading up to
the publication of the draft in the appsawg archives.

> If a client in its protocol engine suddenly starts

But that's what RFC 5321 is about - and your complaint was about the failure to
use a code listed as an initial greeting response in RFC 5321. RFC 5321 is a
protocol specification, and aside from defining a few codes for use in
expressing policy failures, it doesn't discuss mail system policy matters at
all.

Now, you can argue that there should be such a document or documents - and I'd
probably agree - but there aren't. But doesn't mean you can take RFC 5321
as providing guidance in this regard, because it doesn't.

> matter and at the same time that clients won’t look at the last two

Perhaps because the two statements are entirely compatible and in no way
contradictory? I was talking about SMTP client handling of reply codes, which
is almost entirely based on the first digit. (Yes, I'm aware of the handful of
cases where it may not be.) Of course these codes have existence outside of
SMTP clients.

> Of
> without looking at reply codes directly or indirectly.

Gosh, that sure sounds awfully categorical. I guess the ones with million+ user
bases I'm familiar with that don't pay any attention don't count.

> Reply codes can be fed

So let me see if I have this straight. Right now systems often reject mail
because the client IP address doesn't have a PTR entry, or the value of the PTR
entry doesn't contain a valid domain name, or the forward translation of the
value doesn't produce an A/AAAA record matching the client IP, or the name that
was produced was on some sort of blacklist, and so on and so forth.

(Please note that I am only pointing the existence of such checks, not
condoning their use.)

Prior to RFC 7372 there was no enhanced status code defined for use when such a
check fails. It would be helpful to have a code to make it clear to the client
when this is what happened, so one was defined. As part of doing this, a
corresponding regular status code had to be selected, and given the definitions
of such codes in RFC 5321, 550 was the obvious choice.

Unfortunately, I now see I was remiss in not pointing out the fact that none of
this formally applies to the initial server greeting, for the simple reason
that RFC 2034 doesn't specify the use of enhanced status codes in that context.
(This is clearly stated in section 3, point (4).) I thought I noted this in my
original response, but I now see I didn't. My bad.

Even so, it's possible someone could read this text, and from it extrapolate
from it that 550 is the right code to use in a DNS policy rejection in the
initial server greeting, rather than using the clearly totally inappropriate
554 code because it's the only 5yz code RFC 5321 mentions in this context.

(Of course someone could easily arrive at exactly the same conclusion by
reading RFC 5321. I certainly do, and in my experience so do most of the
implementations of such checks that I've seen.)

And because of this some reputation system, which was previously fooled by the
use of a 554 code into thinking that a policy rejection was actually a problem
with the destination system ("no SMTP service here"), will now know the
rejection was done on the basis of policy and will act on that basis?

If I have this right, then I'm afraid I feel comfortable in concluding that
your concerns here are seriously overstated.

> support.

Actually, I'm well aware of the many stupid things people have done in the past
and no doubt will continue to do. But what I'm not seeing is how this
particular specification is going to have the dire effects you claim. In fact I
think it's far more likely to have the situation better, not worse.

>  There are weird and not so weird application of reply codes.

> As for facts rather than evidence, see above.

I saw it. That doesn't mean I believe a word of it.

> As for code 554: Am I the only one who doesn’t have that much an issue wit
> it?

Based on the other responses I've seen so far, yes, it seems you are.

> RFC 5321?  I don’t think so.

Correct, since RFC 7372 by its very nature doesn't apply to the initial
greeting. I would howver argue that on the basis of RFC 5321 alone the best
code to use in this case is 550, not 554.

> “mandatory”?  That’s interesting.

It doesn't say that because it's all about enhanced status codes, which don't
apply to the initial greeting.

> If that doesn’t matter, why have it a MUST in RFC 5321?

I don't know what MUST you're talking about. And frankly, given the hyperbolic
tone you have taken in this message, I don't really care.

And at this point I'm done - I will no longer be reading or responding
to subsequent messages on this topic. 

				Ned

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 22 Sep 2014 00:01:39 -0700

> as providing guidance in this regard, because it doesn't.

I would say the overriding prose is in Section 4.3.2, where implementers
are advised to anticipate new codes being registered, or codes not in the
sequences listed later in that section, to appear.  To me, that means the
appearance of 550 at CONNECT shouldn't cause anything to blow up even
though it's not explicitly listed.  In fact, that prose actually does
explicitly say that only the first digit matters when an implementation
sees a code it didn't expect.

I also note that RFC5248, which created the IANA registry of enhanced
status codes, says the "Associated Reply Code" is the one "usually"
associated with the status code being registered.  That does leave some
flexibility.

But as Ned points out, enhanced status codes don't have any meaning at
CONNECT, because their use at that point has not been negotiated, so this
is largely moot.

-MSK