Subject: Re: [ietf-smtp] Fwd: [dispatch] Forced SMTP redirects

From: Dave Crocker <[email-masked]>
Date: Tue, 16 Mar 2021 08:26:50 -0700

On 7/9/2020 9:01 PM, John C Klensin wrote:
> Also remember that SMTP encourages, nay requires, that if an
> SMTP-sender gets back a reply code it does not recognize, it is
> required to ignore anything but the first digit.  That predicts
> a fairly hard road at this point for new codes that are intended
> to convey subtle information.

Recent list discussion prompted me to look back for earlier reference to 
the first digit 'rule', concerning test in Section 4.2 of RFC 5321.

The above logic escapes me.

While it gives direction when the full code is not (yet) understood, it 
says nothing about adoption processes that might or might not eventually 
lead to support for new codes.

That is, new codes will of course not be understood by any 
implementation initially, but useful codes are likely to be understood 
by 'most' implementations eventually.

As such, the first digit rule, in RFC 5321, really has nothing at all to 
do with the efficacy of adding code or adding text to codes.  It only 
has to do with behavior of implementations that do not (yet) understand 
a full code.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 16 Mar 2021 17:26:49 -0200

> On Mar 16, 2021, at 1:26 PM, Dave Crocker <[email-masked]> wrote:
> 
> Recent list discussion prompted me to look back for earlier reference to the first digit 'rule', concerning test in Section 4.2 of RFC 5321.
> 
> While it gives direction when the full code is not (yet) understood, it says nothing about adoption processes that might or might not eventually lead to support for new codes.
> 
> That is, new codes will of course not be understood by any implementation initially, but useful codes are likely to be understood by 'most' implementations eventually.
> 
> As such, the first digit rule, in RFC 5321, really has nothing at all to do with the efficacy of adding code or adding text to codes.  It only has to do with behavior of implementations that do not (yet) understand a full code.

Postfix has been around since 1997 (alpha), or if you prefer 2001 (1.0 prod).
The SMTP client in Postfix only ever[1] looks at the first digit.  I expect this
is fairly common.  There isn't yet any known response code where we'd expect
the rest to matter.  I'd prefer to keep it that way.  The remaining digits are
useful for forensics.

-- 
	Viktor.

[1]  There is one special case, 421 in response to EHLO is taken as a no
     service indication, rather than a reason to try HELO instead.  This
     should probably include 521 (perhaps an oversight).

---

From: Dave Crocker <[email-masked]>
Date: Tue, 16 Mar 2021 12:32:58 -0700

On 3/16/2021 12:26 PM, Viktor Dukhovni wrote:
> There isn't yet any known response code where we'd expect

That's nice, but you won't be surprised to hear that there are other 
implementations that /do/ pay attention to more than the first digit.

d/

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 16 Mar 2021 17:52:09 -0200

> On Mar 16, 2021, at 5:32 PM, Dave Crocker <[email-masked]> wrote:
> 

Sure, but I worry that these are somewhat brittle, because with all
the manual anti-spam measures that users tweak into the rulesets of
production MTAs via lookup tables and milters, where the user gets to
specify the 4XX or 5XX response code, it is much to optimistic to 
expect them to get these right.

Were the (XX) in 4XX or 5XX in response to the usual MAIL transaction
commands (MAIL, RCPT, DATA, BDAT, ".") to actually matter, it would be
liable to get confused by all manner of misleading signals.

The non-transaction commands (AUTH, STARTTLS, ...) tend to see less
random meddling from milters.

-- 
	Viktor.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 16 Mar 2021 13:21:09 -0700

On 3/16/2021 12:52 PM, Viktor Dukhovni wrote:
> Sure, but I worry that these are somewhat brittle,

Well, we have decades of operational experience.

Where is documentation that demonstrates this brittleness?

d/

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 16 Mar 2021 19:47:14 -0200

> On Mar 16, 2021, at 6:21 PM, Dave Crocker <[email-masked]> wrote:
> 

Well, I rather expect that in fact most implementations don't
actually care about the 2nd and 3rd digit that much.  In particular,
for a negative 4XX or 5XX reply is not one of the expected ones from
5321 for that command, they'll do the standard thing for a temporary
or permanent error, and if it is, it'll still be a temporary or
permanent error, be one where it may be able to log something more
precise, ...  So distinguishing between ignoring those digits and
not ignoring them should be rather difficult.

-- 
	Viktor.

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 16 Mar 2021 19:10:59 -0400

Viktor Dukhovni writes:

> is fairly common.

Ditto for Courier. The relevant wording in 2?821 never meant anything else  
to me.

   4yz   Transient Negative Completion reply
      The command was not accepted, and the requested action did not
      occur.  However, the error condition is temporary and the action
      may be requested again.

   [ ... ]

   5yz   Permanent Negative Completion reply
      The command was not accepted and the requested action did not
      occur. The SMTP client is discouraged from repeating the exact
      request (in the same sequence).

This is followed by some discussion of the 2nd and the 3rd digit; but none of  
that ammends the concise definition stated above. So far, nobody was able to  
convince me that this means something other than its plain meaning.

I have some dim recollection of very, very few instances, over the last  
several decades, of parties who were (apparently) very much invested into  
having their mail delivered, trying to convince me that someone telling them  
5xx didn't really mean that, because of the specific error message that was  
assigned to the specific numeric code. I never found those arguments very  
convincing.

The most innovating argument revolved around the interpretation of "in the  
same sequence". The argument was that it only means "hey, you just can't  
resend it immediately, but if your RSET, or maybe disconnect or reconnect,  
its okay-dokay".

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 16 Mar 2021 23:04:54 -0200

> I have some dim recollection of very, very few instances, over the last several decades, of parties who were (apparently) very much invested into having their mail delivered, trying to convince me that someone telling them 5xx didn't really mean that, because of the specific error message that was assigned to the specific numeric code. I never found those arguments very convincing.
> 
> The most innovating argument revolved around the interpretation of "in the same sequence". The argument was that it only means "hey, you just can't resend it immediately, but if your RSET, or maybe disconnect or reconnect, its okay-dokay".

Postfix does give the MTA administrator some rope to hang himself with via
SMTP "reply filters" that allow transformations between arbitrary [45]xy
codes to other code (typically 5xx -> 4xx) allowing them opportunities
to retry some hard error with a problem peer, or perhaps fall back to
an backup relay.  That leaves any matching of codes and reply text to
the user...

... What dogs hear ... 

-- 
	Viktor.

---

From: "John Levine" <[email-masked]>
Date: 17 Mar 2021 20:32:33 -0400

>the rest to matter.  I'd prefer to keep it that way.  The remaining digits are

qmail which is about as old as postfix looks for 250 and 220 after
EHLO/HELO and STARTTLS but otherwise only looks at the first digit.

I'd flip it around.  Do we have examples of MTAs that look at the other digits
in responses to MAIL, RCPT, or DATA comands?

R's,
John

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 18 Mar 2021 10:55:20 +0100

> in responses to MAIL, RCPT, or DATA comands?

I searched github for "smtp 354" now, on the theory that that number won't 
be used for much other than the response code. Finding the first smtp 
client that expects 354 instead of 3 took maybe a whole minute. That 
package is a tool used by innumerable web sites to send mail. I expect I'd 
find many more if I were to spend a few more minutes looking.

While scrolling down I saw lots of unit tests that used 354, which makes me 
suspect that half of that software would break if fed 353 or 355 (and some 
would break in exciting ways).

Arnt

---

From: John C Klensin <[email-masked]>
Date: Thu, 18 Mar 2021 09:51:40 -0400

--On Thursday, March 18, 2021 10:55 +0100 Arnt Gulbrandsen
<[email-masked]> wrote:

> I searched github for "smtp 354" now, on the theory that that
> number won't be used for much other than the response code.
> Finding the first smtp client that expects 354 instead of 3
> took maybe a whole minute. That package is a tool used by
> innumerable web sites to send mail. I expect I'd find many
> more if I were to spend a few more minutes looking.
> 
> While scrolling down I saw lots of unit tests that used 354,
> which makes me suspect that half of that software would break
> if fed 353 or 355 (and some would break in exciting ways).

Especially given the above and thinking about 5321bis rather
than anything more immediate...

My own understanding of the "first digit" rule has always been
that the intent was that:

 * Servers should send the codes as specified (and try to
	avoid inventing their own).
	
 * Clients should read the three-digit code and take
	whatever action on then that they deem appropriate.  Of
	course, if that action is to ignore all but the first
	digit in some or all cases, that does not violate the
	spec.
	
 * _Only_ if a client reads a code and does not
	understand it (because it is not explicitly in the list
	specified by 821/2821/5321, standards track
	registrations, because it is a private extension or
	experiment, etc.) does the spec expect it to use the
	first digit rather than, e.g., treating the message as
	undeliverable.  Because 5321 says that a message can be
	rejected or not sent for any reason, "expect" is about
	the strongest statement that can be made.

Now, if there is agreement that is the intent and either I have
gotten it muddled or have failed to un-muddle the text of
others, I apologize.  And someone should take this to EMAILCORE,
ideally suggesting text but at least pointing to sections and
paragraphs that need work.

There are, I think, two unresolved question about the
interactions with extended reply codes.  One is what a client is
expected to do it receives an unrecognized code --either one in
the ranges used by SMTP/5321 or completely out of range (666
anyone?)-- followed by some digits and periods that might be an
extended code?  Even with the robustness principle. are such
strings  to be treated as extended codes or ignored on the
assumption that, with a problem reply code, the strings cannot
be assumed to be extended response codes.    The other is that,
if it were really the case that most MTAs are ignoring all but
the first digit, what are they transmitting back toward the
sender and does it include the extended codes and original text.
Fortunately, at least for me, both questions are far out of
scope for 5321bis.

    john

---

From: Paul Smith <[email-masked]>
Date: Thu, 18 Mar 2021 14:19:29 +0000

On 18/03/2021 13:51, John C Klensin wrote:
> extended code?

 From our SMTP client's point of view. An *extended* status code can be 
anything, but the basic status codes have to be 1xx to 5xx (xx can be 
any non-space character - we're quite forgiving). A response of "451 
8.5.1 whoopsie" would be treated as a response of 451 with text "8.2.1 
whoopsie"

An invalid basic status such as '666' would be treated as a generic 5xx 
response. I'm not really sure what else could be done with it.

We just use the first digit of the basic status code, and pretty much 
ignore the extended status code. There's the *option* of doing something 
with it, and the user (server administrator) can set it up to do 
something with it, but, as far as I am aware, no one ever does, as it's 
too fragile, and gives negligible benefit for our customers. (For mail 
servers handling more mail, then I wouldn't be surprised if they do 
something with the extra digits, but our software is aimed at SMEs, not 
at Gmail-wannabees.)

IME, the code details and extended codes are slightly useful for 
diagnostic and troubleshooting, but, not really, as most of them are 
only useful to the remote mail server "administrators", and in most 
cases those people haven't got a clue what would trigger the different 
codes.

> sender and does it include the extended codes and original text.

In SMTP responses we generate we use the 5321 response codes and 3463 
extended codes, or the nearest suitable ones (by our guesstimates, since 
the RFCs don't give values for all possible reasons)

In NDRs we generate we use the codes and text given by the remote 
server, whatever they are. Ours is not to reason why... One of my pet 
hates is mail servers which decide that ANY 452 code means "Requested 
action not taken: insufficient system storage" even when the original 
text is nothing like that. Yes, I have seen that happen, and it's a 
nightmare to diagnose through.

-- 
Paul
Paul Smith Computer Services
[email-masked] - 01484 855800

-- 

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

---

From: Laura Atkins <[email-masked]>
Date: Thu, 18 Mar 2021 14:55:22 +0000

> On 18 Mar 2021, at 14:19, Paul Smith <[email-masked]> wrote:
> 
> From our SMTP client's point of view. An *extended* status code can be anything, but the basic status codes have to be 1xx to 5xx (xx can be any non-space character - we're quite forgiving). A response of "451 8.5.1 whoopsie" would be treated as a response of 451 with text "8.2.1 whoopsie"
> 
> An invalid basic status such as '666' would be treated as a generic 5xx response. I'm not really sure what else could be done with it.
> 
> We just use the first digit of the basic status code, and pretty much ignore the extended status code. There's the *option* of doing something with it, and the user (server administrator) can set it up to do something with it, but, as far as I am aware, no one ever does, as it's too fragile, and gives negligible benefit for our customers. (For mail servers handling more mail, then I wouldn't be surprised if they do something with the extra digits, but our software is aimed at SMEs, not at Gmail-wannabees.)

MTAs that send high volumes of email do pay attention to the other digits and the extended codes. In fact, many of them do full text parsing to try and determine how they should handle future mail to that address. https://smtpfieldmanual.com is one example. 

> IME, the code details and extended codes are slightly useful for diagnostic and troubleshooting, but, not really, as most of them are only useful to the remote mail server "administrators", and in most cases those people haven't got a clue what would trigger the different codes.

The code details and extended codes are very useful for high volume senders trying to manage large bulk sends and their databases. They are parsed, and tracked and updated so that senders can follow what is happening with their sends. It is a challenge to programatically interpret the responses to manage future sends to an address. 

laura 

-- 
Having an Email Crisis?  We can help! 800 823-9674 

Laura Atkins
Word to the Wise
[email-masked]
(650) 437-0741		

Email Delivery Blog: https://wordtothewise.com/blog

---

From: Sam Varshavchik <[email-masked]>
Date: Thu, 18 Mar 2021 19:03:19 -0400

Paul Smith writes:

> We just use the first digit of the basic status code, and pretty much ignore  
> the extended status code.

The most common uses of extended status code appears to be in user-facing  
email systems. When their users' mail bounces the extended status code  
selects one of templates consisting of a long novel for a bounce message,  
decorated with pretty fonts and animated graphics, that boils down to  
"please don't contact our customer service/helpdesk about this, we can't do  
anything about it", that attempts to explain to the reader why the message  
cannot be delivered.

That's pretty much all that status code seems to be used for. When I'm  
implementing the logic for an SMTP client the only thing I need to know, for  
any given command's results is whether it succeeded, failed, or I should try  
again later. So, if it's '1', '2', or '3', the operation is deemed as a  
success, '4' means try again, and everything else means a failure. I've yet  
to find myself in any situation where any other outcome is possible.

---

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 19 Mar 2021 03:39:58 -0400

> On Mar 18, 2021, at 10:55 AM, Laura Atkins <[email-masked]> wrote:
> 

Is this "paying attention" really happening *during* delivery, to
guide further progress of the SMTP transaction, or, as seems more
likely, rather an after-the-fact forensic analysis to keep lists
in good hygiene.

Detailed SMTP server replies are certainly useful forensically,
what's under discussion in this thread, is to what extent the SMTP
protocol engine in the sending client really needs to be concerned
with such details.

Occasionally, an MTA with an uncertain reputation may choose to
conditionally soften a 5XX reply to a 4XX reply, if it has another
means to reach the destination (from a different IP address, ...).
Perhaps that's the every-day reality for some sending systems, and
in that case I can see these closely scrutinising the server reply.
In most other cases, there's little reason to look beyond the first
digit.

-- 
	Viktor.

---

From: Laura Atkins <[email-masked]>
Date: Fri, 19 Mar 2021 08:45:06 +0000

> On 19 Mar 2021, at 07:39, Viktor Dukhovni <[email-masked]> wrote:
> 

I believe both. For instance, there it at least one, and I believe multiple, MTA(s) that actively monitor the codes during delivery and change the speed / connections based on the current set of response codes. For instance, the VMG servers will start temp failing mail with a 452 [TSS04] and that is a signal that there is a reputation related limit on that particular mail stream. The MTA then adjusts, or even stops, sending for a programatically determined period of time. These are real time controls and they are already implemented, but in a non-standard way. 

> protocol engine in the sending client really needs to be concerned

There are MTA vendors that advertise things like “	Automatic backoff rules in case of delivery problems”. In many cases delivery problems are indicated by non-standard SMTP responses by the receiver indicating there is an issue with that set of deliveries.

> In most other cases, there's little reason to look beyond the first

This is not my experience dealing with the bulk end of the industry at all. They are heavily scrutinising and monitoring sends on a real time basis and that includes looking at all the full text of the SMTP response, not just the first digit. 

laura  

Laura Atkins
Word to the Wise
[email-masked]
(650) 437-0741

---

From: Ned Freed <[email-masked]>
Date: Fri, 19 Mar 2021 15:32:17 -0700 (PDT)

> > MTAs that send high volumes of email do pay attention to the other
> > digits and the extended codes. In fact, many of them do full text parsing to try
> > and determine how they should handle future mail to that address.
> > https://smtpfieldmanual.com is one example.

> likely, rather an after-the-fact forensic analysis to keep lists

Exactly. The problem with paying attention during delivery and modifying your
behavior acordingly more or less precludes pipelining. And since the vast
majority of deliveries are successful, the cost of doing this far
outweighs the benefits.

> protocol engine in the sending client really needs to be concerned

Well... not just forensically. A particular response may divulge the
fact that a rate limit has been hit, an IP address has suffered
a reputation or may not have an established reputation, and so on.

But regardless of how the information is used, your point is valid: This all
happens later, where later may be the very next transaction, the next session,
or even after some committee has met and reviewed the logs.

> means to reach the destination (from a different IP address, ...).

Quite right. And there are also a few cases where it makes sense to treat a
temporary error as permanent.

But this is all based on observation of server behavior. It's at a completely
different level than that of how an SMTP client is supposed to behave
during an SMtP session. 

> In most other cases, there's little reason to look beyond the first

And even when there is reason, let's not forget that the focus of RFC 5321bis is
and needs to be on basic client behavior. The supposedly "simple" mail transfer
protocol is sufficiently complicated in practice that just doing that is more
than enough for the document to cover.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Fri, 19 Mar 2021 15:47:38 -0700 (PDT)

> I believe both. For instance, there it at least one, and I believe multiple,
> MTA(s) that actively monitor the codes during delivery and change the speed /
> connections based on the current set of response codes.

Commonly called "backoff". (A term I dislike because it's the term we chose
years earlier for specifying retry intervals, and now our documentation has
become confusing as hell ;-)

> For instance, the VMG servers will start temp failing mail with a 452 [TSS04]
> and that is a signal that there is a reputation related limit on that particular
> mail stream. The MTA then adjusts, or even stops, sending for a programatically
> determined period of time. These are real time controls and they are already
> implemented, but in a non-standard way.

But it seems that you are now effectively agreeing with Viktor - altering your
behavior for subsequent transaction is a very very different thing than altering
it during a transaction. We absolutely implement the former but not the latter -
I can't think of a case where this sort of analysis causes a change in client
behavior.

> There are MTA vendors that advertise things like “	Automatic backoff rules
> in case of delivery problems”. In many cases delivery problems are indicated
by>  non-standard SMTP responses by the receiver indicating there is an issue with
th> at set of deliveries.

> This is not my experience dealing with the bulk end of the industry at all.
> They are heavily scrutinising and monitoring sends on a real time basis and that
> includes looking at all the full text of the SMTP response, not just the first
> digit.

Again, I think you are basically in agreement here. Unless I'm misinderstanding
something, Viktor is saying that such scrutiny does take place and is fine, but
when it happens it doesn't alter client behavior during the transaction. Nothing
you're said indicates this is happening, and given how badly it interacts with
pipelining I'm skeptical that high-end delivery systems would implement it.

				Ned

---

From: Laura Atkins <[email-masked]>
Date: Sat, 20 Mar 2021 11:20:13 +0000

> On 19 Mar 2021, at 22:47, Ned Freed <[email-masked]> wrote:
> 
> 
> 

Language is definitely an issue. SMTP-speak, deliverability-speak and email-marketing-speak all use the same words in different ways and setting the language parameters for any discussion is important. 

> I can't think of a case where this sort of analysis causes a change in client

You get a 452 for this transaction and you stop the transaction and then modify future transactions. The MTA developers can confirm that’s what they’re doing, but my understanding is this is what’s happening under the covers. 

> pipelining I'm skeptical that high-end delivery systems would implement it.

That’s my mental model of how it works, but I don’t know for sure. I’m hoping the folks writing the MTAs will comment here. 

laura 

Laura Atkins
Word to the Wise
[email-masked]
(650) 437-0741

---

From: Ned Freed <[email-masked]>
Date: Thu, 25 Mar 2021 16:25:01 -0700 (PDT)

> anything about it", that attempts to explain to the reader why the message

This is exactly they were intended to do: Provide detailed - and localized -
explanations of why an error occurred.

> to find myself in any situation where any other outcome is possible.

Use to classify faiures as "hard" or "soft" is quite common, as are any
number of more specialized (and mostly nonstandard) uses. AFAICR this
usage was never envisioned at the time they were standardized.

				Ned