Subject: Re: Syntax extensions in RFC 821

From: Graham Klyne <[email-masked]>
Date: Sun, 07 Feb 1999 10:36:38 +0000

At 21:41 05/02/99 -0000, D. J. Bernstein wrote:
>Barry Leiba writes:
>> implementations that met the letter, but not the intent, of RFC821
>
>Jon Postel said that
>
>   * it was inexcusable to _generate_ syntax extensions but
>   * it was nice to _accept_ syntax extensions.

He also said, in RFC821 (Appendix E, p48):

           2yz   Positive Completion reply

               The requested
action has been successfully completed.  A
               new request may
be initiated.

I submit that sending arbitrary data when the server is expecting a new
request does not constitute a "syntax extension" so much as a "syntax
violation".

Also, in section 4.4, the state diagram clearly indicates that data may be
sent only in response to a 3yz response.  Thus, it seems clear to me that
sending data following receipt of a 2yz status code is clearly in violation
of RFC821.

Since you have provoked me into responding, I'll also pick up on an earlier
assertion you made:

>There's an installed base of MTAs delivering millions of messages a day
>through SMTP clients that treat 2xx just like 3xx.
>
>Declaring those MTAs non-compliant, and demanding that they be changed,
>means punishing the users.

This is a non sequitur.  Declaring the MTAs non-compliant imposes no
penalty on any user for whom their existing product delivers performance
that meets their needs.  If anyone is penalized, it is the developer who,
for marketing reasons, wishes to *claim* that their implementation is
compliant.

#g

------------
Graham Klyne
([email-masked])

---

From: "D. J. Bernstein" <[email-masked]>
Date: 7 Feb 1999 19:54:57 -0000

The theory of reply codes in RFC 821, Appendix E, states that the
command is ``accepted'' if the server returns code 1yz.

RFC 821, page 5, states that if DATA is ``accepted'' then the server
``considers all succeeding lines to be the message text.''

Therefore, if the server returns code 100 to DATA, then it considers all
succeeding lines to be the message text.

So why are clients not required to treat 100 the same way as 354?

Because RFC 821 doesn't allow this situation in the first place. RFC
821, page 5, says that if DATA is ``accepted'' then the server uses code
354. RFC 821, page 30, doesn't allow code 100 under any circumstances.

Graham Klyne writes:
> sent only in response to a 3yz response.

Wrong. Section 4.4 describes a ``simple-minded SMTP implementation.''
There is no requirement that all SMTP implementations be simple-minded.

---Dan

---

From: Robert Elz <[email-masked]>
Date: Mon, 08 Feb 1999 08:13:48 +1100

Date:        7 Feb 1999 19:54:57 -0000
    From:        "D. J. Bernstein" <[email-masked]>
    Message-ID:  <[email-masked]>

  | Therefore, if the server returns code 100 to DATA, then it considers all
  | succeeding lines to be the message text.
  | So why are clients not required to treat 100 the same way as 354?

If you were only making the argument that sending the data after
receiving a 1xx reply was an acceptable thing to do, then I doubt
I would argue (or not a lot).   I certainly would not be requiring
servers to do that, but I would not object a lot if one did, as were
a 1xx code ever permitted by 821, it would mean that more was expected
for the command which had been issued (though as I read it, the more
would always be more commands, but as it doesn't ever occur, who cares).

But arguing that sending data after a 2xx reply, which clearly means
"I have processed that command to completion successfully, send me
more (different) commands" is absurd, and cannot be considered acceptable
smtp client behaviour under any circumstances.

Further, now that Jon is no longer here to state his position for himself,
I consider it offensive in the extreme for you to place words into his
mouth.   From having worked with him for several years, I am fairly sure
that he would never have considered sending data in response to a 2xx reply
to be anything beyond dangerous in the extreme and not to be done.
Remember "be conservative in what you send" - we're talking about sending
here (sending the data).   Your earlier point about not necessarily
requiring exactly 354, but allowing any 3xx (I think you suggested 387)
was better taken, and I think has been accepted - that's more in line
with being liberal in what you accept (all 3xx codes mean "part done, send
me the rest").

Anyone who cared at all about not losing mail (not sending it to an unknown
abyss) would never even comtemplate sending the data afte getting a 2xx reply.
Sure enough the server that sends the 2xx is broken, but communicating with
a broken server is not an excuse for discarding messages, or for doing anything
which even permits that as a distant possibility.  If you're talking to a
server that you can detect is broken, far better to return the message to its
sender (or try a few times, and then return it).

  | Wrong. Section 4.4 describes a ``simple-minded SMTP implementation.''
  | There is no requirement that all SMTP implementations be simple-minded.

There is also no suggestion anywhere that a less than simple minded
implementation would treat the reply codes and their meanings with an
even more cavalier attitude than a simple minded one.   Only a less than
feeble minded implementation would ever countenance such a strategy.

kre

---

From: Eliot Lear <[email-masked]>
Date: Sun, 07 Feb 1999 14:53:06 -0800

I agree with Robert Elz.  It is not beyond the realm of possibility that
an SMTP server might lose sync (due to bugs and such).  How the client
handles this sort thing might be debated, but not being aware of it
would produce the most astonishment.
-- 
Eliot Lear
[email-masked]

---

From: "D. J. Bernstein" <[email-masked]>
Date: 8 Feb 1999 04:10:33 -0000

Eliot Lear writes:
> It is not beyond the realm of possibility that
> an SMTP server might lose sync (due to bugs and such).

Gee. What if all the 354s and 250s that you see are actually out-of-sync
repetitions of codes from earlier in the connection?

Maybe the server is _trying_ to reject all your commands! You might be
sending your message to the bit bucket! Yikes!

Aha, here's a brilliant solution: require that clients send at most one
message per connection! That way, if the server starts repeating codes,
then it can't send 354 in response to DATA, because there weren't any
previous 354s in the connection! And, even better, this solution doesn't
affect _my_ favorite client!

[end sarcasm]

This is not a sane way to design protocols. Blaming the client for the
server's loss of synchronization is crazy. It is not possible for the
client to singlehandedly work around such a bug; synchronization is not
some minor protocol feature that the client can stop using.

---Dan

---

From: Eliot Lear <[email-masked]>
Date: Sun, 07 Feb 1999 20:49:29 -0800

The problem, Dan, is that you are demanding relaxed behavior of servers
so that clients can cut corners.  That works just fine until a server
breaks down or cuts a corner of its own.  Then you turn around and say,
"[it's not fair to] blame the clients".  The protocol should be designed
so that errors are spotted and treated conservatively.  That's all I'm
saying.  No more no less.
-- 
Eliot Lear
[email-masked]

---

From: "D. J. Bernstein" <[email-masked]>
Date: 8 Feb 1999 06:40:36 -0000

> The problem, Dan, is that you are demanding relaxed behavior of servers

Wtf are you talking about? _I_ am not proposing changes in RFC 821.

> so that clients can cut corners.

These clients are full SMTP implementations. They are compatible with
every RFC 821-compliant server. They also go out of their way to handle
quite a bit of non-compliant garbage. They are widely deployed and work
just fine.

You want to punish the users for the sake of nonexistent, non-compliant,
buggy servers that everyone admits MUST NOT be deployed on the Internet.
This is ridiculous. Try thinking about how you'd explain it to the FTC.

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 8 Feb 1999 06:50:16 -0000

Robert Elz writes:
> mouth.

Katie Hafner and Matthew Lyon quote Jon Postel as writing

   It is nice that many mail-reading programs will accept mail that does
   not conform to the standard, but that does not justify mail-sending
   programs' violation of the standard.

As I said before, he understood the difference between

   * _accepting_ syntax extensions---``nice''---and
   * _sending_ syntax extensions---``violation of the standard.''

Why do you keep confusing these? They're completely different actions.

> here (sending the data).

According to RFC 1123, section 1.2.2, ``be conservative in what you
send'' means trying to avoid legal but obscure protocol features. That
has no relevance to blatant protocol violations by nonexistent servers.

---Dan

---

From: Robert Elz <[email-masked]>
Date: Mon, 08 Feb 1999 18:19:09 +1100

Date:        8 Feb 1999 06:50:16 -0000
    From:        "D. J. Bernstein" <[email-masked]>
    Message-ID:  <[email-masked]>

  | Katie Hafner and Matthew Lyon quote Jon Postel as writing

And that I fully believe.   That would have related to the mail content
itself though, not to the smtp transaction, where there is very little
room for variation.

  | As I said before, he understood the difference between
  |    * _accepting_ syntax extensions---``nice''---and
  |    * _sending_ syntax extensions---``violation of the standard.''

Yes.   And no, I am not confusing those.   You are if you believe that
taking notice of a 200 reply is "accepting a syntax extension".   A 200
reply to a DATA command (meaning "send me the data") could never be a
syntax extension, it would be a total redefinition of 821 reply codes.

One could imagine a syntax extension where 200 to a DATA command might
mean that the data had already been accepted, or wasn't needed, or similar.
In that case, sending the data would be fundamentally wrong.

Next, (to avoid sending unnecessary messages), in reply to Elliot Lear
you claimed that someone is wanting to punish some users or other.

Nonsense.    User's clients/servers will continue to work after 821bis
appears just as well (or not) as they do now.   No users will suffer.
Some implementors may not be able to simply claim (without doing any work)
"my server is 821bis conformant", but that is the only effect.   There is
no-one about to go racing about attempting to test anyone's servers for
821bis conformance and banish those which do not conform (any more than
there has for 821 conformance).

Lastly, no-one I know of admits that buggy servers MUST NOT be deployed on
the internet - if that were a requirement we would have had almost no
servers on the internet, and most likely still would have none.   There will
always be servers with all kinds of bugs.   As much as is reasonably possible,
clients must be perpared to deal with those bugs (just as servers ought to
deal with bugs in clients).   "Deal with" here means to operate in some
reasonable and defined way, with the primary aim of not losing mail.
Validating that the server is generating some reasonable reply code, and
treating it as either broken, or at least temporarily confused, is one obvious
way of achieving that.

In any case, it appears that there is no-one else here who has any sympathy at
all with your view - and it doesn't appear that you're changing anyone's
mind.   Give up.

kre

---

From: "D. J. Bernstein" <[email-masked]>
Date: 8 Feb 1999 09:01:26 -0000

Robert Elz writes:
> There is no-one about to go racing about attempting to test anyone's
> servers for 821bis conformance and banish those which do not conform

I wish I could believe you. How much will you pay if you're wrong?

> the internet

You are misquoting me. My statement was about servers that lose sync.
That's a disastrous bug; blaming the client for it would be idiotic.

---Dan

---

From: Brad Knowles <[email-masked]>
Date: Mon, 8 Feb 1999 12:52:35 +0100

>appears just as well (or not) as they do now.   No users will suffer.

    Correct.  People buy products, not standards.  Those products may (or
may not) implement a standard, but the standard itself is not a tangible
object which can be obtained.  As such, when the standard is changed,
there are no products which immediately start failing over night.

    If we were to redefine the SCSI standard so that hard disk drives
were required to play "Fur Elise" every time they started up, then I
doubt that all the disk drives across the entire world would suddenly
stop working the moment such a silly standard were officially ratified.

    By defining the standard more tightly, we increase the probability
that future SMTP clients and servers will correctly interoperate with
each other, and that current SMTP clients and servers will become more
interoperable with each other as development on them continues.

    Old SMTP clients and servers, on which no further development will be
done, may or may not slowly become less and less interoperable with
future clients and servers, as the standards continue to evolve -- that's
just how life happens.  However, we are *not* required to retain 100%
backwards bug-compatibility with every single SMTP client or server out there.

    IMO, our only goal should be to make the standard more clear and
explicit in what, where, why, how, and when things MUST/MUST NOT,
SHOULD/SHOULD NOT, and MAY/MAY NOT be done.

    If we've done our jobs properly, it should be physically impossible
for there to be multiple compliant but non-interoperable implementations.
 Of course, we'll never arrive at this "Holy Grail" of standards, but the
real goal is the journey, not the destination.

-- 
  These are my opinions -- not to be taken as official Skynet policy
 ____________________________________________________________________
|o| Brad Knowles, <[email-masked]>            Belgacom Skynet NV/SA |o|
|o| Systems Architect, News/mail/FTP Admin   Rue Col. Bourg, 124   |o|
|o| Phone/Fax: +32-2-706.11.11/12.49         B-1140 Brussels       |o|
|o| http://www.skynet.be                     Belgium               |o|
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
                The Sky is no longer the limit

---

From: "D. J. Bernstein" <[email-masked]>
Date: 8 Feb 1999 17:27:03 -0000

Brad Knowles writes:
> each other,

There is no interoperability excuse for requiring that clients barf on
an imaginary, non-compliant, disastrously buggy response to DATA.

---Dan

---

From: Robert Elz <[email-masked]>
Date: Tue, 09 Feb 1999 04:14:05 +1100

Date:        8 Feb 1999 09:01:26 -0000
    From:        "D. J. Bernstein" <[email-masked]>
    Message-ID:  <[email-masked]>

  | That's a disastrous bug; blaming the client for it would be idiotic.

No-one plans on blaming the client for any server bugs.   However
where it is reasonably possible for a client to determine that a
server has broken (by losing sync, or any other way) I consider it
a client bug to fail to do so.   Obviously it is possible for servers
to break in ways that are undetectable to the client, there's nothing
the client can do about that.   But when the client can detect it,
easily, then it ought.

When the code to do that is

	if (reply_type == '3') {
		/* all normal case code */
	}
	/* handle various error cases here */

rather than

	if (reply_type <= '3') {
		/* only '3' is legal but I am too lazy to check it */
		/* normal case here */
	}
	/* error cases here */

then I think it is entirely reasonable to expect the client
to detect that particular kind of breakage.

Also, note that if the client is attempting to be more than simple
minded, (eg: implementing smtp command pipelining) it is also entirely
possible that it is the client which loses sync, rather than the server.

kre

---

From: Eliot Lear <[email-masked]>
Date: Mon, 08 Feb 1999 16:05:09 -0800

If you honestly believe this then you completely misunderstand what it
is that Jon Postel was saying.  Look back fifteen years when just about
every implementation was buggy beyond words (or so it seems now).  The
whole point was that things *will* break and you *better* deal.

Your argument is that when the servers break the clients should BLAME. 
Let's see how fast that gets the mail through.

> an imaginary, non-compliant, disastrously buggy response to DATA.

By the way, Dan, your mail is bouncing.
--
Eliot Lear
[email-masked]

---

From: "D. J. Bernstein" <[email-masked]>
Date: 9 Feb 1999 06:53:24 -0000

> Your argument is that when the servers break the clients should BLAME. 

My SMTP clients work around dozens of common server bugs. My SMTP
servers work around dozens of common client bugs. These implementations
deliver mail to millions of people every day.

My web pages at http://pobox.com/~djb/proto/smtp.html contain the most
comprehensive available list of current SMTP interoperability problems.
I've repeatedly asked for such lists to be added to 821bis and 822bis.

This doesn't mean that I'm going to change my clients to uselessly barf
on an imaginary, non-compliant, disastrously buggy response to DATA.

> Let's see how fast that gets the mail through.

Stupid argument. Barfing isn't going to get the mail through either.

> The whole point was that things *will* break and you *better* deal.

That depends. Sometimes the benefits outweigh the costs; sometimes they
don't. I do engineering, not religion.

> By the way, Dan, your mail is bouncing.

See http://pobox.com/~djb/docs/copies.html.

---Dan