Subject: Re: syntax errors

From: "D. J. Bernstein" <[email-masked]>
Date: 1 Nov 1998 20:42:20 -0000

Robert Elz writes:
> by being as rigorous as we can

That is unacceptably expensive.

If you think it's so easy, why don't _you_ write a ``rigorous'' server?
Show us your fantasy code that catches the flaws in tomorrow's clients.

---Dan
1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

---

From: "D. J. Bernstein" <[email-masked]>
Date: 1 Nov 1998 20:56:25 -0000

Robert Elz writes:
> So exactly why would be not have a problem is the
> standards weren't ``full of garbage'' ??

I said that TIME wouldn't be a problem FOR PEOPLE READING THE MAIL SPECS
if those specs weren't full of garbage.

There are always going to be people who don't read the specs at all, as
discussed in http://pobox.com/~djb/proto/design.html. That has to be
addressed during protocol design. It's too late to fix SMTP.

---Dan

---

From: [email-masked]
Date: Mon, 2 Nov 1998 02:52:44 +0200 (GMT)

> Show us your fantasy code that catches the flaws in tomorrow's clients.

	How so ?   Having the necessary machinery in place, I added
	5 lines of code to DATA/RSET/QUIT commands each, which previously
	just ignored any extra parameters.
	(If you want to try:  nic.funet.fi )

	Everything else has already been rigorous -- as rigorous as you can
	be in the M$ dominated world...

	In practice this means:
		- HELO/EHLO must be sloppy, bloody many PC users would
		  be unable to send anything otherwise, even many servers
		  place "interesting" stuff into the HELO parameter they
		  send out. In the logs I am referring at latter, there were:
			 300155 HELO
			 857474 EHLO
			1157629 total
		  If the HELO parameter processing were done strict, 38000
		  of those would be rejected (ok, 3.3% of total, but they
		  are all of HELOs, thus nearly 13% of all of those..)
		  (And mind you, these logs are from a server which our
		   corporate clients use, thus they should have good
		   software out there.  The truly bizarre misuses would
		   be gotten from servers that our ISP clients use, but
		   I don't care to do it this time..)

		- MAIL FROM/RCPT TO can be strict -- well, only WinCE
		  (versions prior to 3.0 ?) seems to be unable to send
		  these addresses with the mandatory "<"..">" characters.
		  Often also some extra spaces appear before and/or after
		  the leading "<", like also before the trailing ">".
		  (Usually from various windows clients, but I seem to
		   recall such happening even before windows existed..)

		- DATA can be strict.  From smtp-server session logs
		  covering 1.08 million DATA input sessions (two weeks):

1084405 DATA
      2 Data
      1 Data x
   2481 data
      2 data bytes. We send the first byte(address byte) with the parity set
      1 data, so some people have wound up back on this list who

		  Those last two lines are results from poorly
		  written SPAM-feed attempts. (Hit & run type of
		  operations, I guess.)

		  The third line is interesting, the "Data x" is
		  a test by somebody to see something, and it hasn't
		  repeated since.

		  In the same logs there were 17827 BDAT sessions
		  with 14712 "LAST" tags. ("CHUNKING")

		- RSET and QUIT should probably just ignore possible
		  extra junk.  In the logs nobody had used them with
		  extra trailing junk.

>1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

	So ?  I do the same with SMTP, with fully written down
	protocol specification.  All you need is smart way to
	use PIPELINING at the client side -- and of course the
	server must support it too.

	Originally I implemented the PIPELINING in my MTA to overcome
	throughput problems on long delay satellite links feeding
	large lists..  The bandwidth was high, but so was the latency,
	and latency really kills the basic half-duplex SMTP.

/Matti Aarnio	<[email-masked]>
		<[email-masked]>

---

From: "D. J. Bernstein" <[email-masked]>
Date: 2 Nov 1998 05:18:16 -0000

[email-masked] writes:
> Having the necessary machinery in place, I added
> 5 lines of code to DATA/RSET/QUIT commands each,

What exactly is the benefit of demanding that Eric Allman and dozens of
other implementors deploy this change?

What percentage of tomorrow's interoperability disasters will come from
clients sending RSET BLAH BLAH BLAH to unprepared servers?

You are _not_ being ``as rigorous as [you] can.'' There is a tremendous
difference between ONE syntax check and a COMPLETE syntax check. You are
catching only one out of a vast array of possible client flaws.

> > 1000 recipients, 28.8 modem, 10 seconds.
> I do the same with SMTP,

By using PIPELINING, by violating the 100-RCPT limit in RFC 821, and by
writing an order of magnitude more code, you have achieved results that
are still not as fast as mine. I am not impressed.

---Dan

---

From: [email-masked]
Date: Mon, 2 Nov 1998 20:12:56 +0200 (GMT)

<[email-masked]> writes:
...
> catching only one out of a vast array of possible client flaws.

  I am as rigorous as possible.  What I added were those tests I didn't
do previously.  The  MAIL FROM and RCPT TO scanners are already really
rigorous, and they are what really matters in the email world.

As we have learned, there are poorly written clients because the majority
of the world runs servers which are sloppy at the accepted input syntax.
(And we learned about WinCE because of my rigorous server..)

> are still not as fast as mine. I am not impressed.

Sure there is lots of code, as I get high performance out of slightly
augmented SMTP without writing a completely new protocol.  Depending
if you count support libraries into the client code, or not, it is 
anything from 4 to 15 percent of source lines.

Writing a PIPELINING client which becomes upset when talking against
plain RFC-821 server would be somewhat smaller, no doubt.

Here is a variant of what I do with that 100-RCPT limit:

"In the EHLO responses, the server reports how many recipients it
 accepts in single message transaction.  If this EHLO keyword is
 not present, clients must assume the RFC-821 given limit of 100.

   RCPTLIMIT nnn

 If the server gives there value less than 100, it must not get upset,
 when clients send up to 100 recipients.  If the server gives higher
 limit, then clients recognizing this mechanism will know, that they
 have permission to go until that limit."

This needs to be written into an RFC, I suppose.  But DRUMS isn't
adding these new functionalities into basic document.
(I use "X-RCPTLIMIT nnn")

---

From: "D. J. Bernstein" <[email-masked]>
Date: 2 Nov 1998 20:20:01 -0000

[email-masked] writes:
> I am as rigorous as possible.

False. Here are twenty examples of syntax flaws that you fail to catch:

   220 nic.funet.fi ZMailer Server 2.99.50-s10 #14 ...
   helo dude                         (extra space at the end)
   250 ...
   mail from:<>                      (extra space at the end)
   250 ...
   rset                              (extra space at the end)
   250 ...
   mail from:<>                      (extra tab at the end)
   250 ...
   rset                              (extra tab at the end)
   250 ...
   mail from: <>                     (extra space in the middle)
   250 ...
   helo dude                         (extra tab at the end)
   250 ...
   mail from:   <>                   (extra tab in the middle)
   250 ...
   helo    dude                      (tab in the middle)
   250 ...
   mail from:<[email-masked]>     (username too long)
   250 ...
   rcpt to:<postmaster>              (extra space at the end)
   250 ...
   rcpt to:<postmaster>              (extra tab at the end)
   250 ...
   rcpt to: <postmaster>             (extra space in the middle)
   250 ...
   rcpt to:        <postmaster>      (extra tab in the middle)
   250 ...
   rcpt to:<123...890@localhost>     (username too long)
   250 ...
   rcpt to:<postmaster>              (missing CR at the end)
   250 ...
   ehlo dude                         (extra space at the end)
   250 ...
   mail from:<...> size=00000000000000000000000001     (too long)
   250 ...
   mail from:<...>  size=18446744073709551615          (extra space)
   250 ...
   quit                              (extra space at the end)
   221 ...

In each case it is possible that we'll get an interoperability disaster
from a widely deployed client sending the invalid syntax to a future
server that can't handle it.

> (And we learned about WinCE because of my rigorous server..)

Windows CE had trouble with qmail too. That wasn't because I was being
``rigorous''; it was because my code started parsing after the first <.

> augmented SMTP without writing a completely new protocol.

The new protocol, in this case QMQP, is easier to implement than the
extensions that you're using, has a shorter spec, produces faster
results, and already has multiple independent implementations.

---Dan

---

From: Randall Gellens <[email-masked]>
Date: Wed, 4 Nov 1998 15:38:57 -0800

>100-RCPT limit in RFC 821

>mail from:<[email-masked]>     (username too long)

These limits are really the minimum limit that an implementation can 
impose.  Implementations are free to accept more recipients, or longer 
usernames.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 5 Nov 1998 00:35:40 -0000

Randall Gellens writes:
> Implementations are free to accept more recipients, or longer usernames.

Certainly. However, clients are not permitted to send them. See RFC 821,
section 4.5.3.

In fact, for obvious reasons, there have been far more interoperability
disasters involving long usernames than involving RSET BLAH BLAH BLAH.
Why isn't Elz demanding that all servers enforce the length limit?

---Dan

---

From: Robert Elz <[email-masked]>
Date: Thu, 05 Nov 1998 13:57:58 +1100

Date:        5 Nov 1998 00:35:40 -0000
    From:        "D. J. Bernstein" <[email-masked]>
    Message-ID:  <[email-masked]>

  | In fact, for obvious reasons, there have been far more interoperability
  | disasters involving long usernames than involving RSET BLAH BLAH BLAH.
  | Why isn't Elz demanding that all servers enforce the length limit?

If we were to write the spec to say "the maximum user name length is ..."
then I would be.   But that isn't what it says.

Nor am I demanding what you claim, what I want is for the spec to be
absolutely clear about what is and is not legal, and for servers to reject
what is not.

If you want to argue that "RSET BLAH BLAH BLAH" should be ignored by servers
(that is, treated equivalent to "RSET") then the spec can say that, I don't
mind (it isn't my preferred option, but it isn't important to me either).
What I don't want is to have the spec say "the server can issue an error when
it sees RSET BLAH, or it can treat it as RSET, whatever it likes".

kre