Subject: Legal chars in 821bis

From: Jeffrey Streifling <[email-masked]>
Date: Fri, 26 Mar 1999 12:08:00 -0800 (PST)

I've hunted through the draft and through some
archived comments and it looks like the issue of
what to do with control characters in SMTP has yet
to be addressed.

Although there seems to be a leaning towards passing
lines (and messages) through unchanged, nobody has
defined a suitable representation for how control
characters should be presented to the user.  We
certainly don't want arbitrary control-characters sent
straight from the net to a user's terminal!
(Aren't the security reasons evident?)  It is probably
useful to except the TAB character from the following.

The good ol' SMTP of RFC 821 was theoretically implemented
on top of the TELNET protocol (although I have yet to
see an SMTP implementation that actually invokes a TELNET
program to communicate through).  The TELNET protocol defined
the behaviours of a few key control characters, told us we
had to support CR, LF, and NUL, and told us to strip everything
we didn't support.  (Go back and actually read RFC 854 p. 11)
The result of inputting lines using the TELNET protocol is
a string of printable USASCII characters; all the control
characters are eaten as a part of processing the protocol.

Now I'm all in favour of internationalizing the protocol to
use 8-bit characters or even UNICODE, but nobody has come up
with a useful representation for control characters (aside from
nonstandard stuff like is built into PC display hardware and
quoting tricks like ^A, ^B, etc).  Passing unfiltered control
characters through to mailboxes or other protocols is asking
for unpredictable results, if not trouble and security problems.

In my opinion, the new messaging specification ought to address
the issue of control characters in general.  In the name of
e-mail messages which are actually capable of being read by
physical human beings that actually exist, don't allow them
complete transparency through the messaging system.  Either the
transparency of the MTA should be well defined (the MTA is
specified to be transparent with respect to a particular list
of control characters) and the MUA given the responsibility of
handling those control characters in a standardized way (which
opens up a class of possible security holes in the MUA) or the
MTA should strip all control characters as a part of its line
input algorithm for compatibility with RFC 821 and RFC 854.

As long as all possibly useful messages can be expressed with
printable characters, CRLFs, and TABs, there's no reason to open
mailboxes, terminals, scripts regexps, and possibly buggy MUAs to
arbitrary binary data from possible attackers; on the other hand,
if somebody wants to standardize safe and useful semantics for
these characters, it should be put in the standard along with a
designation of the software component responsible for translating
them to something civilized.

(We all know about MIME and uuencode.  There has never been any
possibility of sending "real" binary data unquoted through the
mail system.  I'm not denying that it is useful to send arbitrary
binary data through the mail system, but it is always transformed
into something "safe" for transport.  Note that this binary data
is not sent to terminals or sent through the mail machinery to trip 
things up.)

Jeffrey Streifling
<[email-masked]>

---

From: Pete Resnick <[email-masked]>
Date: Fri, 26 Mar 1999 16:42:03 -0600

On 3/26/99 at 12:08 PM -0800, Jeffrey Streifling wrote:

>I've hunted through the draft and through some archived comments and 
>it looks like the issue of what to do with control characters in 
>SMTP has yet to be addressed.
>
>Although there seems to be a leaning towards passing lines (and 
>messages) through unchanged, nobody has defined a suitable 
>representation for how control characters should be presented to the 
>user.

SMTP is the transport protocol. It has nothing at all to do with 
presentation to the user. Therefore how to deal with control 
characters, unless it in some way impacts transport, is justifiably 
left out of this document.

On the other hand, the RFC 822 rewrite on the message format:

        <ftp://ftp.ietf.org/internet-drafts/draft-ietf-drums-msg-fmt-07.txt>

does address these issues. See section 3.5 and section 5.

The message format is where to properly deal with these issues, which 
it does. SMTP should not address this.

pr
-- 
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (619)651-4478, Fax: (619)651-1102

---

From: John C Klensin <[email-masked]>
Date: Mon, 29 Mar 1999 05:43:06 -0500

> what to do with control characters in SMTP has yet

So far, deliberately.

> straight from the net to a user's terminal!

This is not been discussed in DRUMS, but my sense of the network
consensus over the last couple of decades is that these issues 
-- in terms of what to display, how to display it, and what to 
filter entirely-- are matters for the receiving MUA or, if 
different, the user's display agent/ drivers, and not for the 
transport protocol.  

> we didn't support.  (Go back and actually read RFC 854 p. 11)

That text in 854 dates from the days of general belief in a 
strong NVT concept, including an NVT-specific restricted 
character set. The language you cite was, in fact, implemented 
and became a prohibition against sending non-NVT characters 
Informally, the "NVT character set" started out as "no controls
other than SP, HT, and CRLF" or pretty close to it, and the 854 
text tried to embody that rule.   (HT is questionable because 
different hosts assumed different virtual tab stops and "don't 
send them" eliminated a lot of display problems.)

A side effect of telnet option negotiation was that we got 
ourselves onto a slippery slope that eventually ended up with 
the combination of "binary" and "terminal type" which was the 
exact opposite of the intent behind NVT (and efforts like SUPDUP
and notions of a standardized network virtual graphics device). 
With those options in combination, the receiving system was 
essentially expected to understand and be able to interpret the 
data streams of any terminal supported by the sender.  The NVT/ 
SUPDUP theory was that the sender would send out a network- 
canonical stream, and the receiver would figure out how to 
render it into a form that made sense for the local system, 
terminals, and operating constraints. 

The question then becomes "if we are commonly using things like 
VT100 control/data streams instead of NVT over Telnet, and some 
documents have reinterpreted NVT as any ASCII character below 
0xFF, what is to be done with email?".   Unlike Telnet, it is not
possible to negotiate a terminal type on an end to end basis.  
An answer seems to be "send what is necessary and let the 
receiving system figure it out" (note nice symmetry between this
and the robustness principle).

> characters are eaten as a part of processing the protocol.

Yep, that was where we started, long, long, ago.  We could 
debate endlessly whether we gained or lost by effectively 
changing the rule (and I could take either side), but the war 
is, IMO, long over.

>...

e-mail messages haven't ever been readable by "physical human 
beings" without some translation process.   Bits on a wire are 
that way.   This argument is important for those for whom "cat" 
(and similar things) is an instance of an MUA, but we sort of 
gave those people up when we introduced MIME (and the specs were
never too sympathetic to them).

    john