Subject: [ietf-smtp] Stray <LF> in the middle of messages

From: Leo Gaspard <[email-masked]>
Date: Sat, 06 Jun 2020 19:06:29 +0200

Hello world,

I am in the process of writing an SMTP server, which obviously is going
to be the best of all SMTP servers ever written and that will ever be
written in our eon.

However, in the process of taking over the world, I am facing something
that surprises me.

I read, in RFC5321, §2.3.8, this paragraph:

> Lines consist of zero or more data characters terminated by the
> sequence ASCII character "CR" (hex value 0D) followed immediately by
> ASCII character "LF" (hex value 0A).  This termination sequence is
> denoted as <CRLF> in this document.  Conforming implementations MUST
> NOT recognize or generate any other character or character sequence
> as a line terminator.  Limits MAY be imposed on line lengths by
> servers (see Section 4).

Which appear to clearly indicate that <LF> is not a valid line
terminator.

However, I notice that every single time I have tried to use `netcat` to
send emails for demo purposes, it succeeded *without* sending <CRLF> and
by sending only <LF>. While `telnet` does appear to convert typed <LF>
into <CRLF>, it looks like (my version of) `netcat` does not. So most of
the SMTP servers I have met with appear to consider <LF> as a valid line
ending.

This, in most cases, is not a big deal, because <LF> is not a valid
character in SMTP commands, so saying that receiving an <LF> is
equivalent to receiving a <CRLF> is not that big a problem.

However, there is one case where the semantics is important: should one
escape the <LF>. sequence while in a DATA block?

I would guess that the fact that other SMTP servers appear to usually
accept <LF>.<LF> as a terminator indicates that <LF>. should be escaped
even though it is not strictly conforming with the RFC, but… I wanted to
have the opinion of other people on this, before diving too deep in the
implementation?

The following paragraph also makes me wonder:

> In addition, the appearance of "bare" "CR" or "LF" characters in text
> (i.e., either without the other) has a long history of causing
> problems in mail implementations and applications that use the mail
> system as a tool.  SMTP client implementations MUST NOT transmit
> these characters except when they are intended as line terminators
> and then MUST, as indicated above, transmit them only as a <CRLF>
> sequence.

Should I understand this paragraph as meaning that if I ever receive
such an ill-formed message, I… can? should? must? accept it and… can?
should? must? convert the <LF> into proper <CRLF>?

Thank you in advance for any thoughts you may have!
  Leo

PS: If you really want to, you can find the current code of said SMTP
server here, but it's still quite far from completion and I'm a hopeless
perfectionist, so don't hold your hopes too high even if the design
described in the book appeals to you — even though I'd be happy to hear
your thoughts about it too!

  https://github.com/Ekleog/yuubind

---

From: Dave Crocker <[email-masked]>
Date: Sat, 6 Jun 2020 10:22:29 -0700

> should? must? convert the <LF> into proper <CRLF>?

Encountering practice that differs from theory -- or, in this case, 
specification -- can be an excellent teachable moment.

I think your opportunity, here, is to appreciate the pragmatics of 
Postel's Law.  It is often taken to mean that one can send whatever one 
wants, but Jon was never that sloppy.  You should try to generate only 
according to the strictest interpretation of the spec.

But there are realities of different interpretations and some tolerable 
sloppiness that apply to this case, given a long history of systems that 
do use LF as line terminator and network software that fails to map it 
to the network standard CRLF.

I believe systems still vary on how they respond to an incoming LF.  I 
even suspect there are arguments for treating it only as data and 
arguments for treating it as newline.  I suspect, therefore, the answer 
to your question will be how you assess the benefits and detriments of 
each choice.

This being an IETF list, we should expect strong, differing comments 
from others.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Claus Assmann <[email-masked]>
Date: Sat, 6 Jun 2020 19:46:46 +0200

> send emails for demo purposes, it succeeded *without* sending <CRLF> and

Which MTA have you tried?  Did you try different servers/implementations?

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Jun 2020 14:15:49 -0400

Actually, the comment in his last paragraph notwithstanding and
other histories of disagreement, I completely agree with Dave.
I would add one additional cautionary note: we now have several
security-related tools, in difference degrees of active use,
that digitally sign message bodies, headers, or both.  If
something sees a bare LF and converts it after those signatures
are computed. testing them will typically fail.  That is just
one more reason for observing Dave's (and Postel's) advice: be
really careful that what you send (or allow to be sent) out
conforms to a narrow reading of the standard.  How much
deviation you want to accept in something you receive depends on
your circumstances.

    john

--On Saturday, June 6, 2020 10:22 -0700 Dave Crocker
<[email-masked]> wrote:

>>> In addition, the appearance of "bare" "CR" or "LF"
>>> characters in text (i.e., either without the other) has a
>>> long history of causing problems in mail implementations and
>>> applications that use the mail system as a tool.  SMTP
>>> client implementations MUST NOT transmit these characters
>>> except when they are intended as line terminators and then
>>> MUST, as indicated above, transmit them only as a <CRLF>
>>> sequence.
>> Should I understand this paragraph as meaning that if I ever
>> receive such an ill-formed message, I… can? should? must?
>> accept it and… can? should? must? convert the <LF> into
>> proper <CRLF>?
> 
> 
> 
> Encountering practice that differs from theory -- or, in this
> case, specification -- can be an excellent teachable moment.
> 
> I think your opportunity, here, is to appreciate the
> pragmatics of Postel's Law.  It is often taken to mean that
> one can send whatever one wants, but Jon was never that
> sloppy.  You should try to generate only according to the
> strictest interpretation of the spec.
> 
> But there are realities of different interpretations and some
> tolerable sloppiness that apply to this case, given a long
> history of systems that do use LF as line terminator and
> network software that fails to map it to the network standard
> CRLF.
> 
> I believe systems still vary on how they respond to an
> incoming LF.  I even suspect there are arguments for treating
> it only as data and arguments for treating it as newline.  I
> suspect, therefore, the answer to your question will be how
> you assess the benefits and detriments of each choice.
> 
> This being an IETF list, we should expect strong, differing
> comments from others.
> 
> d/

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sat, 06 Jun 2020 14:30:24 -0400

> should? must? convert the <LF> into proper <CRLF>?

To add to what Dave Crocker just said:

Unless you have insight into *why* you got a bare <LF> (for example,
you know *for a fact* that the other end of the connection is that ancient
creeping horror over in the Receiving department that has Known Bug XYZ),
there is in general no guaranteed "correct" way to deal with the situation.

Although smashing a bare <LF> into a <CRLF> is probably the most
expedient way to recover, that *does* have the (probably small) risk
of data corruption if the sending end actually *did* mean to have a bare
<LF> in the data.

However, it's 2020, and quoted-printable and friends are approaching
3 decades in age.  I'm fairly sure that for the vast majority of MTAs.
the code that accepts and converts <LF> is equally as old, and has
stayed in the code base just in case somebody is using netcat or similar
tool to talk to the receiving MTA.  I doubt there's anything out there
that claims to be an actual MTA that still generates bare <LF> anymore.

The scary part is, of course, what *ELSE* that creeping horror MTA might
be doing wrong.

The equally scary part is my "probably not anything out there" claim,
because I really *should* know better... :)

Back in the 90s, I had an AIX workstation at work that was advertised as
a open-access NTP server I was running on a volunteer basis. In 1998, our networking
group started a production NTP service for the enterprise, and I removed
the workstation from the official clocks.txt file.  In July 1999, I migrated from
AIX to a Linux box with a different hostname, and the AIX box was powered
down, the name removed from DNS, and the IP address flagged as "do not use"
in our management system (to prevent some poor soul getting the address and
going nuts when their firewall rules exploded).

Every few years, I'd play some ifconfig/iptables/tcpdump games to see
what inbound traffic was still there. As of July 2018, there were *still* well
over 500 separate IP addresses sending NTP traffic to that address, even
though nothing had actually answered packets for almost 2 decades.

And a lot of those hosts were sending NTPv4 packets, which implies they
had at least 1 software upgrade after I shut that service down....

---

From: Paul Smith <[email-masked]>
Date: Sat, 6 Jun 2020 19:36:19 +0100

On 06/06/2020 18:22, Dave Crocker wrote:
> arguments for treating it as newline.  I suspect, therefore, the 
> answer to your question will be how you assess the benefits and 
> detriments of each choice. 

 From personal experience - the mail server we publish *used* to treat 
either LF or CRLF as a line ending. That stopped over 20 years ago 
because of the unending problems it caused. Now LF is just a character 
line any other if it's not preceded by a CR (similarly with a lone CR)

The problem is that if someone sends a message with LF.LF, and you treat 
LF as a line ending, you've got a message terminator in the middle of a 
message, causing all sorts of grief.

You can't expect a previous MTA to have dot-padded it, because the RFCs 
don't say they should, so you end up with a situation where your mail 
server behaves counter to what the user expects, and the RFCs say its 
behaviour is wrong, so users are right to complain.

OTOH Treating ONLY CRLF as a line break is what the RFCs say, so it's 
perfectly defensible behaviour.

I suppose with a submission server you could be more flexible, but, to 
be honest, I've *never* come across a case where not treating a lone LF 
as a line break has been a problem.

(To be honest, I'd be tempted to treat a lone LF as a 99.9999% reliable 
indicator of spam. Similarly with a NULL (0x00) character in the middle 
of an (RFC5322) message. Legitimate mail will just never have it unless 
it was generated by something very dodgy).

-- 
Paul

-- 

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Jun 2020 16:03:40 -0400

--On Saturday, June 6, 2020 19:36 +0100 Paul Smith
<[email-masked]> wrote:

>... 
> I suppose with a submission server you could be more flexible,
> but, to be honest, I've *never* come across a case where not
> treating a lone LF as a line break has been a problem.
>...

And that is exactly why the last sentence of the introductory
paragraph of Section 8 of RFC 4409 says:

	"Any message forwarded or delivered by the MSA MUST
	conform to the requirements of [SMTP-MTA] and
	[MESSAGE-FORMAT]."

In other words, perfectly reasonable for a submission server to
reach a local agreement that bare LFs are to be treated as line
terminators but, by the time the message reaches the public
Internet, it needs to have fixed those.

    john

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sat, 06 Jun 2020 16:06:33 -0400

On Sat, 06 Jun 2020 14:15:49 -0400, John C Klensin said:
> are computed. testing them will typically fail.

I'm unaware of anything that does digital signatures that doesn't
already mandate the use of a canonical encoding that would prevent
a bare LF from escaping.  I suppose that somewhere, somebody wrote
a signature routine that was expecting canonical input and failed to
check for same and flag an error

On the other hand, the case can be made that causing the signature
to invalidate isn't an error - and possibly even rises to a 2119 SHOULD
fail.

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Jun 2020 16:09:13 -0400

Sorry, that should have been a reference to the next-to-last
paragraph of Section 8 of RFC 6409.  But the principle (and,
with the exception of handwaving about extensions, the text) are
unchanged.

  john

---

From: "John Levine" <[email-masked]>
Date: 6 Jun 2020 17:02:23 -0400

>either LF or CRLF as a line ending. That stopped over 20 years ago 

I think that's the key observation. Back in ye olden days there might
have been real mail systems that send bare LF but I haven't seen one
in a long time.

I also agree with the observation that defects in the SMTP protocol
are a very reliable indicator that you're talking to spamware. That's
why after all these years greylisting is still fairly effective. About
40% of the hosts that I greylist never retry.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Jun 2020 17:52:15 -0400

--On Saturday, June 6, 2020 16:06 -0400 Valdis Klētnieks
<[email-masked]> wrote:

>> I would add one additional cautionary note: we now have
>> several security-related tools, in difference degrees of
>> active use, that digitally sign message bodies, headers, or
>> both.  If something sees a bare LF and converts it after
>> those signatures are computed. testing them will typically
>> fail.
> 
> I'm unaware of anything that does digital signatures that
> doesn't already mandate the use of a canonical encoding that
> would prevent a bare LF from escaping.  I suppose that
> somewhere, somebody wrote a signature routine that was
> expecting canonical input and failed to check for same and
> flag an error
> 
> On the other hand, the case can be made that causing the
> signature to invalidate isn't an error - and possibly even
> rises to a 2119 SHOULD fail.

I think this gets into the range of kicking of dead horses, but
the point is that sending bare LFs (or bare CRs) is looking for
trouble, whether it is misinterpretation of intended

   foo
      bar

sequences, messing up of signatures, or being confused with a
spammer (and/or complete idiot who hasn't managed to fix those
problems in nearly 40 years).   If a submission server has
private agreements with its clients about those things and how
to fix them, so be it as long as they get fixed.  But, for an
SMTP sender (or submission server) to send that stuff out not
only violates the spec if they intended to have LF interpreted
as CRLF but counts on whatever happens downstream to apply the
same interpretation... and they probably won't.

   john

---

From: Hector Santos <[email-masked]>
Date: Mon, 08 Jun 2020 10:03:49 -0400

Hi Leo,

Here is quick summary for the three main operating systems. When it 
comes to the EOL (End Of Line) terminator of stored text files or 
messages:

o The *nix (unix-based) OSes use the <LF> (Linefeed) as an EOL terminator,

o The Apple Mac OS, use the <CR> (Carriage Return) as an EOL 
terminator, and

o The DOS/Windows OSes, used the <CR><LF> (Carriage Return, Linefeed) 
as the EOL terminator.

The Internet Mail format officially starting with RFC 822 selected the 
DOS/Windows format <CR><LF> format.

Why?

Maybe because at that point in time, Microsoft had owned 90% of the 
growing Personal Computer (PC) market.  The Mac was still considered 
(legally) a luxury commodity (otherwise their anti-trust status would 
no longer apply), and *nix was still mostly at the IT networking level.

But I think there may had been other technical reasons. Dave Crocker 
the editor of the RFC822, can maybe tell us if X.400 used the <CR><LF> 
format. I did work on X.400 mail when I worked at Big Circle W 
(Westinghouse) but I don't recall the format it used.

The point here, you need to keep the above differences in mind, when 
it comes to exchanging files or data in a heterogeneous network of 
three different text-based storage or transmission formats.

While the *nix or Mac may store the email in their native format, when 
it comes to SMTP transmission of the DATA payload , they MUST 
translate  it to a <CR><LF> format. In principle, all compliant SMTP 
senders and receiver MUST conform to the <CR><LF> end of line terminator.

It is great to see developers do their own thing, even "reinvent the 
world" rather than be reliant and dependent on other popular systems. 
It is a good way to learn.

Good Luck with your SMTP server project!

-- 
Hector Santos,
https://secure.santronics.com
https://twitter.com/hectorsantos

-- 
Hector Santos,
https://secure.santronics.com
https://twitter.com/hectorsantos

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 8 Jun 2020 18:22:47 +0200

On Mon 08/Jun/2020 16:03:49 +0200 Hector Santos wrote:
> While the *nix or Mac may store the email in their native format, when it comes
> to SMTP transmission of the DATA payload , they MUST translate  it to a
> <CR><LF> format. In principle, all compliant SMTP senders and receiver MUST
> conform to the <CR><LF> end of line terminator.
> 

CRLF is required for IMAP and POP too.  The inconvenience is having to compute
the length of the message, in octets.  The FS can only tell how many octets the
native format takes.  One has to add the number of lines.  Storing that info in
the file name is a possibility, if you don't have a dedicated FS.

Best
Ale
--

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Wed, 10 Jun 2020 06:34:10 -0400

> no longer apply), and *nix was still mostly at the IT networking level.

Unlikely.  When RFC822 came out in 1982, the IBM PC was a whole whopping 12
months old, even TCP/IP was equally young, the concept of a consumer ISP was
still a decade away, and most of the Arpanet/Milnet and CSNet were various
Unixoid systems, except for some oddball IBM mainframes and all the 36-bit
boxes running TOPS-10 and -20 and Multics, and you couldn't easily hook a
Windows box to an Ethernet because the Clarkson Packet drivers were like 5
years away, KA9Q was still 3 years away, and even MIT's PC/IP was out for only
6 months.

Oh, and the Macintosh wasn't a "luxury item" - it was non-existent, as the first
Macintosh was famously announced in a killer ad during the 1984 Super Bowl.

(When Clarkson got on the Arpanet in summer 1984, the 30 or so Sun and Unix
boxes added an entire 1% to the size of the Internet, which was only around
3,000 directly connected machines at the time. And I had to hear in details
what Russ Nelson thought of Microsoft's OS because he was writing the packet
drivers in the cubicle next to mine. :)

---

From: Keith Moore <[email-masked]>
Date: Wed, 10 Jun 2020 07:33:03 -0400

> no longer apply), and *nix was still mostly at the IT networking level.

The CRLF convention for transfer of Internet text files (in FTP, which 
was the first protocol used to send email on the ARPAnet), predates the 
existence of both PC-DOS and MacOS by several years.   I am guessing 
that FTP got its end-of-line convention for text files from TELNET.

Of the systems in use on the early ARPAnet, I have read that TOPS-10 
supported CRLF as line terminators.  I recall correctly from some of the 
early RFCs listing ARPAnet hosts (back when the ARPAnet was small) 
TOPS-10 systems were a bit more widely represented than others.   But I 
don't know how much this had to do with the selection of CRLF.   There 
were very many conventions in use for representation of text files, and 
some systems even supported multiple representations and even multiple 
sizes for characters (5, 6, 7, 8, or 9 bits) on the same system.

CRLF also may have had an advantage in that text files using that 
representation could be sent directly to an ASCII terminal (like a 
teletype or CRT) or printer without the need for translation.

Keith

---

From: Paul Smith <[email-masked]>
Date: Wed, 10 Jun 2020 14:10:40 +0100

On 10/06/2020 12:33, Keith Moore wrote:
> was the first protocol used to send email on the ARPAnet), predates 
> the existence of both PC-DOS and MacOS by several years.   I am 
> guessing that FTP got its end-of-line convention for text files from 
> TELNET.

Strictly speaking, CRLF is the "right" one to use given those control 
codes' definitions.

CR takes you back to the start of the current line, and LF takes you to 
the next line (at the current position), so to get to the start of the 
next line, you need either CRLF or LFCR (AIUI CRLF was chosen to give 
the print head more time to go back to the start of the line).

I remember sending CR (without LF) to a printer, and then reprinting the 
last line to make it print 'bold'.

-- 
Paul

--

---

From: Dave Crocker <[email-masked]>
Date: Wed, 10 Jun 2020 06:32:46 -0700

> DOS/Windows format <CR><LF> format.

Any interesting computer network seeks interoperability among 
heterogeneous systems.  All sorts of differences need to be dealt with, 
including differences in the end-of-line construct.

The simplest and most scalable way to accomodate such differences is to 
define a /single/ standard and require each system to between their 
local convention and the network's canonical convention.

This approach was as the core or the early Arpanet protocols, including 
FTP as already noted in this thread.  That's from the early 1970s 
timeframe.

RFC 822's predecessor, RFC 733, was 1977.  SMTP was 1982.  They merely 
continued the convention.

Telnet and FTP chose CRLF as the end of line construct.

d/

---

From: John C Klensin <[email-masked]>
Date: Wed, 10 Jun 2020 11:15:59 -0400

Paul, Keith,

A few small additions (no disagreement)...

--On Wednesday, June 10, 2020 14:10 +0100 Paul Smith
<[email-masked]> wrote:

>> The CRLF convention for transfer of Internet text files (in
>> FTP, which  was the first protocol used to send email on the
>> ARPAnet), predates  the existence of both PC-DOS and MacOS by
>> several years.   I am  guessing that FTP got its
>> end-of-line convention for text files from  TELNET.

It also, and it is probably relevant, predates Unix, certainly
any general availability of Unix, by some years.

> Strictly speaking, CRLF is the "right" one to use given those
> control codes' definitions.

And those definitions were very clear in the first version(s) of
ASCII, including the one against RFC 20 was written.   They got
somewhat fuzzier as assorted systems had conventions of their
own and the language of the standard was revised and adapted.
 
> CR takes you back to the start of the current line, and LF
> takes you to the next line (at the current position), so to
> get to the start of the next line, you need either CRLF or
> LFCR (AIUI CRLF was chosen to give the print head more time to
> go back to the start of the line).
> 
> I remember sending CR (without LF) to a printer, and then
> reprinting the last line to make it print 'bold'.

Also approximating certain Latin script diacritical and other
markings by printing the line or ASCII letters and then going
back and printing accent marks, tilde, "/", comma as an
approximation to cedilla, and so on over them.  That particular
mechanism was also the source of the non-destructive backspace
(about which the standard was also very clear even if common
practice differed).

   john

---

From: Dave Crocker <[email-masked]>
Date: Wed, 10 Jun 2020 08:24:28 -0700

> any general availability of Unix, by some years.

Because some nitpicking is just too easy:

      Unix was created in 1969, as was the first Arpanet operation.

      But use of CRLF as an Arpanet convention came a bit later. For 
example, it is not defined in RFC 20.

d/

---

From: "John Levine" <[email-masked]>
Date: 10 Jun 2020 14:12:47 -0400

In article <[email-masked]> you write:
>the next line (at the current position), ...

Except when it doesn't. In 1969 when people at Bell Labs were writing
the early versions of Unix, they had a lot of the otherwise obscure
Model 37 Teletype, which had upper and lower case and where the 012
character was a new line (NL), moving the print position to the
beginning of the next line. It had unusually sophisticated buffering
for the era and didn't need any delay characters. Teletype was an
AT&T subsidiary so it's not surprising that the Labs would get early
versions of their products.

Other places the models 33 and 35 were the most popular terminals,
both of which needed CR/LF to print properly. But as Dave noted, we
had to pick some line end convention and given how common CR/LF was at
the time, it was as good a convention as any.

Even the earliest Unix systems had tty drivers that could adapt to
model 33 or 35, adding CR in front of NL and providing escapes to
enter lower case letters and other characters not on the keyboard.

---

From: Ned Freed <[email-masked]>
Date: Wed, 10 Jun 2020 12:19:01 -0700 (PDT)

On the TOPS-10 side, the use of CRLF goes back to the PDP-6 Monitor OS. (Before
that I have no idea.) This takes us back to 1965. Except that it wasn't
precisely CRLF - in some cases you were allowed to omit the CR on the second
and subsequent blank lines.

I think this died out fairly quickly, but I recall seeing code to accomodate it
in some CUSP (commonly used system program) or other.

AFAIK Tenex copied TOPS-10 in this area.

				Ned

> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: Hector Santos <[email-masked]>
Date: Thu, 11 Jun 2020 09:02:54 -0400

Hi Keith,

> teletype or CRT) or printer without the need for translation.

+1.

My early software development career was working on teletypes, punch 
card machines, dumb, smarter, raster vs vector graphics capable 
terminals, printers, include lasers and plotters. I provided tools to 
make "life" easier for the corp, so I am well-verse with the 
input/output control designs.

<OPTIONAL_READING>

My burned-out memory is fading, but here goes:

You are correct ASCII control codes (0 to 31) were used for text-based 
devices to control the movement of the printer head or the terminal 
cursor position. Graphical capable terminals also had own set, 
usually, ESCAPE sequence based, to control their CURSOR/HEAD.

The 80x25 screen dimension with the 80 columns, was an emulation of 
punch card machines.  8x8 pixel cells were used hence why early 
terminals in graphic mode were 640x400 pixels dimension, 80x8=640, 
8x25=400. I believe the IBM terminals had different capabilities. I 
never programmed them.

With smarter graphical-capable terminals (switchable from raster vs 
vector graphics) allowed for 132 columns (in graphic mode) which was 
an extension for additional columns punch card emulation.

Programming Punch Card machines was basically mechanical for 
repeatability to reduce typing and auto push out the current and add 
the next card.

I have written ANSI/VT10x drivers/emulators, telecomm clients that 
worked with early hosting software. Over X.25, PCBOARD was among the 
first, if not the first, hosting app reprogrammed (upon my request) to 
support non-modem (RS232) profiles also called NULL modems to support 
EICON and X.25 with PAD (Packet Assembler/Disassembler) devices. They 
worked by waking up the host with a CR character or it raised the 
hardware carrier detection signal. PADs also had to be programmed to 
be in passthru mode and switched from software-based Flow Control to 
Hardware-based Flow Control. Otherwise binary file transfer would stop 
if there was a control S, ^S, characters in the payload that were not 
escaped.

In my early X.400 work, I worked a program ("Emailatator"), an MUA per 
se, to capture ASCII screen dumps of X.400 messages to print the 
messages for managers/engineers to read first thing in the morning 
with their coffee and donuts. People may recall when we used our 
office telephone as a data connection to some device (probably a PAD) 
that woke up the X.400 software.  That is why I didn't recall what 
X.400 used for its EOL because the host displayed the email line by 
line. The screen capture software did not see control characters, just 
the rendered screen output. With CompuServe, an early Offline Mail 
Readers (MUA) called TapCIS used the same primitive screen capturing 
method.  But on PCs, now the "Smarter" terminal now only with 
graphics, it has Memory and Storage now. When the backend host began 
to offer "add-on" backend hooks allowing Mail Packing software to be 
installed (on the backend) to allow a companion frontend apps (on the 
frontend, a MUA pe se) to pickup compressed mail packets, and also 
send back. Qmail Deluxe was the first to do this and Silver Xpress (my 
first commercial product) followed suit using the QWK and OPX mail 
format. Al Gore had yet to "invent the internet" but when he did, that 
was my first venture into the IETF when I added RFC822 support to the 
Silver Xpress MUA. The backend used UUCP/UUCICO as a store and forward 
mail gateway.

Today Wildcat! still supports VT10x/ANSI connection detection and 
emulations for the various text-based interfaces devices, example;

telnet winserver.com

It will know if you are human using an ANSI/TERMINAL or an automated 
mail data transfer app.  It also detected PPP, "mini-ppp" and the old 
RIP protocol (obsolete) which as an early competitor to HTML and of 
course, HTML won.  The proprietary Mini-PPP used a similar PPP 
detection designed to interface with a graphical Frontend.

The text-based terminals and client software, all offered 
configurations add either just send the CR or CRLF with the enter key. 
So I think the OSes used CR or CRLF as a EOL for that reason or LF for 
*nix although I don't see why a human would type a LF as a terminator 
when it takes two key (^J) but then again, today, many online 
messaging/comment logic don't use the enter key to submit the text. 
Oh boy, many will be in trouble is the ENTER key was the EOL. But 
technically, LF does not move the cursor or "file position" to the 
beginning of the next line, so for storage, why did *nix use LF for 
its EOL?

When I worked with text and laser printers, such as HP, OkiData 
printers, one of my early programs which started on a VAX then moved 
to the PC, was my LPRINT program to print text files with first page 
user name banner similar to the batch job print outputs we use to get. 
I still have lprint:

lprint V3.00/32 - Laser print utility by SSI
usage: lprint [file spec] <file spec>...<file spec> <options>

        infile   - any text file
        options:
         -H produce header with file name, page #
         -D print current system date on header (-D promotes -H)
         -L landscape print -B bold print, -C condense
         -B bold print
         -C condense print
         -X print preformatted files, like WORD. All other switches 
ignored.
         -PLxx set the page length
         -L6 set the 6 lines per inch (default, Auto sets PL to 55)
         -L8 set the 8 lines per inch. (Auto sets PL to 75
         -WW set word wrap on
         -TXxx expand tabs to XX blanks (default 8)
         -Ooutfile (default is LPT1. Overwritten by environment PRN 
string)
         -GB (Generate Banner Page, default None)
         -V  verify file to print (Ask you Y/N to print file)
         -DG DG laser printer HP Mode

sample: LPRINT source.c *.h *.mak -h -d -c

Lprint will format your file for text/laser printing. A banner page 
will be
printed with your file name, your user name, and date. The user name is
defined by having a SET ID= environment string (or USERNAME). Lprint also
looks for SET PRN= environment string if the output file is not defined.
Within the text file itself, you may embed the following keywords in
column one:

       //PAGE or /*PAGE*/          :  force pagination of your print.
       //BOLD or /*BOLD*/          :  force bold print.
       //CONDENSE or /*CONDENSE*/  :  force condense print.
       //NORMAL  or /*NORMAL*/     :  force normal print.

I recall using the built-in (default) BIOS character set to get the 
8x8 pixel cells per character and used that to do the large print 
banners of your name, i.e. HSANTOS, KMOORE, etc.

I also worked on FAX devices which used scan lines. It also used 8x8 
cells, at least early ones as I recalled programming one.

Anyway, as you can see, I was well verse with all these earlier 
terminal/printer HEAD/CURSOR control design considerations.

I did not why with CRLF was used for RFC822 but as you said FTP was an 
early start into transferring "mail".

But I have always believed (presumed) when the dumb terminals were 
used to program and test the early text-based client/server protocols 
like FTP, POP3, SMTP, NNTP, etc the developers was hitting enter (CR) 
key. It was all the human had to naturally do to end the line and send 
it to there to the backend host. Either the terminal was send the CR 
or optionally CRLF.  The backend could also echo back each character 
in full duplex mode including the CR and/or LF.

Microsoft Telnet has these two options:

o Local echo off
o Line feed mode - Causes return key to send CR

The line feed mode sends an CR because the backend (telnet server) is 
expected to echo back the LF. When the terminal is in Local Echo Off 
mode, it is expected of the  backend will echo back all characters.

Ironically, Legacy applications still thrive!  One my current projects 
is writing a cSharp (c#) SDK libraries for ANSI/VT100 and IBM Extended 
character emulation. 3rd party developers are writing C#-based 
applications are spawned by the Host to take over the COMM I/O with 
end users.  Many of them are writing games or had some text/andi/vt100 
service.

</OPTIONAL_READING>

---

From: Hector Santos <[email-masked]>
Date: Thu, 11 Jun 2020 09:37:13 -0400

On 6/10/2020 9:32 AM, Dave Crocker wrote:
>> The Internet Mail format officially starting with RFC 822 selected
>> the DOS/Windows format <CR><LF> format.
>>
>> Why?
>
>
> heterogeneous systems.  All sorts of differences need to be dealt
> with, including differences in the end-of-line construct.
>
> The simplest and most scalable way to accomodate such differences is
> to define a /single/ standard and require each system to between their
> This approach was as the core or the early Arpanet protocols,
> including FTP as already noted in this thread.  That's from the early
> 1970s timeframe.
>

+1.

But there has to be a more practical "ergonomic UI" reason and it 
probably started with how the terminal/client application worked 
and/or was configured to

- ECHO back characters.
- Send CRLF or CR when the Enter/Return key was hit
- Half or Full Duplex mode.

Half means you enter the entire line first locally. When the 
Enter/Return key were hit, it sent the entire line. This can be viewed 
like a "Batch Mode". Full meant each character was sent and processed 
one at a time, echoing back the character processed. This was akin to 
being Interactive and the early means of interactive spying on people 
by detecting each character, including what you began to type but 
decided not to hit enter/return. In your own mind, you believe the 
server was not watch or understand what you decide not to send.

These terminal option was very important and it had to fit with how 
the backend was expected to interact with the client.

So I always believe (and no sweat if I am wrong), the developers 
designing text-based client/server protocols most likely were 
designing and testing the protocol commands and state machine 
manually, online.  It explains the online HELP too. Who sees that? Who 
is it for? I suggest it was to assist the human, not the companion 
protocol automated client software.

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Fri, 12 Jun 2020 00:58:52 -0400

> probably started with how the terminal/client application worked

There *has* to be?  Why, exactly?

You're assuming facts not in evidence.  The fact that an ASR-33 (and by
extension, anything modeling one) needed both a CR and a LF to Do The Right
Thing was *quite* sufficient on its own merits to make that the default
end-of-line indicator for TELNET even when the network was still running NCP
and not IPv4, and there was never sufficient reason to change from that.

---

From: John C Klensin <[email-masked]>
Date: Fri, 12 Jun 2020 02:00:32 -0400

--On Wednesday, June 10, 2020 06:34 -0400 Valdis Klētnieks
<[email-masked]> wrote:

> (When Clarkson got on the Arpanet in summer 1984, the 30 or so
> Sun and Unix boxes added an entire 1% to the size of the
> Internet, which was only around 3,000 directly connected
> machines at the time. And I had to hear in details what Russ
> Nelson thought of Microsoft's OS because he was writing the
> packet drivers in the cubicle next to mine. :)

I can't remember the exact timing (Ned probably would), but
there were also VAX/VMS machines running native Internet stacks
and PDMF by that time.   VMS, IIR, preferred to use records with
length counts rather than any flavor of line terminators.    But
that doesn't change your conclusion about MSDOS having nothing
to do with the decisions -- as I think someone else pointed out,
CRLF was established as a convention for email messages with
mail-over-FTP, long before even 821/822.

    john