Subject: Re: DATA Pipelining
From: Sabahattin Gucukoglu <[email-masked]>
To: [email-masked]
Date: Sat, 26 Dec 2009 19:28:48 +0000
List-Id: <ietf-smtp.imc.org>

On 26 Dec 2009, at 14:59, Steve Atkins wrote:
> (Writing and testing a robust SMTP submission client is tedious, though, especially when you find you need to handle authentication. Pretty much every language has solid SMTP libraries available for it, so unless you have some unusual requirements you should take a look at libesmtp or Email::Send or whatever the common way of doing SMTP in your language of choice is).

Tcl, which is usually installed alongside the portable tcllib, has a superb SMTP implementation that supports everything and does it right (our very own Marshall Rose wrote it originally, of course).  Unfortunately, it expects to be used in practical conditions, where you will construct the MIME message inside the script, rather than feed it raw headers and body.  That makes it unsuitable for this little script, which I have now completed, and which now works, if only I could find out why news servers keep removing articles in the group range they advertise ...

If you want to see the world's smallest SMTP implementation for yourself (no extensions, assumes eight-bit-clean, and generally cuts every possible corner), I'll put the script up at:
http://get.sabahattin-gucukoglu.com/newsfetch
when done, or at any rate soon.

Cheers,
Sabahattin

Date: Sat, 26 Dec 2009 09:35:12 -0700
From: David MacQuigg <[email-masked]>
To: IETF SMTP <[email-masked]>
CC: Steve Atkins <[email-masked]>
Subject: Re: DATA Pipelining
Sender: [email-masked]

Steve Atkins wrote:
> On Dec 25, 2009, at 10:46 PM, Sabahattin Gucukoglu wrote:
>
>   
>> Hi all,
>>
>> Just writing a minimal implementation of SMTP here to serve a very specific need, that of gatewaying news articles, and have wondered how to handle the DATA phase with regard to pipelined text.  As I see it, there is no reason not to pipeline both the message text lines and the final dot, because nothing is expected from the server and until the dot is received there is no change of state other than the accumulation of the DATA buffer.  But I also know there are implementations out there which don't like it when a command is incomplete, and try to respond to whatever is in their TCP buffer, and wondered if anybody knew how that might work for the final dot if it were stuffed at the end of some text.
>>
>> Is there any specific guidance anywhere on this?  I tried RFC 2920 already.  I really don't want to send each individual line as a single send, and I'm pretty sure that's not necessary or even common.  I do know sendmail saves its final dot for a final send, though.
>>     
>
> You should be fine sending the entire payload as-is.
>
> (Writing and testing a robust SMTP submission client is tedious, though, especially when you find you need to handle authentication. Pretty much every language has solid SMTP libraries available for it, so unless you have some unusual requirements you should take a look at libesmtp or Email::Send or whatever the common way of doing SMTP in your language of choice is).
>   

If you haven't already chosen the language, I highly recommend Python.  
Python makes it easy to write scripts that handle complex operations, 
even if you are not expert in the underlying protocols.  It has a huge 
library of modules for SMTP, DNS, TCP, MIME, whatever you need.  See 
http://docs.python.org/3.1/library/

I might use smtplib, for example, if I need to send some data to another 
machine that is already running an SMTP server.  This allows you to work 
at the application level (helo, mail, rcpt, data, quit).  The data 
function appends an ending dot, so you don't have to worry about that 
detail.

If you need to control things at a lower level, there is a socket module 
that provides a low-level network interface (TCP commands, etc.).   
There is also an email module that takes care of formatting and RFC 
compliance issues.

-- Dave

Subject: Re: DATA Pipelining
From: Steve Atkins <[email-masked]>
Date: Sat, 26 Dec 2009 09:59:55 -0500
To: IETF SMTP <[email-masked]>
Sender: [email-masked]

On Dec 25, 2009, at 10:46 PM, Sabahattin Gucukoglu wrote:

> 
> Hi all,
> 
> Just writing a minimal implementation of SMTP here to serve a very specific need, that of gatewaying news articles, and have wondered how to handle the DATA phase with regard to pipelined text.  As I see it, there is no reason not to pipeline both the message text lines and the final dot, because nothing is expected from the server and until the dot is received there is no change of state other than the accumulation of the DATA buffer.  But I also know there are implementations out there which don't like it when a command is incomplete, and try to respond to whatever is in their TCP buffer, and wondered if anybody knew how that might work for the final dot if it were stuffed at the end of some text.
> 
> Is there any specific guidance anywhere on this?  I tried RFC 2920 already.  I really don't want to send each individual line as a single send, and I'm pretty sure that's not necessary or even common.  I do know sendmail saves its final dot for a final send, though.

You should be fine sending the entire payload as-is.

(Writing and testing a robust SMTP submission client is tedious, though, especially when you find you need to handle authentication. Pretty much every language has solid SMTP libraries available for it, so unless you have some unusual requirements you should take a look at libesmtp or Email::Send or whatever the common way of doing SMTP in your language of choice is).

Cheers,
  Steve

From: Sabahattin Gucukoglu <[email-masked]>
Subject: DATA Pipelining
Date: Sat, 26 Dec 2009 03:46:59 +0000
To: [email-masked]
Sender: [email-masked]

Hi all,

Just writing a minimal implementation of SMTP here to serve a very specific need, that of gatewaying news articles, and have wondered how to handle the DATA phase with regard to pipelined text.  As I see it, there is no reason not to pipeline both the message text lines and the final dot, because nothing is expected from the server and until the dot is received there is no change of state other than the accumulation of the DATA buffer.  But I also know there are implementations out there which don't like it when a command is incomplete, and try to respond to whatever is in their TCP buffer, and wondered if anybody knew how that might work for the final dot if it were stuffed at the end of some text.

Is there any specific guidance anywhere on this?  I tried RFC 2920 already.  I really don't want to send each individual line as a single send, and I'm pretty sure that's not necessary or even common.  I do know sendmail saves its final dot for a final send, though.

Cheers,
Sabahattin

Date: Thu, 10 Dec 2009 22:06:10 -0800
From: Douglas Otis <[email-masked]>
To: Hector Santos <[email-masked]>
CC: Alessandro Vesely <[email-masked]>, IETF SMTP <[email-masked]>
Subject: Re: Stranded postmasters
Sender: [email-masked]

On 12/5/09 12:30 PM, Hector Santos wrote:
>
> Alessandro Vesely wrote:
>
> Well, for me, it is very hard to continue "discussing DKIM" when it
> fundamentally has a known engineering implementation conflict
> (unauthorized remailer signatures not supporting ADSP) which not many
> are interested in fixing. If that is part of what you mean as "stranded"
> then I'm one of them. :)
>
> Specifically the DKIM deployment guide has one section discussing policy
> which addresses unauthorized signing threats and another section
> regarding remailers that effectively ignores the threats that policy
> attempts to address. Can't have it both ways. I specifically ask to fix
> the semantics. DKIM supportive Remailers MUST NOT ignore 1st party
> policy. It is fundamentally inconsistent to have a protocol designed to
> protect mail integrity and unauthorized signings, yet give have an
> exemption for remailers.
>
> See http://mipassoc.org/pipermail/ietf-dkim/2009q4/012648.html

I have responded to John Levine on the ASRG regarding an alternative to 
reputation methods that could be applied in a somewhat automated fashion 
that could be easily based upon DKIM signatures, either for the senders 
or for the feedback.

There is also a scalable and economical  authorization solution that 
could be used with little pain, since this would help the sender better 
ensure delivery of their message, without messing up who should receive 
the feedback.

See:
http://tools.ietf.org/html/draft-otis-dkim-tpa-label-03

-Doug

X-Mail-Sent-By-AEGEE.org-Account: didopalauzov
Date: Wed, 09 Dec 2009 22:46:11 +0100
From: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
To: Tony Finch <[email-masked]>
CC: Steve Atkins <[email-masked]>, IETF SMTP <[email-masked]>
Subject: Last line of the SMTP reply
Sender: [email-masked]

Hello,

In the thread "Re: Returning Details about Spam Evaluation when 
Rejecting Spam" on 27.11 was said by Steve Atkins and Tony Finch, that 
broken MTAs store in there logs only the last line of the rejection reason.

What does this practically mean?

Do the senders get the whole rejection reason and for what reasons MTA 
admins need to check the exact rejection text?

Kind regards,
   Дилян

Tony Finch wrote:
> On Thu, 26 Nov 2009, Steve Atkins wrote:
>> Some MTAs only make one line of the rejection available in their logs, or
>> don't make it easy to get the rest.
> 
> They're broken.
> 
>> If you're returning multiline responses, make sure that the first line has
>> the critical data, and use the remainder for additional info.
> 
> IME broken delivery failure reports include the last line of the SMTP reply.
> 
> Tony.

Date: Sat, 05 Dec 2009 15:30:34 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Alessandro Vesely <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Re: Stranded postmasters
Sender: [email-masked]

Alessandro Vesely wrote:

> 
> Ditto.
> 
> IETF standards convey the false impression that a matter has been 
> thoroughly analyzed and the protocols described are a definitive 
> solution. Actually, any establishment conveys that false impression. 
> And, today more than yesterday, there is a pervasive feeling that 
> exercising one's own judgment, initiative, and free will is not the best 
> option.
> 
> That DKIM deployment draft is avowedly pining for ideas. There is even a 
> figure where all arrows point to an "Identity Assessor". Yet, readers 
> may fail to consider it a call for such objects: piously self-assured 
> about IETF's soundness, they'll be at the mercy of the first one who'll 
> contrive something that can be sold by that name. (I've duped myself 
> like that several times, possibly in the hope to get the task done more 
> quickly than it would take to understand how to do it.)
> 

Well, for me, it is very hard to continue "discussing DKIM" when it 
fundamentally has a known engineering implementation conflict 
(unauthorized remailer signatures not supporting ADSP) which not many 
are interested in fixing. If that is part of what you mean as 
"stranded" then I'm one of them. :)

Specifically the DKIM deployment guide has one section discussing 
policy which addresses unauthorized signing threats and another 
section regarding remailers that effectively ignores the threats that 
policy attempts to address. Can't have it both ways.  I specifically 
ask to fix the semantics. DKIM supportive Remailers MUST NOT ignore 
1st party policy.  It is fundamentally inconsistent to have a protocol 
designed to protect mail integrity and unauthorized signings, yet give 
have an exemption for remailers.

See http://mipassoc.org/pipermail/ietf-dkim/2009q4/012648.html

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sat, 05 Dec 2009 16:42:40 +0100
From: Alessandro Vesely <[email-masked]>
To: Hector Santos <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Re: Stranded postmasters
Sender: [email-masked]

Hector Santos wrote:
>>>> Do RFCs contain more invitations to get out of the strand?

> Alessandro, I am not entirely sure what the purpose of this discussion.

Honestly, neither do I :-) I just felt the impulse to recap.

> But what is "Stranded?"

Run ashore, alone, clueless, and helplessly blocked.

> Your first comment seem to reflect complexity which my quip was 
> regarding "free" vs "commercial."  People paying for packages that 
> attempt to make it "easier." There is no concrete correlation but 
> commercial packages exist for a reason.

Yeah, I had been trapped too. But money can't buy discernment.

> Your second comment seem to 
> reflect a sense of apathy, boredom or that "everything is working." The 
> system is in autopilot, why break it? It may be a little of both.

Yes, everything is working, but not the way it should. That DKIM 
draft, at last, talks about "deterministic exchange of information". 
Just what we'd expect from a computer network, isn't it?

> But overall define postmaster? What is a "postmaster" today vs yesterday?

Good question. Unlike HTTP, SMTP was already an old protocol when 
Internet arrived to the masses. Quickly revamped by adding HTML and 
MIME. Then, as in a Hemingway's novel, sharks began to attack. 
Postmasters are like that old man. They administer their mail servers 
as if they were alone.

"Collaborative exchange", says the draft. Yet it gives no hint about 
how to enter a trusted environment, except for high volume whalers.

> There is also the factor that many people don't get involved until 
> necessary, i.e. a problem perhaps.  [...] Before that I put my trust 
> and faith in the engineers involved with IETF protocols.

Ditto.

IETF standards convey the false impression that a matter has been 
thoroughly analyzed and the protocols described are a definitive 
solution. Actually, any establishment conveys that false impression. 
And, today more than yesterday, there is a pervasive feeling that 
exercising one's own judgment, initiative, and free will is not the 
best option.

That DKIM deployment draft is avowedly pining for ideas. There is even 
a figure where all arrows point to an "Identity Assessor". Yet, 
readers may fail to consider it a call for such objects: piously 
self-assured about IETF's soundness, they'll be at the mercy of the 
first one who'll contrive something that can be sold by that name. 
(I've duped myself like that several times, possibly in the hope to 
get the task done more quickly than it would take to understand how to 
do it.)

Date: Sat, 05 Dec 2009 03:44:00 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Alessandro Vesely <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Re: Stranded postmasters
Sender: [email-masked]

Alessandro Vesely wrote:

>>> Do RFCs contain more invitations to get out of the strand?
>>
>> If you are referring to "stranded" as a result of complexity, this 
>> particular DKIM document is probably a good example of keeping them in 
>> the "strand."
> 
> I'm not sure whether it is the complexity of DKIM --or the mail system 
> as a whole, for that matter-- that refrains postmasters from 
> participating to discussions. Which discussions? Let me try and 
> enumerate their types, AFAIK:
> 
> * IETF or IRTF related mailing lists,
> * tool-specific *-user mailing lists,
> * technique-specific *-discuss mailing list,
> * DNSBL or rfc-compliance web sites,
> * provider-specific feedback-loop web sites,
> * generic web sites that also discuss mail issues.
> 
> It seems a lot of places. However, is the total number of subscribers 
> comparable to, say, 25,665,515 (the "Total domains checked" on 
> spf-all.com today)? I'd guess the percentage of "active" postmaster is 
> around 1~4%, and the rest is stranded "inertial mass."

Alessandro, I am not entirely sure what the purpose of this 
discussion.  But what is "Stranded?"

Your first comment seem to reflect complexity which my quip was 
regarding "free" vs "commercial."  People paying for packages that 
attempt to make it "easier." There is no concrete correlation but 
commercial packages exist for a reason. Your second comment seem to 
reflect a sense of apathy, boredom or that "everything is working." 
The system is in autopilot, why break it? It may be a little of both.

But overall define postmaster? What is a "postmaster" today vs yesterday?

Today, I will venture that the disciplines are merged, especially with 
the help of integrated packages, systems and automation. I will tell 
ya from our experience, the pay scale of those administrating many of 
our installations are lower than they use to be. System Operator 
(sysop) retirement is one of the principle factors with our ~25 years 
old package.  There are times where we have to "teach" SMTP, LIST 
SERVER or FTP because of the lower discipline levels.  I use the old 
idea that our customers are not "unix wienies" and don't edit 
configuration files or don't want to - they want GUI (and today, they 
want it via the WEB interface whether that is Online or Offline).

There is also the factor that many people don't get involved until 
necessary, i.e. a problem perhaps.  AVS was a big factor (no, the main 
factor) for me to get involved back 2003.  Before that I put my trust 
and faith in the engineers involved with IETF protocols.  I still do 
for the most part but no longer care to get involved in the the mental 
battles any more.  My own IETF WG experiences has not been a charm. 
For that, I think another reason is that there operators (vs 
developers) seem to overwhelm the groups here.  So perhaps thats 
another reason.

With SPF, I think you need to look at high value domains, i.e., 
companies, the fortune 1000, the banks, etc. Most have SPF domains.
SendMail has some good stats here:

      http://www.sendmail.org/dkim/survey

According to them, 90% of the Fortune 1000 have SPF domains. 99% of 
the US banks have SPF domains. The levels for DKIM are pathetically 
low to nil.  Now you have to ask why?  A bad survey?  Maybe.

But IMV, SPF was simpler to implement. You didn't need a project 
"TEAM" per se to get SPF started.  DKIM is too complex at many levels 
and automation is probably the only thing that can help it.  Does that 
mean "Stranded" sysops were somewhat interested but punted?  The only 
way our customers will get to use DKIM is if we offered it to them and 
we held their hand thru the setup, configuration and automation of it all.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 04 Dec 2009 08:55:23 +0100
From: Alessandro Vesely <[email-masked]>
To: Hector Santos <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Re: Stranded postmasters
Sender: [email-masked]

Hector Santos wrote:
>>> Being stranded is the result of SMTP having achieved its goal of 
>>> allowing delivery without requiring any prior arrangements. Shouldn't 
>>> it now be duty bound to bring them out of there?
>>
>> AFAIK, the only attempt toward that in
>> - 'DomainKeys Identified Mail (DKIM) Development, Deployment and
>>    Operations '
>>    <draft-ietf-dkim-deployment-09.txt> as an Informational RFC
>>
>> where it mentions a case ``in which the receiving site might wish to 
>> deliver problematic mail, rather than redirecting it, but also of 
>> course contacting the signing organization, seeking resolution of the 
>> problem.''
>>
>> Do RFCs contain more invitations to get out of the strand?
> 
> If you are referring to "stranded" as a result of complexity, this 
> particular DKIM document is probably a good example of keeping them in 
> the "strand."

I'm not sure whether it is the complexity of DKIM --or the mail 
system as a whole, for that matter-- that refrains postmasters from 
participating to discussions. Which discussions? Let me try and 
enumerate their types, AFAIK:

* IETF or IRTF related mailing lists,
* tool-specific *-user mailing lists,
* technique-specific *-discuss mailing list,
* DNSBL or rfc-compliance web sites,
* provider-specific feedback-loop web sites,
* generic web sites that also discuss mail issues.

It seems a lot of places. However, is the total number of 
subscribers comparable to, say, 25,665,515 (the "Total domains 
checked" on spf-all.com today)? I'd guess the percentage of "active" 
postmaster is around 1~4%, and the rest is stranded "inertial mass."

Date: Thu, 03 Dec 2009 12:32:32 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Alessandro Vesely <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Re: Stranded postmasters
Sender: [email-masked]

Alessandro Vesely wrote:

> 
> I wrote:
>>>> [postmasters are a category of stranded people...]
>>
>> Being stranded is the result of SMTP having achieved its goal of 
>> allowing delivery without requiring any prior arrangements. Shouldn't 
>> it now be duty bound to bring them out of there?
> 
> AFAIK, the only attempt toward that in
> - 'DomainKeys Identified Mail (DKIM) Development, Deployment and
>    Operations '
>    <draft-ietf-dkim-deployment-09.txt> as an Informational RFC
> 
> where it mentions a case ``in which the receiving site might wish to 
> deliver problematic mail, rather than redirecting it, but also of course 
> contacting the signing organization, seeking resolution of the problem.''
> 
> Do RFCs contain more invitations to get out of the strand?

If you are referring to "stranded" as a result of complexity, this 
particular DKIM document is probably a good example of keeping them in 
the "strand."

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 02 Dec 2009 10:01:06 +0100
From: Alessandro Vesely <[email-masked]>
To: IETF SMTP <[email-masked]>
Subject: Re: Stranded postmasters
Sender: [email-masked]

I wrote:
>>> [postmasters are a category of stranded people...]
> 
> Being stranded is the result of SMTP having achieved its goal of 
> allowing delivery without requiring any prior arrangements. Shouldn't it 
> now be duty bound to bring them out of there?

AFAIK, the only attempt toward that in
- 'DomainKeys Identified Mail (DKIM) Development, Deployment and
    Operations '
    <draft-ietf-dkim-deployment-09.txt> as an Informational RFC

where it mentions a case ``in which the receiving site might wish to 
deliver problematic mail, rather than redirecting it, but also of 
course contacting the signing organization, seeking resolution of the 
problem.''

Do RFCs contain more invitations to get out of the strand?

Date: Tue, 01 Dec 2009 13:45:59 +0100
From: Alessandro Vesely <[email-masked]>
To: Hector Santos <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Stranded postmasters (was: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

Hector Santos wrote:
>> In addition, postmasters are a category of stranded people...
> 
> They get what they pay for. :)

Being stranded is the result of SMTP having achieved its goal of 
allowing delivery without requiring any prior arrangements. Shouldn't 
it now be duty bound to bring them out of there?