Subject: Re: 400 defer
From: Keith Moore <[email-masked]>
To: Hector <[email-masked]>
Cc: [email-masked]
Date: Wed, 28 Sep 2011 09:05:45 -0400
List-Id: <ietf-smtp.imc.org>

You're making this way too difficult.

If a reply code starts with 4, it's a temporary failure.  The client can retry later.

If the client happens to know what the full code means, a more specific action might be appropriate.  But if it doesn't, it should suffice to just treat it as a temporary failure.

That's the whole point of having the first digit indicate whether the request was successful, a temporary failure, or a permanent failure.

The "x0z" paragraph you quoted was intended as a description of how codes are assigned, not as advice for how to interpret unknown codes.

And in general, it's a Bad Idea to work too hard to accommodate the quirks of obscure SMTP servers.  That way lies madness.

Keith

On Sep 27, 2011, at 5:51 PM, Hector wrote:

> 
> A customer was sending mail to a remote host and got this for a RCPT TO response and wanted to know what it meant:
> 
> 17:25:47.884 C: RCPT TO:<[email-masked]>
> 17:25:47.962 S: 400 defer
> 
> Because of the 4yz response, our MTA queued the message for a retry, but I never seen this 400 and not sure what it suppose to mean.  5321 says:
> 
>   x0z  Syntax: These replies refer to syntax errors, syntactically
>      correct commands that do not fit any functional category, and
>      unimplemented or superfluous commands.
> 
> and it doesn't appear the address was incorrect.
> 
> I looked for the word "defer" and the only thing that "came close" was in section 7.2
> 
>   Whether disabling VRFY provides any real marginal security depends on
>   a series of other conditions.  In many cases, RCPT commands can be
>   used to obtain the same information about address validity.  On the
>   other hand, especially in situations where determination of address
>   validity for RCPT commands is deferred until after the DATA command
>   is received, RCPT may return no information at all, while VRFY is
>   expected to make a serious attempt to determine validity before
>   generating a response code (see discussion above).
> 
> What would make of this "400 defer" response?
> 
> TIA
> 

Date: Wed, 28 Sep 2011 11:35:37 +0200
From: Arnt Gulbrandsen <[email-masked]>
Organization: Me, http://arnt.gulbrandsen.priv.no
To: "Carl S. Gutekunst" <[email-masked]>
Cc: Hector <[email-masked]>, [email-masked]
Subject: Re: 400 defer
Sender: [email-masked]

On 09/28/2011 04:52 AM, Carl S. Gutekunst wrote:
> ... I can't imagine how a syntax error could be a 4xx error, since sending
> it again isn't going to fix the problem.  ...

Well, the server is written in java, isn't all software written in 
java?, the 4xx raises an exception, which triggers a page to the oncall 
engineer, and a fix is in place before the SMTP client gives up on retrying.

Insert smileys according to taste.

Arnt

Date: Tue, 27 Sep 2011 19:52:26 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector <[email-masked]>
CC: [email-masked]
Subject: Re: 400 defer
Sender: [email-masked]

Hector wrote:
> Basically, never seen it use before (AFAIR) and wanted to know what 
> others thought, seen or knew something about it.  It turned out 
> something like a "greylisting like" behavior. A subsequent 4th retry 
> (15 mins, 15 mins, 30 mins) did successfully send it.
>
> The only thing that crossed my mind (after seeing this) was whether 
> using 40z was correct for a retry (without change) and the x0z 
> contradicting it (i.e. syntax error).  It seems 40z should never apply.

I'd venture to guess that it was the class project of a random lab rat 
in the campus data center who wouldn't have recognized a copy of RFC 
2821 if it ran up and bit him in the patuckus.

(I can't imagine how a syntax error could be a 4xx error, since sending 
it again isn't going to fix the problem. That said, I'm reminded of the 
old joke about the sales rep, the hardware engineer, the software 
engineer, and the car with a flat tire....)

<csg>

Date: Tue, 27 Sep 2011 22:03:28 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: 400 defer
Sender: [email-masked]

Basically, never seen it use before (AFAIR) and wanted to know what 
others thought, seen or knew something about it.  It turned out 
something like a "greylisting like" behavior. A subsequent 4th retry 
(15 mins, 15 mins, 30 mins) did successfully send it.

The only thing that crossed my mind (after seeing this) was whether 
using 40z was correct for a retry (without change) and the x0z 
contradicting it (i.e. syntax error).  It seems 40z should never apply.

John C Klensin wrote:
> 
> 
> Hector, could you clarify what you wanted to find out from this
> list?

Date: Tue, 27 Sep 2011 18:51:41 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector <[email-masked]>
CC: [email-masked]
Subject: Re: 400 defer
Sender: [email-masked]

Hector wrote:
>
> A customer was sending mail to a remote host and got this for a RCPT 
> TO response and wanted to know what it meant:
>
> 17:25:47.884 C: RCPT TO:<[email-masked]>
> 17:25:47.962 S: 400 defer

Googling around, it appears this response is given by the graylisting 
service used by Texas A&M, tamu.edu. I found forum posts with deep links 
to pages allegedly describing it, but the links all timed out or got 404 
errors. Still, it's something to start with.

<csg>

Date: Tue, 27 Sep 2011 19:24:17 -0400
From: John C Klensin <[email-masked]>
To: SM <[email-masked]>, Hector <[email-masked]>
cc: [email-masked]
Subject: Re: 400 defer
Sender: [email-masked]

--On Tuesday, September 27, 2011 15:51 -0700 SM
<[email-masked]> wrote:

> 
> Hi Hector,
> At 14:51 27-09-2011, Hector wrote:
>> What would make of this "400 defer" response?
> 
> It's a temporary failure.  Your MTA can retry delivery.  Some
> people are creative with status codes.

Hi SM,

I am quite certain Hector knows that and knew that long before
seeing this particular code.  I'd guess --but it is just a
guess-- that failure to use a standard (i.e., specifically
listed in 5321) code is part of some greylisting scheme,
possibly to avoid using any of those standard codes ("I'm going
to make you try again to see if you are serious about getting
the message through" is definitely not on the 5321 list) and
possibly as an intended trap for the would-be spammer who wasn't
smart enough to read and understand the text in 5321 that
amounts to "if you are a client/ sender and don't recognize the
code, only the first digit counts".

I imagine Hector's real question was whether anyone else had
seen this particular use of 400 before and if it indicates
greylisting or a server that was really, really, confused.   If
that was the question, I, for one, haven't seen it, but I
discontinued surveying the reply codes received by my server
some years ago.

Hector, could you clarify what you wanted to find out from this
list?

  john

Date: Tue, 27 Sep 2011 15:51:21 -0700
To: Hector <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 400 defer
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 14:51 27-09-2011, Hector wrote:
>What would make of this "400 defer" response?

It's a temporary failure.  Your MTA can retry delivery.  Some people 
are creative with status codes.

Regards,
-sm 

Date: Tue, 27 Sep 2011 17:51:21 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: 400 defer
Sender: [email-masked]

A customer was sending mail to a remote host and got this for a RCPT 
TO response and wanted to know what it meant:

17:25:47.884 C: RCPT TO:<[email-masked]>
17:25:47.962 S: 400 defer

Because of the 4yz response, our MTA queued the message for a retry, 
but I never seen this 400 and not sure what it suppose to mean.  5321 
says:

    x0z  Syntax: These replies refer to syntax errors, syntactically
       correct commands that do not fit any functional category, and
       unimplemented or superfluous commands.

and it doesn't appear the address was incorrect.

I looked for the word "defer" and the only thing that "came close" was 
in section 7.2

    Whether disabling VRFY provides any real marginal security depends on
    a series of other conditions.  In many cases, RCPT commands can be
    used to obtain the same information about address validity.  On the
    other hand, especially in situations where determination of address
    validity for RCPT commands is deferred until after the DATA command
    is received, RCPT may return no information at all, while VRFY is
    expected to make a serious attempt to determine validity before
    generating a response code (see discussion above).

What would make of this "400 defer" response?

TIA

Date: Fri, 23 Sep 2011 21:18:48 +0200
From: Arnt Gulbrandsen <[email-masked]>
Organization: Me, http://arnt.gulbrandsen.priv.no
To: Alessandro Vesely <[email-masked]>
Cc: SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 09/23/2011 08:50 PM, Alessandro Vesely wrote:
> Most SMTP servers duly lookup the client's IP and annotate the
> resulting name as comment in Received fields.  However, I don't recall
> denying SMTP access based on the "iprev" test (as RFC 5451 named it.)
>   Was it ever à la mode to do so?

Briefly.

Arnt

Date: Fri, 23 Sep 2011 20:50:27 +0200
From: Alessandro Vesely <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

Hi all!

I recall several FTP servers used to deny anonymous access to clients
calling from IP numbers without rDNS record.  Perhaps it was the
default configuration of some package.

Most SMTP servers duly lookup the client's IP and annotate the
resulting name as comment in Received fields.  However, I don't recall
denying SMTP access based on the "iprev" test (as RFC 5451 named it.)
 Was it ever à la mode to do so?

The reason why I'm asking is to estimate the effectiveness of an
"arpa-whitelist".  This would be a DNS domain similar in structure to
in-addr.arpa, except that its content would be derived from
abuse-contact data of RIRs' whois databases.  For example,

   192.0.2.3.abuse-contact.arpa. TXT "[email-masked]"

That way, abuse-contact.arpa can work as a "quick fix using DNS" for a
functionality that various whois++, rwhois, and crisp/iris failed to
provide.  And it can also work as a whitelist, to exclude a number of
questionable assignments.  Whois settings are less sensitive to
occasional name changes than rDNS.  Thus, an arpa-whitelist might work
a bit better than iprev.

For IPv6, the whitelist of mailers at http://www.ipv6whitelist.eu/
didn't spread beyond the Netherlands.  Would hosting at arpa have
bestowed a better fate upon it?

Date: Tue, 06 Sep 2011 15:59:38 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

Tony Finch wrote:
> Hector <[email-masked]> wrote:
> 
>> But the same reasons POP3 exist is that not all users can be or wish to
>> be "mini-sysops."
> 
> Choice of protocol is orthogonal to usability. For instance, Turnpike is a
> fairly typical user-friendly GUI mail and news reader which includes an
> SMTP server for receiving incoming mail from ISPs like Demon.
> 
> POP mainly exists because back in the day servers didn't have enough disk
> and users did not often have more than one computer.

While it is true for smaller systems and operators disk space was a 
premium, choosing the right cluster size, cost, i.e. $1500 for a 10MB 
Micropolis EIDE drive, considered top of the line, etc, was a 
consideration, it not the main reason and breadthru: time-shifting, 
offloading technology needs was blossoming in many areas for three 
reasons:

      - Reduce Online cost,
      - Higher Productivity offline,
      - Greater Interactive I/O design capabilities (Fancy Frontends)

Online cost saving was the big one.

IBM even had the patent on offloading resources to clients which was 
the important "version 2" of offline mail system when we began to go 
graphical if only first starting with mixed text/iconic symbols, ala 
Minitel, Prodigy type systems.  Before there was HTML, there was RIP 
and it was definitely part of the top commercial Online Hosting 
Systems plans if not already incorporated.  HTML killed RIP.

I speak confidently as this as the 3rd offline mail system of its kind 
after TAPCIS and QMail Deluxe (QWK) and first to add Fidonet and 
RFC822 gating with the Silver Xpress Offline Mail System.  That 
started to wind down when the cost was no longer a factor and Online 
connectivity was now feasible.  RFC-based methods and that included 
WebMail also reduced the needs for Offline Mail/Point systems.

But RFC-based Offline Mail, especially POP3, NNTP methods really only 
help global standardization in an era where there was no standards - 
only technology leaders with particular proprietary methods and it 
really began to take off when internet connectivity cost was less, 
more feasible and accessible by wider mass market of users.

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: Menachem Moystoviz <[email-masked]>
Date: Tue, 6 Sep 2011 20:19:36 +0300
Subject: Re: Recipient is offline
To: John C Klensin <[email-masked]>
Cc: "Murray S. Kucherawy" <[email-masked]>, [email-masked]
Sender: [email-masked]

Thank you very much, all of you.
I appreciate your answering my question and hope to be of service.

Menachem D. Mostowicz

On Fri, Sep 2, 2011 at 4:03 AM, John C Klensin <[email-masked]> wrote:
>
>
>
> --On Thursday, September 01, 2011 16:14 -0700 "Murray S.
> Kucherawy" <[email-masked]> wrote:
>
>>...
>>> > Because this would mean that any user who turns his SMTP
>>> > server on for a couple of hours each day would receive his
>>> > mail.
>>
>> As long as there's a retry during that window, yes. �But that
>> isn't guaranteed.
>
> And, if the sender is doing with 5321 (and 2821 and 1123)
> suggest (I'm not going to have a discussion about what is and is
> not normative in 5321, at least today), the odds of a
> rarely-available receiver getting delivery go down rapidly with
> time. �Even if the sender keeps trying for more than the
> recommended 4-5 days, incremental back off strategies may make a
> short receiver-online connection period an increasingly elusive
> target.
>
> See the discussion in RFC 5321, Section 4.5.4.1, for additional
> hints about this or, as others have suggested, seriously
> consider an intermediate server that is usually or always on-net
> and a mechanism for working with it when your
> intermittently-connected host is online. �Those could be as
> simple as having the intermediate server know the typical
> schedule for the intermittent host being online as well as
> mechanisms like ETRN or pinging the intermediate with some sort
> of message.
>
> � �john
>
>>
>>> I'd say that most probably try for 3 days (gives admins
>>> chance to fix their server after a weekend if it broke on the
>>> Friday evening). Busy ones like gmail could try less because
>>> they have so much mail, and don't care as it's a free
>>> service. As long as they generate an NDR it's compliant with
>>> the standard.
>>
>> After some little-known sysadmin trickery I learned a long
>> time ago[*], I managed to figure out that they do indeed try
>> for at least a couple of days, though the frequency of retries
>> is not constant. �They certainly don't give up after only a
>> few seconds.
>>
>>> However, if a mail server is online for only a few hours a
>>> day, then I'd seriously suggest getting a backup mail server
>>> to accept messages during the downtime (or putting the main
>>> mail server somewhere else with a better connection), unless
>>> you don't really care about getting mail...
>>
>> I concur, especially if it's one that lets you do ETRN
>> (RFC1985) or some other demand-based delivery system.
>>
>> -MSK
>>
>> [*] I asked them.
>>
>
>
>
>
>

Date: Tue, 6 Sep 2011 17:32:44 +0100
From: Tony Finch <[email-masked]>
To: Hector <[email-masked]>
cc: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

Hector <[email-masked]> wrote:

> But the same reasons POP3 exist is that not all users can be or wish to
> be "mini-sysops."

Choice of protocol is orthogonal to usability. For instance, Turnpike is a
fairly typical user-friendly GUI mail and news reader which includes an
SMTP server for receiving incoming mail from ISPs like Demon.

POP mainly exists because back in the day servers didn't have enough disk
and users did not often have more than one computer.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Fair Isle: Cyclonic becoming northwest 5 to 7, occasionally gale 8 later.
Moderate or rough. Rain or squally showers. Moderate or good.

Date: Tue, 06 Sep 2011 03:07:55 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
CC: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
To: [email-masked]
Sender: [email-masked]

Dave CROCKER wrote:

>> Tony wrote:

>> POP has been abused for last-hop delivery to intermittently 
>> connected hosts

> In fact, during the development of the Internet Mail Architecture 
> document, I argued rather vigorously that it really /is/ a last-hop 
> delivery protocol, rather than being and 'access' protocol that parallels IMAP.

Dave, I don't particularly recall this debate, but it sounds this 
would of been a +1 from me.

> Community consensus went vigorously against me on that.  Certainly typical
> implementations don't really support a 'last-hop relay' model well or at 
> all, as evidenced by DSN being issued /before/ POP pickup...

I suggest it might depend on much old school material in retaining 
UUCP queuing, holding and relaying logic in the implementations.

There is no DSN when mail is accepted and stored for a locally hosted 
user - a behavior mode of the receiver MTA was as a MDA (last hop). 
This user-bin mail will normally only be accessed by MUAs. If not, it 
may expire per local policy mail storage policies such as:

    Days to keep Received mail:  21
    Days to keep Unread mail:  14

Of course, these days, those numbers are higher or even disabled (0).

If you are locally hosting other domains, then ala UUCP, it would be 
stored for the domain to pickup or your mailer schedules a UUCICO event.

When SMTP was added, the choice became per hosted domain:

    (o) Route via SMTP
    (_) Route via UUCP

I don't see MUAs here (last hop final destination) hence DSN is a 
possible outcome.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Mon, 05 Sep 2011 17:23:10 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

On 9/5/2011 4:36 PM, Tony Finch wrote:
>>> ODMR is better than POP in retaining the message envelope and not having
>>>  the "keep mail on server" loophole.
>>
>> I'm not quite seeing this if the comparison is with POP.  Can you
>> elaborate?
>
> POP has been abused for last-hop delivery to intermittently connected hosts

In fact, during the development of the Internet Mail Architecture document, I
argued rather vigorously that it really /is/ a last-hop delivery protocol,
rather than being and 'access' protocol that parallels IMAP.

Community consensus went vigorously against me on that.  Certainly typical
implementations don't really support a 'last-hop relay' model well or at all, as
evidenced by DSN being issued /before/ POP pickup...

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 05 Sep 2011 20:16:32 -0400
From: Hector <[email-masked]>
To: Tony Finch <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

But the same reasons POP3 exist is that not all users can be or wish 
to be "mini-sysops."  Point systems (which is what ODMR is) can be 
popular, but more especially when the user became less dependent on 
having any kind of "server" software install, like a QWK mail reader 
or POP3/IMAP/NEWS readers or just web browser. You were able to roam 
around.  It is not at all odd that eventually heavy users do 
eventually install a point system of some sort without the need to 
become full time servers.

I just noting by having a "mini-server" with an ODMR client, just 
shifts the "Keep Mail Over Server" to the mini server presuming the 
user will be using a Mail Reader that picks up mail from his mini server.

thanks

Tony Finch wrote:
> On 5 Sep 2011, at 18:59, Hector Santos <[email-masked]> wrote:
>> Tony Finch wrote:
>>
>>> ODMR is better than POP in retaining the message envelope and not having the "keep mail on server" loophole.
>> I'm not quite seeing this if the comparison is with POP.  Can you elaborate?
> 
> POP has been abused for last-hop delivery to intermittently connected hosts, e.g. fetchmail, Microsoft SBS, the envelope-in-headers conventions that John Levine mentioned, et cetera ad nauseam. ODMR would have been a much better choice if it had been available five years earlier...
> 
> Also I don't like POP as an MUA protocol in the current world of plentiful server storage, multiple clients per user, and multiple mailboxes per account. It leads to too many accidentally deleted inboxes.

Date: Mon, 05 Sep 2011 19:58:23 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

John Levine wrote:
>> Yes, but if a ODMR client did that (rejected after telling the server 
>> to send), the server will discard and denied access to ODMR.  Right?
> 
> No, of course not.  What language in RFC 2645 leads you to that
> peculiar conclusion?

None that I can quickly see/read offhand. It wasn't a conclusion. 
Perhaps a deny is too much but using a "454 Keep Mail On Server" most 
likely won't be tolerated in the same way the POP3 client doesn't have 
any control of backend Mail Storage Policies (see RFC1939 section 8). 
  I agree it would be a loophole to any current mail pickup controls 
using POP3.

Also, since server will be driven by reply codes and not the response 
text, he is can't be a mind reader. If 45x is deemed wasting the 
server's time of repeating sending old mail the ATRN client already 
has, I could see a denial of using ATRN for this client:

     C:  ATRN example.org,example.com
     P:  550 Sorry, Use POP3

-- 
Sincerely

Hector Santos
http://www.santronics.com

Cc: "[email-masked]" <[email-masked]>
From: Tony Finch <[email-masked]>
Subject: Re: Recipient is offline
Date: Tue, 6 Sep 2011 00:36:23 +0100
To: Hector Santos <[email-masked]>
Sender: [email-masked]

On 5 Sep 2011, at 18:59, Hector Santos <[email-masked]> wrote:
> Tony Finch wrote:
> 
>> ODMR is better than POP in retaining the message envelope and not having the "keep mail on server" loophole.
> 
> I'm not quite seeing this if the comparison is with POP.  Can you elaborate?

POP has been abused for last-hop delivery to intermittently connected hosts, e.g. fetchmail, Microsoft SBS, the envelope-in-headers conventions that John Levine mentioned, et cetera ad nauseam. ODMR would have been a much better choice if it had been available five years earlier...

Also I don't like POP as an MUA protocol in the current world of plentiful server storage, multiple clients per user, and multiple mailboxes per account. It leads to too many accidentally deleted inboxes.

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

Date: 5 Sep 2011 22:13:53 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Subject: Re: Recipient is offline
Organization: 
Sender: [email-masked]

>Yes, but if a ODMR client did that (rejected after telling the server 
>to send), the server will discard and denied access to ODMR.  Right?

No, of course not.  What language in RFC 2645 leads you to that
peculiar conclusion?

Date: Mon, 05 Sep 2011 15:33:34 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

Tony Finch wrote:
> John Levine <[email-masked]> wrote:
>> Loophole?
>>
>>  S: 452 keep mail on server
> 
> I should have thought of that :-) But it has more of a "temporarily"
> caveat than POP services usually do.

Yes, but if a ODMR client did that (rejected after telling the server 
to send), the server will discard and denied access to ODMR.  Right?

Good catch John.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Mon, 5 Sep 2011 19:00:30 +0100
From: Tony Finch <[email-masked]>
To: John Levine <[email-masked]>
cc: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

John Levine <[email-masked]> wrote:
>
> Loophole?
>
>  S: 452 keep mail on server

I should have thought of that :-) But it has more of a "temporarily"
caveat than POP services usually do.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Rockall, Malin, Hebrides: Cyclonic or southwest becoming northwest later, 5 to
7, occasionally gale 8 in Rockall and Malin. Moderate or rough, becoming very
rough except in Hebrides. Rain. Moderate or good, occasionally poor.

Date: Mon, 05 Sep 2011 13:59:30 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Finch <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

Tony Finch wrote:

> ODMR is better than POP in retaining the message envelope and not having 
> the "keep mail on server" loophole.

I'm not quite seeing this if the comparison is with POP.  Can you 
elaborate?

If the idea is for the user to now use his POP3 client to connect this 
his lite SMTP server setup, and not his main host, then the "keep mail 
on server" idea was simply shifted to another machine.  No?

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Mon, 05 Sep 2011 13:46:58 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Finch <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

Tony Finch wrote:

> The latter, I think, depending on what you mean by "online hold". 

"Connection Sharing/Caching" (CS) or like methods.

> When a message arrives, Exim normally tries to deliver it immediately. 

In my experience, its been typical of most mailer, if not all, to 
begin delivery of email (netmail) immediately with options to delay of 
course for some.

> If that
> succeeds it looks at the "waiting" hints database to see if there are any
> messages queued for the same host, and delivers them immediately. There
> is no in-memory state for these queued messages. Exim doesn't keep SMTP
> connections idle, and normally runs the queue in a random order.

IMTO, this is the better approach than the connect, hold approach.

> Yes. Exim has an option to serialize deliveries to certain hosts, in which
> case there is another hints database recording currently active
> connections. When Exim is about to connect to a host it checks this
> database and skips the host if there is a record for it. If there aren't
> any other destination hosts the message will be queued and should soon be
> picked up by the waiting mechanism. (This is more likely if the target is
> relatively slow, which is what the feature was designed for.) We use this
> to avoid load spikes caused by large mailing list deliveries to our
> message store servers, but not for general purpose deliveries.

I see.

What I was expressing to Klensin in my generalization of how SMTP 
reduced the complexity for queuing, I should of noted that this was 
highly dependent on leverage multi-threaded technology to maximize 
throughput. More powerful hardware and OS allowed the efficiency of 
1-M and N-M queuing strategies be more feasible and realistic.  I 
suggest that many of the long time mailers uses strategies that was 
appropriate for the era.

Mail Queuing is really no different than everyday queuing concept we 
see, have used and probably still use today. A Super Market/Grocery 
Store checkout lines, a Bank, an airline, etc. The queues they they 
employ is not by accident.  Overall, as an checkout line becomes 
empty, people on other close lines begin to move to it. Even with a 
dedicated queue, such as a "Less than 10 items" queue, if another 
queue is/near empty, people move to it. When people were forced to 
stay in a queue, the efficiency and throughput is shot. If a queue 
become too large, then a new checkout line is opened (we hope <g>). 
Same applies to mail when all service workers/queues are available to 
all messages to maximize their throughput.  The N-M queue model is 
only efficient as it approaches the 1-M model.

If we strictly dealing with the question of how to handle part time 
SMTP receiver failure, then the ultimate conclusion is as Crocker 
eluded to, is defining the final exit strategy,  Today, with SMTP, we 
only have one final exit strategy:

       BOUNCE when all attempts is exhausted

I guess at the implementation level, that may also includes

       BLOCK

SMTP encourages a 2nd MX on a different MX host network. If that is 
available, then the odds are good that there will never be a build up.

Old schooler mailer like Fidonet, it had a mandatory 1 hour period 
(ZMH) where receivers must be available for routing, and among its 
scheduler rules, it had ideas like:

    00-23 Route Mail directly to target host
    05-06 Route Mail via Your Network host (ZMH)

It seems that there might be a need to not bounce or block if the part 
time receiver is known with specific settings set for it.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Mon, 05 Sep 2011 13:03:24 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

Dave CROCKER wrote:

> The lesson here is that any 'retransmission' scheme needs an exit 
> strategy, perhaps better thought of as a means of internally routing 
> around a blockage...

+1

This was a main reason Fidonet had a mandatory ZMH (Zone Mail Hour) 
routing requirement so that delivery was possible if going direct was 
not possible was exhausted or it was listed as a private node in the 
nodelist).

In the better Fidonet mailer, it was typical to have a scheduler with 
very flexible routing rules, especially when POTS phone charges were 
not flat.

    00-23 Route Mail directly
    17-23 Route Mail directly if Area Code is not your Area Code
    05-06 Route Mail via Network host  (ZMH override)

SMTP attempts to emulate the need of "failure to directly connect" 
with the recommendation to have at least two MX records with two 
different SMTP host networks.

Question, do some telco return to have different rates with the mobile 
business?  Does anyone seen this happening more to reduce bandwidth 
loads during the day vs night?

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: 5 Sep 2011 17:02:29 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: Recipient is offline
Organization: 
Sender: [email-masked]

>ODMR is better than POP in retaining the message envelope and not having
>the "keep mail on server" loophole.

Loophole?

 C: mail from:<alice@a>
 S: 250 ok
 C: rcpt to:<bob@b>
 S: 250 ok
 C: data
 S: 354 send it
 C: blah blah
 C: .
 S: 452 keep mail on server

For mail reinjected after delivery, there has long been a convention
to retrieve the envelope from the Return-Path: and Delivered-To: or
X-Envelope-To: headers at the top of the message.  Fetchmail does it,
qmail's serialmail does it.

That's one of those things that probably would have been worth puting
in a standards track RFC long ago, but it's too late now.

R's,
John

Date: Mon, 05 Sep 2011 06:51:17 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
CC: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

On 9/3/2011 11:59 AM, [email-masked] wrote:
> suppose there's a message that crashes the receiving SMTP server. (This happens
> a lot more often than it should, unfortunately.) If you implement a "oldest
> first" or "most often retried first" strategy along with strict "always send
> every pending message once you get a connection open" policy, you can end up in
> a situation where the problematic message ends up blocking other messages which
> would work if you tried them first.

The lesson here is that any 'retransmission' scheme needs an exit strategy, 
perhaps better thought of as a means of internally routing around a blockage...

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 05 Sep 2011 06:49:25 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Tony Finch <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

On 9/3/2011 10:02 AM, Tony Finch wrote:
> On 3 Sep 2011, at 14:39, Dave CROCKER<[email-masked]>  wrote:
>
>> I don't know why RFC 5321 fails to mention ATRN, since it as an enhancement
>> over ETRN.
>
> ATRN is supposed to be used on the ODMR port, not port 25. The client and
> server software tends to be separate from normal SMTP implementations.

Right.  Missed that point.

Still, it's a profile of SMTP and it's standards track.  So I'd expect RFC 5321 
to cite it, just as it cites Submission.

> Demon Internet in the UK was unusual in using SMTP to deliver mail to its
> users.
...
>   The MTA was MMDF

And for those missing this essential bit of historical linkage:  MMDF was the 
software used for CSNet's telephone relay, that prompted the suggestion for this 
capability -- what became the TURN capability during SMTP development.  Having 
MMDF support the standardized, equivalent mechanism therefore would merely be 
natural...

> ODMR is better than POP in retaining the message envelope and not having the
> "keep mail on server" loophole.

+1

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 5 Sep 2011 10:31:57 +0100
From: Tony Finch <[email-masked]>
To: "Robert A. Rosenberg" <[email-masked]>
cc: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

Robert A. Rosenberg <[email-masked]> wrote:
>
> The simple solution to that is to encrypt the data at the sending location and
> decrypt it upon receipt at the receiving location. The data on-the-wire (ie:
> The actual data send during the phone call) is thus secure since any
> wire-tappers would get data that is useless. Using SSL over the Internet is
> basically the same idea.

This is only secure against on-path attackers if you authenticate the
endpoints. If you don't do that you are vulnerable to a man-in-the-middle
attack. If anyone is capable of passive eavesdropping on your connection
they can probably intercept with little extra difficulty.

Note that inter-domain SMTP with STARTTLS is unauthenticated.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Trafalgar: Variable 4. Moderate or rough. Mainly fair. Good.

Date: Mon, 5 Sep 2011 10:27:41 +0100
From: Tony Finch <[email-masked]>
To: hector <[email-masked]>
cc: "[email-masked]" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

hector <[email-masked]> wrote:
> Tony Finch wrote:
>
> > Exim has a database of which messages are waiting for which hosts, so when a
> > host comes back to life it can flush the messages down the same connection
> > without having to wait for the next queue run.
>
> Is that batched (collection pointer, perhaps) before the connection is made or
> is this part of "Online Hold" and EXIM does a database check for any more mail
> for the current target host?

The latter, I think, depending on what you mean by "online hold". When a
message arrives, Exim normally tries to deliver it immediately. If that
succeeds it looks at the "waiting" hints database to see if there are any
messages queued for the same host, and delivers them immediately. There
is no in-memory state for these queued messages. Exim doesn't keep SMTP
connections idle, and normally runs the queue in a random order.

> But if each Ms is transferred concurrently, then the total throughput time is
> no higher then the maximum Ts transaction.   This begins to less true as the
> total concurrency increases to a threshold where scaling and bottlenecking
> issues come into play.

Yes. Exim has an option to serialize deliveries to certain hosts, in which
case there is another hints database recording currently active
connections. When Exim is about to connect to a host it checks this
database and skips the host if there is a record for it. If there aren't
any other destination hosts the message will be queued and should soon be
picked up by the waiting mechanism. (This is more likely if the target is
relatively slow, which is what the feature was designed for.) We use this
to avoid load spikes caused by large mailing list deliveries to our
message store servers, but not for general purpose deliveries.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Dogger, Fisher, German Bight, Humber: South or southwest 4 or 5, increasing 5
to 7, occasionally gale 8 later in Dogger, German Bight and Humber, perhaps
gale 8 in Fisher. Slight or moderate, becoming moderate or rough. Rain or
showers. Moderate, occasionally poor.

Date: Mon, 5 Sep 2011 01:12:02 -0400
To: [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

At 07:20 -0400 on 09/03/2011, John C Klensin wrote about Re: 
Recipient is offline:

>Having worked for a couple of phone companies, I think they (and
>lots of others, of course) are deluding themselves when they
>assume that the PSTN is inherently more secure than a
>properly-set-up IP-based arrangement, especially with
>distributed routing.  Too many people (including some at the
>ITU) still have a model of PSTN routing and security that worked
>only with uninterrupted, non-multiplexed, copper between the
>call endpoints (and with the only real security risk being
>eavesdropping by the operator in the middle with the plug
>cords).   But, no, I have no trouble believing it at all.

The simple solution to that is to encrypt the data at the sending 
location and decrypt it upon receipt at the receiving location. The 
data on-the-wire (ie: The actual data send during the phone call) is 
thus secure since any wire-tappers would get data that is useless. 
Using SSL over the Internet is basically the same idea.

Date: Mon, 5 Sep 2011 01:29:00 -0400
To: "[email-masked]" <[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

At 17:43 +0100 on 09/03/2011, Tony Finch wrote about Re: Recipient is offline:

>On 3 Sep 2011, at 02:54, "Robert A. Rosenberg" <[email-masked]> wrote:
>>
>>  For this to work, there would need to be a list maintained by the 
>>delivering MTA  that lists which other MTAs it has tried to deliver 
>>to which failed due to failure to connect (along with a queue of 
>>messages that need to be delivered to that MTA and a time to next 
>>attempt to connect). When a new message arrives and it gets 
>>delivered (since that connect attempt worked) then the queue of 
>>pending messages can then be immediately flushed to the MTA. How 
>>many MTAs have this type of support to force flush a pending queue 
>>when a message delivery succeeds?
>
>Exim works in roughly this way.

Upon rereading my flow, I noted a possible timing glitch since with 
my method, the triggering message would end up arriving before those 
that were in the queue and would released by the successful 
triggering message delivery. A better method would be to place the 
message at the end of the queue and read (but not remove) the first 
(ie: Earliest) message on the queue and attempt to deliver it. If 
successful then delete that message from the queue and flush the rest 
to the receiving MTA. A failure to connect would just delete the copy 
of the message (it is still at the top of the queue) and all will 
remain until the next scheduled connection attempt (you might want to 
restart the wait clock since an attempt was just done).

>
>Tony.
>--
>f.anthony.n.finch  <[email-masked]>  http://dotat.at/

Date: Sun, 04 Sep 2011 19:08:49 -0400
From: Hector <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: Outbound Order and Priority [was Re: Recipient is offline]
Sender: [email-masked]

Small follow up below:

Hector wrote:
> 
> Tony Finch wrote:
> 
>> Exim has a database of which messages are waiting for which hosts, so 
>> when a host comes back to life it can flush the messages down the same 
>> connection without having to wait for the next queue run.
> 
> ...
>
> So I suggest that the software is collecting, sorting and probably maybe 
> related to total counts, like gmail.com addresses and others common 
> subscription addresses, that it spawn delivery job based on the highest 
> count first and the single host mail as done last and maybe 
> alphabetically.  I suppose total failure history to connect is a factor, 
> but our MX servers are rarely ever down.
> 
> The only point here is to express another reason why we choose to send 
> mail out autonomously, equally as they come for the highest efficient 
> and throughput. When you  priority based on counts, etc, the majority 
> which are count=1 host mail will be delayed far longer than it has to been.
> 
> The only time wcSMTP stops and blocks a host domain is with NXDOMAIN or 
> connect failures attempts is exhausted.   We provide operators a URL 
> link to removed them off the block list.  This only happens with 
> non-full time low end operators and sites.  ETRN will not work at this 
> point for any old mail that was already bounced.

Once upon a time, we had an auto-unblock feature where if an email 
came in from a outbound blocked host, it would auto-remove the blocked 
host for future outbound activity to the host.

That worked very well for good guys, except that BY FAR, the auto 
failure to connect block feature was a super major success against bad 
guys.  So the feature to auto unblock a host when new inbound mail 
negated the overwhelming worth gain in blocking bad guys with outbound 
failures.   So we basically made it a manual removal idea and combined 
it with the URL LINK for good guys to use which sends a notification 
message to the local operator to remove the host from the list.

--
HLS

Date: Sun, 04 Sep 2011 18:14:21 -0400
From: Hector <[email-masked]>
To: Tony Finch <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Outbound Order and Priority [was Re: Recipient is offline]
Sender: [email-masked]

Tony Finch wrote:

> Exim has a database of which messages are waiting for which hosts, so 
> when a host comes back to life it can flush the messages down the 
> same connection without having to wait for the next queue run.

This brings up a side point when host collection, sorting and queuing 
is done.

I only suggest this as reasonable explanation for this example 
observation.

In this list and others, like IETF, I noticed the my subscription 
address with a host to santronics.com, takes am extremely long time 
for one of my messages to get posted. Sometime its pretty quick like 
within the hour for IETF-SMTP, IETF-DKIM but it could be many hours to 
half a day to even the next day.  For the IETF list, it is perpetually 
worst and I have seen my santronics.com take 2-3 days before it is 
posted.  When I saw that, I asked the list operator and he indicated 
appeared to be a problem somewhere he could not explain and could not 
address yet until he return from a business trip.

I asked about queuing and he wasn't sure what the special magic was. 
So I subscribed with my junk alias gmail.com alias and now all my post 
come in pretty fast in both IETF-SMTP and IETF list within the hour or 
so.  That is why I have been posting with my gmail account.

So I suggest that the software is collecting, sorting and probably 
maybe related to total counts, like gmail.com addresses and others 
common subscription addresses, that it spawn delivery job based on the 
highest count first and the single host mail as done last and maybe 
alphabetically.  I suppose total failure history to connect is a 
factor, but our MX servers are rarely ever down.

The only point here is to express another reason why we choose to send 
mail out autonomously, equally as they come for the highest efficient 
and throughput. When you  priority based on counts, etc, the majority 
which are count=1 host mail will be delayed far longer than it has to 
been.

The only time wcSMTP stops and blocks a host domain is with NXDOMAIN 
or connect failures attempts is exhausted.   We provide operators a 
URL link to removed them off the block list.  This only happens with 
non-full time low end operators and sites.  ETRN will not work at this 
point for any old mail that was already bounced.

--

Cc: "[email-masked]" <[email-masked]>
From: Tony Finch <[email-masked]>
Subject: Re: Recipient is offline
Date: Sun, 4 Sep 2011 13:05:31 +0100
To: "Murray S. Kucherawy" <[email-masked]>
Sender: [email-masked]

On 4 Sep 2011, at 06:27, "Murray S. Kucherawy" <[email-masked]> wrote:
> 
> To my knowledge, sendmail implements the queue with the factors Ned listed above included in its sorting, but it doesn't have a system that detects "Oh hey, that host is available again, I'm going to flush everything I have queued for it."  That just falls out naturally as one aspect of a complete queue run.

Exim has a database of which messages are waiting for which hosts, so when a host comes back to life it can flush the messages down the same connection without having to wait for the next queue run.

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Sat, 3 Sep 2011 22:27:15 -0700
Subject: RE: Recipient is offline
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
> Sent: Saturday, September 03, 2011 11:59 AM
> To: Tony Finch
> Cc: Robert A. Rosenberg; [email-masked]
> Subject: Re: Recipient is offline
> 
> > > message delivery succeeds?
> 
> > Exim works in roughly this way.
> 
> So does our product (currently called Oracle Communications Messaging Server,
> but who knows what the name will be tomorrow ;-), although there are a number
> of additional details like message priority and number of previous delivery
> attempts that have to be factored in.
> 
> I've always assumed that a fair number of other MTAs also do this sort of
> thing.

To my knowledge, sendmail implements the queue with the factors Ned listed above included in its sorting, but it doesn't have a system that detects "Oh hey, that host is available again, I'm going to flush everything I have queued for it."  That just falls out naturally as one aspect of a complete queue run.

What it does do is notice that a connection to some host has failed, and then remember that for a while so it doesn't try again and have to wait for connect() to time out (which can take a while, slowing things down considerably).

From: [email-masked]
Cc: "Robert A. Rosenberg" <[email-masked]>, "[email-masked]" <[email-masked]>
Date: Sat, 03 Sep 2011 11:59:08 -0700 (PDT)
Subject: Re: Recipient is offline
To: Tony Finch <[email-masked]>
Sender: [email-masked]

> > For this to work, there would need to be a list maintained by the
> > delivering MTA  that lists which other MTAs it has tried to deliver to which
> > failed due to failure to connect (along with a queue of messages that need to
> > be delivered to that MTA and a time to next attempt to connect). When a new
> > message arrives and it gets delivered (since that connect attempt worked) then
> > the queue of pending messages can then be immediately flushed to the MTA. How
> > many MTAs have this type of support to force flush a pending queue when a
> > message delivery succeeds?

> Exim works in roughly this way.

So does our product (currently called Oracle Communications Messaging Server,
but who knows what the name will be tomorrow ;-), although there are a number
of additional details like message priority and number of previous delivery
attempts that have to be factored in.

I've always assumed that a fair number of other MTAs also do this sort of
thing.

But sometimes these strategies cause problems of their own. For example,
suppose there's a message that crashes the receiving SMTP server. (This happens
a lot more often than it should, unfortunately.) If you implement a "oldest
first" or "most often retried first" strategy along with strict "always send
every pending message once you get a connection open" policy, you can end up in
a situation where the problematic message ends up blocking other messages which
would work if you tried them first. And this issue has gotten more pernicious
over time as email service has consolidated onto fewer, larger service
providers, who sadly aren't in any way immune from the crashing message issue.

				Ned

Date: Sat, 03 Sep 2011 14:21:58 -0400
From: Hector <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

Dave CROCKER wrote:

> Here's the essential bit:  Occasional connectivity fits into the 
> Internet mail model only as a last-hop, receive-side issue.  There must 
> be an always-connected server working on behalf of the 
> sometimes-connected receiver and holding mail for it.  Hence, use of 
> ODMR is a 'local' matter.  Do not expect random, remote servers to 
> participate.

The HUB!

> [1]  http://isoc.org/wp/newsletter/?p=1098

DISCO was dead by this point! <g>

> 
> [2]  http://bbiw.net/recent.html#PostelCSNet
> 
> [3]  It was interesting presenting the SMTP "working group" with the 
> challenge of this operational style, since it was entirely foreign to 
> the existing world of the Arpanet.  Folks were not all that receptive to 
> it until Vint Cerf, then at Arpa and always looking for ways to justify 
> funding worthy projects, suddenly said "submarines!"  At that point, 
> everybody grok'd the operational model that leaves availability for 
> delivery as a choice of the receiver.

Very interesting David.  Fidonet had a Policy4 membership ZMH (Zone 
Mail Hour) requirement to allow routes via your hub and no humans were 
allowed to tie the line during ZMH.

    http://www.fidonet.us/policy4.html

I guess the 821 question I have is, Given Cerf's direction, I don't 
see any description or is hidden somewhere in 821 about connection 
errors and retries recommendations.  Was that because not enough 
intelligence yet or you guys just had a "Card Blanc" with full time 
reliable connectivity (I suspect delays was par)?

Cc: "[email-masked]" <[email-masked]>
From: Tony Finch <[email-masked]>
Subject: Re: Recipient is offline
Date: Sat, 3 Sep 2011 18:02:45 +0100
To: "[email-masked]" <[email-masked]>
Sender: [email-masked]

On 3 Sep 2011, at 14:39, Dave CROCKER <[email-masked]> wrote:

> I don't know why RFC 5321 fails to mention ATRN, since it as an enhancement over ETRN.

ATRN is supposed to be used on the ODMR port, not port 25. The client and server software tends to be separate from normal SMTP implementations.

> Here's the essential bit:  Occasional connectivity fits into the Internet mail model only as a last-hop, receive-side issue.  There must be an always-connected server working on behalf of the sometimes-connected receiver and holding mail for it.  Hence, use of ODMR is a 'local' matter.  Do not expect random, remote servers to participate.

Yes.

Demon Internet in the UK was unusual in using SMTP to deliver mail to its users. The radius servers kicked off a couple of processed when a user dialled in: first, set up he routing for the user's static IP address, and second, kick off a queue runner for that user's mail. It is not safe to do this kind of mail delivery without static IP addresses! The MTA was MMDF (Demon started out as a SCO shop but was all BSD and Solaris when I was there) and the queue was split with a directory per user so delivery was fast. The POP service was added later.

ODMR is better than POP in retaining the message envelope and not having the "keep mail on server" loophole.

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

Cc: "[email-masked]" <[email-masked]>
From: Tony Finch <[email-masked]>
Subject: Re: Recipient is offline
Date: Sat, 3 Sep 2011 17:43:41 +0100
To: "Robert A. Rosenberg" <[email-masked]>
Sender: [email-masked]

On 3 Sep 2011, at 02:54, "Robert A. Rosenberg" <[email-masked]> wrote:
> 
> For this to work, there would need to be a list maintained by the delivering MTA  that lists which other MTAs it has tried to deliver to which failed due to failure to connect (along with a queue of messages that need to be delivered to that MTA and a time to next attempt to connect). When a new message arrives and it gets delivered (since that connect attempt worked) then the queue of pending messages can then be immediately flushed to the MTA. How many MTAs have this type of support to force flush a pending queue when a message delivery succeeds?

Exim works in roughly this way.

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

Date: Sat, 03 Sep 2011 12:06:05 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

John C Klensin wrote:
> 
> --On Saturday, September 03, 2011 03:23 -0400 Hector
> <[email-masked]> wrote:
> 
>> You might this reading interesting by Chuck Forberg of the
>> early comm issues, including UUCP, leading up to his
>> development of ZMODEM:
>>
>>       http://www.omen.com/zmdmev.html
> 
> Saw it when it came out.  Chuck hurt his case a bit by those who
> knew both sides by going a little over the top in his outrage
> (e.g., claiming that Columbia's non-profit mailing certificate
> is equivalent to mailing at public expense).  Frank responded by
> going over the top too, which didn't help -- Kermit's main
> strength was transfers among dissimilar machines with option
> negotiation, especially with dissimilar character sets or other
> data encodings, while X/Y/ZMODEM assumed either homogeneity or a
> "get information about encodings out of band and then sort it
> out at the endpoints" logic.  ...

My memory with this was mostly: used Columbia Kermit for testing SSL 
TELNET & Kermit protocol implementation and for recommending it to 
customers with SSL Telnet needs; Mostly unix clients that used it, 
probably universities; Was not impressed with performance, i.e. more 
problems reported. If I remember mostly related to Window sizes and 
different client/server kermit versions.   I think it worked better 
under RLOGIN as the other XYZ protocols. ZMODEM and even YMODEM-G1K 
was excellent and for batching. I never saw problems as you described 
with ZMODEM over any wire, but ultimately not as efficient with a TCP 
stream since TCP already did the error corrections.

This is a typical arrangement with higher end customers:

    TCP/Dialup --> PBX/TELCO --> CommServer --> TCP services

TCP Services could be RLogin, FTP, HTTP, SMTP, POP3. Customer 
programmed methods. CommServers includes many vendors such older EICON 
(Dialogic), Cisco, Pattons (like these) routers and others. With 
automated dialup text based console scripts already done for dialup, 
clients did not lose with a passive dialup virtual comm to RLOGIN. 
The backend was now able to start reducing modem banks.

>> - RFC822 headers resolved major issues with Kludge lines
>> (network control lines) in the Fidonet mail format,

> I would add that the introduction of really clear boundaries
> between applications-layer transport and message functions
> (i.e., the invention of the envelope-message distinction) was
> also important and, ultimately simplifying (even though, in
> retrospect, they got the details wrong)

That is what I meant with FTN1 packets, which had:

    Fixed FTN1 block
    ^A control lines (routing/sender/reply addresses
                      msgid, pid, etc)
    Message Body
    --- [31 bytes MailerID]  <--- required tear line with brag line
    ^A SeenBy lines for distribution and dupe protection
    Fixed FTN1 block
    ...

> As far as adding more commands is concerned... yes, but doing so
> caused horrible interoperability problems, resulting in our
> having to invent the rather complex EHLO handshake mechanisms in
> order to take advantage of that flexibility.

I only added EHLO after you were finished with RFC2821 :) but with 
821, some proprietary local only helpdesk, stats, debug, dumps, etcs 
commands were added.

> What the Internet had instead was Postel's robustness principle 
> and its aggressive (perhaps excessive) application to making email work.

Its a natural principle applicable to many areas. I had a "Flow 
Control" rule of thumb for RS232:

        Send LOW, Receive HIGH

Receivers know how much they handle, but they know what the other side 
can handle. Another thing you
didn't have to worry with TCP using the same receive/transmit packet size.

> Having worked for a couple of phone companies, I think they (and
> lots of others, of course) are deluding themselves when they
> assume that the PSTN is inherently more secure than a
> properly-set-up IP-based arrangement, especially with
> distributed routing.  

+1 I keep quiet on that!  I just presumed different trunk lines were 
used but I am increasing seeing stories that its all put under same 
trunks. Even then, encrypted modems can be used to help. A recent DOE 
customer is using this with encrypted modems:

        http://www.winserver.com/public/wcnavigator.wct

This uses a dialup with a mini-PPP frame wire.  It also has a 821 SMTP 
MTA to send new messages.  This was to avoid APIs calls over the wire 
to post a message with attachments. SMTP was easier. :)

> Too many people (including some at the
> ITU) still have a model of PSTN routing and security that worked
> only with uninterrupted, non-multiplexed, copper between the
> call endpoints (and with the only real security risk being
> eavesdropping by the operator in the middle with the plug
> cords).   But, no, I have no trouble believing it at all.

Or Russian Vans with electronic magnetic field snooping which required 
Tempest-ready computer and devices! :)

> but otherwise in many ways more
> similar to the FidoNet model than to today's MUA-> MSA -> SMTP
> Relay -> SMTP Relay -> Delivery MTA -> Mailstore <-> Split-UA
> server -> Split-UA MTA story.

You mean Fidonet inherent dual role vs SMTP single role?

>>  HTTP WebSockets will make that one!
> 
> While, I'm sure, causing their own set of problems :-(

High Potential though. Hosting systems should not ignore.

Date: Sat, 03 Sep 2011 06:39:26 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

On 9/2/2011 8:43 AM, Tony Finch wrote:
> There's also ODMR / ATRN - RFC 2645.

Right.

The model for the Internet's email infrastructure is always-connected, 
supporting an active push to the recipient's machine at the sender's initiative. 
  Unavailability is supported in terms of unplanned outages, not basic 
connectivity limitations such as being connected only part of the time.

uucp-based Usenet was a very early example of a service designed for 
partial-connectivity.  CSNet[1][2] served as the first example within the 
Internet (Arpanet) mail model, with remote sites dialing up when they could 
afford to.

SMTP was developed shortly after this and we included efforts to get a "give me 
mail that you are holding for me" mechanism to work[3]:  The 
periodically-connected site signals that it is now available to receive waiting 
mail.  Like the sequence that produced the MX record, finding the right design 
for the TURN mechanism took a few iterations.  In this case, getting the 
security details right was the major challenge. Although the first effort was 
deprecated, later efforts are still standards-track. (I don't know why RFC 5321 
fails to mention ATRN, since it as an enhancement over ETRN.)

Here's the essential bit:  Occasional connectivity fits into the Internet mail 
model only as a last-hop, receive-side issue.  There must be an always-connected 
server working on behalf of the sometimes-connected receiver and holding mail 
for it.  Hence, use of ODMR is a 'local' matter.  Do not expect random, remote 
servers to participate.

d/

[1]  http://isoc.org/wp/newsletter/?p=1098

[2]  http://bbiw.net/recent.html#PostelCSNet

[3]  It was interesting presenting the SMTP "working group" with the challenge 
of this operational style, since it was entirely foreign to the existing world 
of the Arpanet.  Folks were not all that receptive to it until Vint Cerf, then 
at Arpa and always looking for ways to justify funding worthy projects, suddenly 
said "submarines!"  At that point, everybody grok'd the operational model that 
leaves availability for delivery as a choice of the receiver.

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Sat, 03 Sep 2011 07:20:10 -0400
From: John C Klensin <[email-masked]>
To: Hector <[email-masked]>
cc: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

--On Saturday, September 03, 2011 03:23 -0400 Hector
<[email-masked]> wrote:

> 
> John C Klensin wrote:
> 
>> A very significant and important improvement, IMO.
>> Presumably, if a telephone# still appears and is supported at
>> both ends, it continues to work.
> 
> It was from the standpoint it allowed Fidonet mailers to get
> away from the FTN1 protocol (XMODEM, no error correction)
> constraints not working under X.25 and TCP and use instead its
> more modern WAZOO (Yoohoo / Yoohoo2) text based handshaking
> protocol to negotiate capabilities, authentication with
> matching network addresses, and selection of the data transfer
> protocol, ZMODEM was the default designed specifically for
> X.25.
> 
> You might this reading interesting by Chuck Forberg of the
> early comm issues, including UUCP, leading up to his
> development of ZMODEM:
> 
>       http://www.omen.com/zmdmev.html

Saw it when it came out.  Chuck hurt his case a bit by those who
knew both sides by going a little over the top in his outrage
(e.g., claiming that Columbia's non-profit mailing certificate
is equivalent to mailing at public expense).  Frank responded by
going over the top too, which didn't help -- Kermit's main
strength was transfers among dissimilar machines with option
negotiation, especially with dissimilar character sets or other
data encodings, while X/Y/ZMODEM assumed either homogeneity or a
"get information about encodings out of band and then sort it
out at the endpoints" logic.  Unless the machine and encoding
architectures were identical, comparing the two approaches on
transfer speed alone was just silly (whether Frank understood
how to run a speed test or not -- I agree with Chuck that he did
not, but do not go so far in assuming malice).  To make a fair
test, one would have had to assign some measure to the costs of
transferring protocol information out of band (pretty arbitrary
and hence unlikely to be meaningful) and then to measure the
time and costs of post-transfer conversions.  Otherwise, while
the costs of simply keeping the line up (or transferring a few
extra packets) were a significant consideration, comparing
ZMODEM and Kermit was a bit like comparing TCP and FTP -- raw
data stream versus a fairly sophisticated processor.   (Of
course, Kermit's common data model and conversions echoed FTP
and TYPE A, on which it was modeled, and which Columbia needed
to have, initially in order to deal with transfers between
EBCDIC and ASCII machines.)

Disclaimer: FWIW, I did most of the development for the CP/M-86
(aka DR-DOS) and Concurrent CP/M versions of Kermit and made
some contributions to the OS/2 version.

> I recall really easily liking SMTP because it reminded me of
> flexibility WAZOO provided and how much more power and
> flexibility RFC822/821 had to offer.
 
> SMTP (821/822) just simplified things:
> 
> - Design complexity for Queuing/Schedule reduce, less used and
>    even removed.

I might argue that wasn't true.  It just shifted parts of that
work outside the protocol and into recommendations that involved
a lot of handwaving.   Certainly its queuing-retry and relay
models were much more complex than FTP-based email,which had
neither.

> - Dynamic Delivery/Receiving benefits were immediate.
> 
> - Remove the need for error-correction protocols. TCP did that
> for us.
> 
> - RFC822 headers resolved major issues with Kludge lines
> (network control lines) in the Fidonet mail format,
> 
> - RFC822 text base format allowed for expansion and growth,
> 
> - RFC821 was flexible, you can easily add more command line
>    mode commands,

I would add that the introduction of really clear boundaries
between applications-layer transport and message functions
(i.e., the invention of the envelope-message distinction) was
also important and, ultimately simplifying (even though, in
retrospect, they got the details wrong)

As far as adding more commands is concerned... yes, but doing so
caused horrible interoperability problems, resulting in our
having to invent the rather complex EHLO handshake mechanisms in
order to take advantage of that flexibility.

> and it was an added bonus the Internet had no "rules" unlike
> the FTSC restrictions where non-compliancy got you (developer
> and operators) off the Fidonet NodeList. :)

Depends on when you start counting and what you count.  During
the ARPANET period, hosts could get removed from the network for
egregious behavior and sometimes were.  One might argue that
those decisions were a little less political (in terms of whose
aberrations were tolerated) than the FidoNet ones, but that is a
separate issue.  What the Internet had instead was Postel's
robustness principle and its aggressive (perhaps excessive)
application to making email work.
 
>>> Why Dialup?
>>> 
>>> # 1, its one way to win a fast entry contract with the feds
>>> # as the exception to HIPAA secured internet transfer
>>> # requirements.
>> 
>> That application had not occurred to me.  I find it
>> fascinating as well as worthwhile.  The other two examples
>> surprise me less.
> 
> I only mentioned this one because almost all the insurance
> companies, providers, subcontractors and clearing houses do
> this. But would you believe the CIA, FBI, DOD, DOE, State,
> Local law enforcement, many Fire Departments, payroll, banks,
> many with  end of day activities, inventories (i.e. hand
> scanners), etc, that continue to use dialup to avoid any
> security issues and/or cost?

Having worked for a couple of phone companies, I think they (and
lots of others, of course) are deluding themselves when they
assume that the PSTN is inherently more secure than a
properly-set-up IP-based arrangement, especially with
distributed routing.  Too many people (including some at the
ITU) still have a model of PSTN routing and security that worked
only with uninterrupted, non-multiplexed, copper between the
call endpoints (and with the only real security risk being
eavesdropping by the operator in the middle with the plug
cords).   But, no, I have no trouble believing it at all.
 
> But for many, its the demand of their clients. So even if
> network security isn't the reason and secured internet portals
> are available, many of their clients (any size) simply won't
> tolerate or budge on having to increase their communication
> cost or, a biggy, have to develop software or pay for an
> administrator,  especially if its not a 24x7 need.

Or, I assume, having seen it in too many other places, simply
consider any change to what they are doing because it would
disrupt operations, require retraining people, etc.   But these
are the same organizations who, for similar reasons, were still
running 650 and 7070 emulators on their computers until not long
ago (I have no reason to believe that all of them have stopped).
That may sound negative, but really isn't -- I've done enough
analysis of the costs of retraining people, reviewing and
changing procedures, certifying new systems and proving their
equivalent to older ones, and so that I suspect they may be
behaving rationally.

>  Not many
> service providers will do this unless they have special client
> software or offset the cost some how, so they just say "Here
> are phone numbers, secured ftp, smtp, http addresses and you
> choose how you want send/pickup stuff."

Right.

> Anyway, I will note, and maybe that is what I am stuck with
> John, in my view, the paradigm and of course, with variations,
> the model has never really changed from:
> 
>      CONNECT
>        SEND, IF ANYTHING
>        PICKUP, IF ANYTHING
>      DISCONNECT
> 
> What is a large part for us is:
> 
>      CONNECT
>        PICKUP OUTPUT, if ANY
>        SEND   --> BACKEND PROCESS --> OUTPUT
>      DISCONNECT

> How client do the above was up to them. One Sessions, Two
> Sessions, etc. FTP keeps that as one.  SMTP/POP3 split it that
> into two.

>From a more traditional Internet point of view, one could make
"one or two" arguments about SMTP with and without TURN, but
POP3 was irrelevant (and not invented) until client-server
models with clients a lot less capable than servers became
popular.  For the first 15 years or so of my email history, MUAs
read directly from mailstores or, more commonly, MUAs read from
mailboxes to which delivery had been made directly from an MTA
with no distinct mailstore.  Remote and very dumb terminals
connected to much bigger iron, but otherwise in many ways more
similar to the FidoNet model than to today's MUA-> MSA -> SMTP
Relay -> SMTP Relay -> Delivery MTA -> Mailstore <-> Split-UA
server -> Split-UA MTA story.

>  HTTP, I guess from different POVs, one or two
> sessions.  HTTP WebSockets will make that one!

While, I'm sure, causing their own set of problems :-(

best,
   john

Date: Sat, 03 Sep 2011 03:23:55 -0400
From: Hector <[email-masked]>
CC: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

John C Klensin wrote:

> A very significant and important improvement, IMO.  Presumably,
> if a telephone# still appears and is supported at both ends, it
> continues to work.

It was from the standpoint it allowed Fidonet mailers to get away from 
the FTN1 protocol (XMODEM, no error correction) constraints not 
working under X.25 and TCP and use instead its more modern WAZOO 
(Yoohoo / Yoohoo2) text based handshaking protocol to negotiate 
capabilities, authentication with matching network addresses, and 
selection of the data transfer protocol, ZMODEM was the default 
designed specifically for X.25.

You might this reading interesting by Chuck Forberg of the early comm 
issues, including UUCP, leading up to his development of ZMODEM:

      http://www.omen.com/zmdmev.html

I recall really easily liking SMTP because it reminded me of 
flexibility WAZOO provided and how much more power and flexibility 
RFC822/821 had to offer.

SMTP (821/822) just simplified things:

- Design complexity for Queuing/Schedule reduce, less used and
   even removed.

- Dynamic Delivery/Receiving benefits were immediate.

- Remove the need for error-correction protocols. TCP did that for us.

- RFC822 headers resolved major issues with Kludge lines (network
   control lines) in the Fidonet mail format,

- RFC822 text base format allowed for expansion and growth,

- RFC821 was flexible, you can easily add more command line
   mode commands,

and it was an added bonus the Internet had no "rules" unlike the FTSC 
restrictions where non-compliancy got you (developer and operators) 
off the Fidonet NodeList. :)

>> Why Dialup?
>>
>> # 1, its one way to win a fast entry contract with the feds as
>> # the exception to HIPAA secured internet transfer requirements.
> 
> That application had not occurred to me.  I find it fascinating
> as well as worthwhile.  The other two examples surprise me less.

I only mentioned this one because almost all the insurance companies, 
providers, subcontractors and clearing houses do this. But would you 
believe the CIA, FBI, DOD, DOE, State, Local law enforcement, many 
Fire Departments, payroll, banks, many with  end of day activities, 
inventories (i.e. hand scanners), etc, that continue to use dialup to 
avoid any security issues and/or cost?

But for many, its the demand of their clients. So even if network 
security isn't the reason and secured internet portals are available, 
many of their clients (any size) simply won't tolerate or budge on 
having to increase their communication cost or, a biggy, have to 
develop software or pay for an administrator,  especially if its not a 
24x7 need.  Not many service providers will do this unless they have 
special client software or offset the cost some how, so they just say 
"Here are phone numbers, secured ftp, smtp, http addresses and you 
choose how you want send/pickup stuff."

Anyway, I will note, and maybe that is what I am stuck with John, in 
my view, the paradigm and of course, with variations, the model has 
never really changed from:

     CONNECT
       SEND, IF ANYTHING
       PICKUP, IF ANYTHING
     DISCONNECT

What is a large part for us is:

     CONNECT
       PICKUP OUTPUT, if ANY
       SEND   --> BACKEND PROCESS --> OUTPUT
     DISCONNECT

How client do the above was up to them. One Sessions, Two Sessions, 
etc. FTP keeps that as one.  SMTP/POP3 split it that into two.  HTTP, 
I guess from different POVs, one or two sessions.  HTTP WebSockets 
will make that one!

-- 

HLS

Date: Sat, 03 Sep 2011 02:10:42 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: Received: and sending IP
Sender: [email-masked]

Hector Santos wrote:

> Just to make sure I am following the question, are you referred to the 
> angle bracketed domain literal?

Oops, meant square bracketed domain literals.

> 
> If so, insights predating 1995-2000 DRUMS years with the 1982 RFC821 and 
> the 1989 "Holy Bible" RFC1123 and the already existing Received: header 
> format would help with the prior art.  RFC1123 cleared it up:
> 
>       5.2.8  DATA Command: RFC-821 Section 4.1.1
> 
>          Every receiver-SMTP (not just one that "accepts a message for
>          relaying or for final delivery" [SMTP:1]) MUST insert a
>          "Received:" line at the beginning of a message.  In this line,
>          called a "time stamp line" in RFC-821:
> 
>          *    The FROM field SHOULD contain both (1) the name of the
>               source host as presented in the HELO command and (2) a
>               domain literal containing the IP address of the source,
>               determined from the TCP connection.
> 
>          *    The ID field MAY contain an "@" as suggested in RFC-822,
>               but this is not required.
> 
>          *    The FOR field MAY contain a list of <path> entries when
>               multiple RCPT commands have been given.
> 
> Is this what you are referring to?

RFC821 says:

          Sometimes a host is not known to the translation function and
          communication is blocked.  To bypass this barrier two numeric
          forms are also allowed for host "names".  One form is a decimal
          integer prefixed by a pound sign, "#", which indicates the
          number is the address of the host.  Another form is four small
          decimal integers separated by dots and enclosed by brackets,
          e.g., "[123.255.37.2]", which indicates a 32-bit ARPA Internet
          Address in four 8-bit fields.

I will note murray, I don't seem to recall any special thinking other 
than taking what HELO field value and doing a macro replacing for the 
Received template:

     Received: from HELO.HOST by LOCAL.HOST; TIME-STAMP

But I also do recall looking at RFC822 messages to see what other 
receivers were doing for the Received: trace line, not just for this 
but for other informations to add.

Date: Sat, 03 Sep 2011 00:33:33 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
CC: "[email-masked]" <[email-masked]>
Subject: Re: Received: and sending IP
To: [email-masked]
Sender: [email-masked]

Murray S. Kucherawy wrote:
>> -----Original Message-----

>> To the point where if we were to support such a change at this point, it would
>> be optional, off by default, and I doubt we'd recommend enabling it.
> 
> I'm not really proposing to make this change in a 5322bis or anywhere else.  
> I'm just curious as to why the current syntax won out over something 
> that seems less dissonant.

Just to make sure I am following the question, are you referred to the 
angle bracketed domain literal?

If so, insights predating 1995-2000 DRUMS years with the 1982 RFC821 
and the 1989 "Holy Bible" RFC1123 and the already existing Received: 
header format would help with the prior art.  RFC1123 cleared it up:

       5.2.8  DATA Command: RFC-821 Section 4.1.1

          Every receiver-SMTP (not just one that "accepts a message for
          relaying or for final delivery" [SMTP:1]) MUST insert a
          "Received:" line at the beginning of a message.  In this line,
          called a "time stamp line" in RFC-821:

          *    The FROM field SHOULD contain both (1) the name of the
               source host as presented in the HELO command and (2) a
               domain literal containing the IP address of the source,
               determined from the TCP connection.

          *    The ID field MAY contain an "@" as suggested in RFC-822,
               but this is not required.

          *    The FOR field MAY contain a list of <path> entries when
               multiple RCPT commands have been given.

Is this what you are referring to?

What alternative format would you proposed?

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 2 Sep 2011 21:54:57 -0400
To: [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: Recipient is offline
Sender: [email-masked]

At 10:08 -0400 on 09/02/2011, John C Klensin wrote about Re: 
Recipient is offline:

>And I was told to put the text into 5321 that pointed out that using 
>an inbound
>email message to trigger a delivery attempt for messages in the
>queue might avoid the need for ETRN (or TURN in some cases).  I
>didn't make that up although I've seen it put to good use.

For this to work, there would need to be a list maintained by the 
delivering MTA  that lists which other MTAs it has tried to deliver 
to which failed due to failure to connect (along with a queue of 
messages that need to be delivered to that MTA and a time to next 
attempt to connect). When a new message arrives and it gets delivered 
(since that connect attempt worked) then the queue of pending 
messages can then be immediately flushed to the MTA. How many MTAs 
have this type of support to force flush a pending queue when a 
message delivery succeeds? I know that ETRN support will do this but 
that is due to the intended recipient MTA having connected to the 
delivery MTA and saying to it "I'm here to receive the pending 
messages - Send them now".

From: [email-masked]
Cc: "[email-masked]" <[email-masked]>
Date: Fri, 02 Sep 2011 18:30:45 -0700 (PDT)
Subject: RE: Received: and sending IP
To: "Murray S. Kucherawy" <[email-masked]>
Sender: [email-masked]

> > -----Original Message-----
> > From: Ned Freed [mailto:[email-masked]]
> > Sent: Friday, September 02, 2011 4:25 PM
> > To: Rolf E. Sonneveld
> > Cc: [email-masked]; Murray S. Kucherawy; [email-masked]
> > Subject: Re: Received: and sending IP
> >
> > To the point where if we were to support such a change at this point, it would
> > be optional, off by default, and I doubt we'd recommend enabling it.

> I'm not really proposing to make this change in a 5322bis or anywhere else. 
> I'm just curious as to why the current syntax won out over something that seems
> less dissonant.

Well, I've already explained my understanding of why people supported the
approach back when this was debated. Since I disagree with that position (and
so did the one other person who has weighed in here) I think you need to look
to those who supported it for further justification and explanation.

In other words, I'm shutting up now.

				Ned

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Fri, 2 Sep 2011 17:30:29 -0700
Subject: RE: Received: and sending IP
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: Ned Freed [mailto:[email-masked]]
> Sent: Friday, September 02, 2011 4:25 PM
> To: Rolf E. Sonneveld
> Cc: [email-masked]; Murray S. Kucherawy; [email-masked]
> Subject: Re: Received: and sending IP
> 
> To the point where if we were to support such a change at this point, it would
> be optional, off by default, and I doubt we'd recommend enabling it.

I'm not really proposing to make this change in a 5322bis or anywhere else.  I'm just curious as to why the current syntax won out over something that seems less dissonant.

Date: Fri, 02 Sep 2011 20:08:57 -0400
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

--On Friday, September 02, 2011 15:52 -0400 Hector Santos
<[email-masked]> wrote:

> 
> John C Klensin wrote:
>>> I don't know why I didn't think of it before, but UUCP is
>>> still available as an option, at least for FreeBSD, and
>>> probably works just fine.  Expect funny looks when you tell
>>> people you're using it, though...
>> 
>> So, to save Hector or others from writing a note, is Fidonet
>> in parts of the world where it is still useful.  
> 
> Cheap shot. I was all prepared to not further add to the
> obvious. But if that is a question you have above:

Hector, it was intended as a complement and recognition of your
work in that area, not as a cheap shot (or any other type of
shot) at all.  We have never discussed it, but FidoNet was very
important to some work I was involved in some years ago and was
absolutely critical in getting a lot of people who were involved
in that context onto the net.   It was, and is, a really good
protocol for its purpose (and some other purposes).   Absolutely
no disrespect intended... precisely the opposite.

As should have been clear from other comments I've made in this
thread, I don't consider SMTP-over-TCP to be really optimal for
intermittently-connected server.  It is usually appropriate for
well-connected hosts who meet other requirements, but, as a last
mile (or, for that matter, first-mile if 4409 isn't appropriate)
arrangement for intermittently-connected systems that hold
actual user mailboxes, lots of other things --including UUCP,
FidoNet, and an assortment of "batch SMTP" arrangements (some
involving writing onto physical media and then transporting
those media)-- are typically a lot more useful... even if that
delivery/mailbox host then offer POP or IMAP connections for
people to pick up mail and especially if it does not.

> Yes, in small numbers, and surprising that it is to us, we
> still have people buying:
> 
>     http://www.santronics.com/products/pxpress
 
> Its no longer developed, we do as little support as needed and
> encourage user-support to help themselves.  The difference
> John, is the addition of the internet layer.  So dialup is a
> backup, not the main frontend. So instead of seeing a
> telephone # in the Fidonet Nodelist, you have an IP address.
>...

A very significant and important improvement, IMO.  Presumably,
if a telephone# still appears and is supported at both ends, it
continues to work.

> Its a reflection of the economy and people reestablishing
> their own little BBS, business and services A.K.A Mom and Pop,
> Small Businesses.
> 
> Yet, I got the IBM's with Dialup Store and Forward setups in
> California, Texas and I need to make a tech call very shortly.
> 
> Why Dialup?
> 
># 1, its one way to win a fast entry contract with the feds as
># the exception to HIPAA secured internet transfer requirements.

That application had not occurred to me.  I find it fascinating
as well as worthwhile.  The other two examples surprise me less.

 best,
    john

From: [email-masked]
Cc: [email-masked], "Murray S. Kucherawy" <[email-masked]>, "[email-masked]" <[email-masked]>
Date: Fri, 02 Sep 2011 16:24:53 -0700 (PDT)
Subject: Re: Received: and sending IP
To: "Rolf E. Sonneveld" <[email-masked]>
Sender: [email-masked]

> On 9/2/11 11:18 PM, [email-masked] wrote:
> >
> >>> -----Original Message-----
> >>> From: Frank Ellermann [mailto:[email-masked]]
> >>> Sent: Friday, September 02, 2011 12:50 PM
> >>> To: Murray S. Kucherawy
> >>> Cc: [email-masked]
> >>> Subject: Re: Received: and sending IP
> >>>
> >>> IIRC many servers note the client (sender) IP as comment
> >>> only if it does not match any IP of the EHLO name, but I
> >>> guess that is not your question.
> >> Yes, it's a comment.  But still others show the name (if any) and the IP (in all cases).
> >> The question is really: Why a comment and not an additional preposition clause?
> > As I understand it, the feeling was that introducing a new formal clause
> > was likely to break extant parsers for this stuff.
> >
> > I personally did not and do not agree with this choice, but the consensus
> > went the other way, and now the installed base is kinda difficult to ignore.

> Yep. I have at least two customers which run core business applications
> in which they use (home-grown) code that parses the Received lines, to
> determine a date/time timestamp. I'm sure there are many others out
> there. Changing the format of the Received lines will cause problems
> here and there, no doubt.

To the point where if we were to support such a change at this point, it would
be optional, off by default, and I doubt we'd recommend enabling it.

				Ned

Date: Sat, 03 Sep 2011 00:40:58 +0200
From: "Rolf E. Sonneveld" <[email-masked]>
To: [email-masked]
Cc: "Murray S. Kucherawy" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: Received: and sending IP
Sender: [email-masked]

On 9/2/11 11:18 PM, [email-masked] wrote:
>
>>> -----Original Message-----
>>> From: Frank Ellermann [mailto:[email-masked]]
>>> Sent: Friday, September 02, 2011 12:50 PM
>>> To: Murray S. Kucherawy
>>> Cc: [email-masked]
>>> Subject: Re: Received: and sending IP
>>>
>>> IIRC many servers note the client (sender) IP as comment
>>> only if it does not match any IP of the EHLO name, but I
>>> guess that is not your question.
>> Yes, it's a comment.  But still others show the name (if any) and the IP (in all cases).
>> The question is really: Why a comment and not an additional preposition clause?
> As I understand it, the feeling was that introducing a new formal clause
> was likely to break extant parsers for this stuff.
>
> I personally did not and do not agree with this choice, but the consensus
> went the other way, and now the installed base is kinda difficult to ignore.

Yep. I have at least two customers which run core business applications 
in which they use (home-grown) code that parses the Received lines, to 
determine a date/time timestamp. I'm sure there are many others out 
there. Changing the format of the Received lines will cause problems 
here and there, no doubt.

/rolf

From: [email-masked]
Cc: "[email-masked]" <[email-masked]>
Date: Fri, 02 Sep 2011 14:18:11 -0700 (PDT)
Subject: RE: Received: and sending IP
To: "Murray S. Kucherawy" <[email-masked]>
Sender: [email-masked]

> > -----Original Message-----
> > From: Frank Ellermann [mailto:[email-masked]]
> > Sent: Friday, September 02, 2011 12:50 PM
> > To: Murray S. Kucherawy
> > Cc: [email-masked]
> > Subject: Re: Received: and sending IP
> >
> > IIRC many servers note the client (sender) IP as comment
> > only if it does not match any IP of the EHLO name, but I
> > guess that is not your question.

> Yes, it's a comment.  But still others show the name (if any) and the IP (in all cases).

> The question is really: Why a comment and not an additional preposition clause?

As I understand it, the feeling was that introducing a new formal clause
was likely to break extant parsers for this stuff.

I personally did not and do not agree with this choice, but the consensus
went the other way, and now the installed base is kinda difficult to ignore.

				Ned

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Fri, 2 Sep 2011 12:54:29 -0700
Subject: RE: Received: and sending IP
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: Frank Ellermann [mailto:[email-masked]]
> Sent: Friday, September 02, 2011 12:50 PM
> To: Murray S. Kucherawy
> Cc: [email-masked]
> Subject: Re: Received: and sending IP
> 
> IIRC many servers note the client (sender) IP as comment
> only if it does not match any IP of the EHLO name, but I
> guess that is not your question.

Yes, it's a comment.  But still others show the name (if any) and the IP (in all cases).

The question is really: Why a comment and not an additional preposition clause?

Date: Fri, 02 Sep 2011 15:52:07 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
CC: [email-masked]
Subject: Re: Recipient is offline
To: [email-masked]
Sender: [email-masked]

John C Klensin wrote:
>> I don't know why I didn't think of it before, but UUCP is
>> still available as an option, at least for FreeBSD, and
>> probably works just fine.  Expect funny looks when you tell
>> people you're using it, though...
> 
> So, to save Hector or others from writing a note, is Fidonet in
> parts of the world where it is still useful.  

Cheap shot. I was all prepared to not further add to the obvious. But 
if that is a question you have above:

Yes, in small numbers, and surprising that it is to us, we still have 
people buying:

    http://www.santronics.com/products/pxpress

Its no longer developed, we do as little support as needed and 
encourage user-support to help themselves.  The difference John, is 
the addition of the internet layer.  So dialup is a backup, not the 
main frontend. So instead of seeing a telephone # in the Fidonet 
Nodelist, you have an IP address.

Its a reflection of the economy and people reestablishing their own 
little BBS, business and services A.K.A Mom and Pop, Small Businesses.

Yet, I got the IBM's with Dialup Store and Forward setups in 
California, Texas and I need to make a tech call very shortly.

Why Dialup?

#1, its one way to win a fast entry contract with the feds as the 
exception to HIPAA secured internet transfer requirements.

#2, you can't force people (even BIG companies) to change and invest 
money on things they don't need and can get the job done more cheaply 
in old traditional ways.

#3, you can save money, like high bandwidth internet cost, but 
switching to dialup on certain applications. i.e. $30K to $3K per 
month as one example.

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: Frank Ellermann <[email-masked]>
Date: Fri, 2 Sep 2011 21:49:56 +0200
Subject: Re: Received: and sending IP
To: "Murray S. Kucherawy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sender: [email-masked]

On 2 September 2011 18:48, Murray S. Kucherawy wrote:

> So if someone can spare a few cycles

IIRC many servers note the client (sender) IP as comment
only if it does not match any IP of the EHLO name, but I
guess that is not your question.

-Frank

Subject: Re: Recipient is offline
From: Keith Moore <[email-masked]>
Date: Fri, 2 Sep 2011 15:15:13 -0400
Cc: [email-masked], [email-masked]
To: "John Levine" <[email-masked]>
Sender: [email-masked]

On Sep 2, 2011, at 1:14 PM, John Levine wrote:

> More to the point, SMTP is designed to transfer mail to servers that
> are online and available most of the time.  The retry stuff is
> intended to deal with temporary overloads and failures, not a system
> that is only connected for part of each day.
> 
> Twenty years ago, there were still a lot of part time connections.
> There's nothing magic about SMTP as a way to get mail from one host to
> another, and we had a variety of hacks to get mail to them, from TURN
> to uucp to, in extreme cases, magtapes sent by mail.  All the hacks
> are still available for people who need them.  This isn't an SMTP
> problem.

+1.  UUCP was tailor-made for this, works well, and AFAIK high-quality, zero-cost implementations are still available.  

Keith

Date: Fri, 02 Sep 2011 14:48:56 -0400
From: John C Klensin <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>, [email-masked]
Subject: RE: Recipient is offline
Sender: [email-masked]

--On Friday, September 02, 2011 10:21 -0700 "Murray S.
Kucherawy" <[email-masked]> wrote:

> ...
>> Twenty years ago, there were still a lot of part time
>> connections. There's nothing magic about SMTP as a way to get
>> mail from one host to another, and we had a variety of hacks
>> to get mail to them, from TURN to uucp to, in extreme cases,
>> magtapes sent by mail.  All the hacks are still available for
>> people who need them.  This isn't an SMTP problem.

> I don't know why I didn't think of it before, but UUCP is
> still available as an option, at least for FreeBSD, and
> probably works just fine.  Expect funny looks when you tell
> people you're using it, though...

So, to save Hector or others from writing a note, is Fidonet in
parts of the world where it is still useful.  BITNET mail is
pretty much gone, and there aren't a lot of nine-track tape
drives left around and in active use, but I had a mail
spool/queue/batch delivered via CDROM and an express carrier
less than a year ago.  If the alternative is not getting the
mail through in a reasonably timely way, I think we can all
learn to ignore the funny looks :-)

But that was exactly where the rules came from that an
intermediate relay, reached through a low-preference (high code
value) MX, needed to accept responsibility for the message but
were not required to deliver if via SMTP-over-TCP even if a
higher-preference MX was present.  If the operator of that relay
knows that some hack will get the mail to the delivery host more
efficiently than using SMTP-over-TCP, there are no 5321
restrictions on her taking advantage of that knowledge.

    john

Date: Fri, 02 Sep 2011 10:21:54 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: John Levine <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

John Levine wrote:
> ... and we had a variety of hacks to get mail to them, from TURN
> to uucp to, in extreme cases, magtapes sent by mail.

"Never underestimate the bandwidth of a pickup truck."

<csg>

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Fri, 2 Sep 2011 10:21:18 -0700
Subject: RE: Recipient is offline
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of John Levine
> Sent: Friday, September 02, 2011 10:15 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: Recipient is offline
> 
> Twenty years ago, there were still a lot of part time connections.
> There's nothing magic about SMTP as a way to get mail from one host to
> another, and we had a variety of hacks to get mail to them, from TURN
> to uucp to, in extreme cases, magtapes sent by mail.  All the hacks
> are still available for people who need them.  This isn't an SMTP
> problem.

I don't know why I didn't think of it before, but UUCP is still available as an option, at least for FreeBSD, and probably works just fine.  Expect funny looks when you tell people you're using it, though...

Date: 2 Sep 2011 17:14:39 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: Recipient is offline
Organization: 
Sender: [email-masked]

More to the point, SMTP is designed to transfer mail to servers that
are online and available most of the time.  The retry stuff is
intended to deal with temporary overloads and failures, not a system
that is only connected for part of each day.

Twenty years ago, there were still a lot of part time connections.
There's nothing magic about SMTP as a way to get mail from one host to
another, and we had a variety of hacks to get mail to them, from TURN
to uucp to, in extreme cases, magtapes sent by mail.  All the hacks
are still available for people who need them.  This isn't an SMTP
problem.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Fri, 2 Sep 2011 09:48:36 -0700
Subject: Received: and sending IP
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

--_000_F5833273385BB34F99288B3648C4F06F13512DFA4AEXCHC2corpclo_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

A while ago I asked a few of the DRUMS veterans off-list why it was that th=
ey chose to record the client IP address in a parenthesized (and thus, comm=
ent) part of a Received: field in RFC5321.  The answers I got were of the w=
e-know-it's-weird, held-our-noses form, but alas I can't find the emails an=
ymore.

So if someone can spare a few cycles, can I get a refresher?  In particular=
, since the form of Received: is basically a set of <preposition> <data> pa=
irs (e.g., "from foo by bar with baz for user"), why not add a "via" or suc=
hlike and stick the IP information there?

Thanks,
-MSK

--_000_F5833273385BB34F99288B3648C4F06F13512DFA4AEXCHC2corpclo_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head><meta http-equiv=3DContent-Type content=
=3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator content=3D"Micros=
oft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue vli=
nk=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal>A while ago I as=
ked a few of the DRUMS veterans off-list why it was that they chose to reco=
rd the client IP address in a parenthesized (and thus, comment) part of a R=
eceived: field in RFC5321.&nbsp; The answers I got were of the we-know-it&#=
8217;s-weird, held-our-noses form, but alas I can&#8217;t find the emails a=
nymore.<o:p></o:p></p><p class=3DMsoNormal><o:p>&nbsp;</o:p></p><p class=3D=
MsoNormal>So if someone can spare a few cycles, can I get a refresher?&nbsp=
; In particular, since the form of Received: is basically a set of &lt;prep=
osition&gt; &lt;data&gt; pairs (e.g., &#8220;from foo by bar with baz for u=
ser&#8221;), why not add a &#8220;via&#8221; or suchlike and stick the IP i=
nformation there?<o:p></o:p></p><p class=3DMsoNormal><o:p>&nbsp;</o:p></p><=
p class=3DMsoNormal>Thanks,<o:p></o:p></p><p class=3DMsoNormal>-MSK<o:p></o=
:p></p></div></body></html>=

--_000_F5833273385BB34F99288B3648C4F06F13512DFA4AEXCHC2corpclo_--

Date: Fri, 2 Sep 2011 18:34:34 +0200
From: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

ETRN is good. It diminishes the chances that people receive mail out of 
order. When old mail waits a half-hour after the line comes up while new 
mail arrives, misunderstandings happen.

Arnt

Date: Fri, 02 Sep 2011 11:56:07 -0400
From: John C Klensin <[email-masked]>
To: Tony Finch <[email-masked]>
cc: Mark Andrews <[email-masked]>, Paul Smith <[email-masked]>, Menachem Moystoviz <[email-masked]>, [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

--On Friday, September 02, 2011 16:43 +0100 Tony Finch
<[email-masked]> wrote:

> John C Klensin <[email-masked]> wrote:
>> Mark Andrews <[email-masked]> wrote:
>> > 
>> > TURN is useful if the is not a direct incoming path. e.g.
>> > Running multiple MTA's behind a firewall/nat that blocks
>> > incoming connections. ETRN is useful if the is a direct
>> > incoming path.
>> 
>> Mark, I said "it was deprecated".  I didn't say that there
>> were not arguments for keeping it for special circumstances,
>> nor that there were not other tradeoffs.  The WG considered
>> those issues, deprecated it, and introduced ETRN to deal with
>> the pieces of the problem that affected the public Internet.
> 
> There's also ODMR / ATRN - RFC 2645.

Sure.  And maybe other things.  If you think ATRN should be
mentioned along with ETRN in 5321bis (assuming that eventually
happens), I recommend generating an erratum that says so.
Similarly for any other IETF-approved technique for stimulating
a potential sender into action.    Because 5321 has already gone
down the path of saying "this is a way to do this, I don't mind
adding more even though I think that whomever suggests such a
reference should be prepared to demonstrate that the method /
spec to be referenced is widely deployed and tested sufficiently
to justify an explicit pointer from a full standard.

On way to describe it is that any pointer to a technique from
5321bis should be treated as a recommendation for a tested,
established, and deployed protocol, not an advertisement for
something that folks think is a good idea.

For those interested in that distinction, see the recent
discussions of 4409bis on the YAM and IETF mailing lists.  For
2645, I think the community would probably insist on the
equivalent of an implementation report whether there was an
effort to move it to DS or not.

    john

Date: Fri, 2 Sep 2011 16:43:57 +0100
From: Tony Finch <[email-masked]>
To: John C Klensin <[email-masked]>
cc: Mark Andrews <[email-masked]>, Paul Smith <[email-masked]>, Menachem Moystoviz <[email-masked]>, [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

John C Klensin <[email-masked]> wrote:
> Mark Andrews <[email-masked]> wrote:
> >
> > TURN is useful if the is not a direct incoming path. e.g. Running
> > multiple MTA's behind a firewall/nat that blocks incoming connections.
> > ETRN is useful if the is a direct incoming path.
>
> Mark, I said "it was deprecated".  I didn't say that there were
> not arguments for keeping it for special circumstances, nor that
> there were not other tradeoffs.  The WG considered those issues,
> deprecated it, and introduced ETRN to deal with the pieces of
> the problem that affected the public Internet.

There's also ODMR / ATRN - RFC 2645.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Northwest FitzRoy, Sole: Southwesterly 4 or 5, increasing 5 to 7, veering
northwesterly later, then decreasing 4 or 5 in Sole. Moderate becoming rough.
Rain or showers. Moderate or good, occasionally poor.

Date: Fri, 02 Sep 2011 10:08:30 -0400
From: John C Klensin <[email-masked]>
To: Mark Andrews <[email-masked]>
cc: Paul Smith <[email-masked]>, Menachem Moystoviz <[email-masked]>, [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

--On Friday, September 02, 2011 12:44 +1000 Mark Andrews
<[email-masked]> wrote:

>...
>> That is one of several reasons why TURN was deprecated years
>> ago.
> 
> TURN is useful if the is not a direct incoming path.
> e.g. Running multiple MTA's behind a firewall/nat that blocks
> incoming connections.
> ETRN is useful if the is a direct incoming path.

Mark, I said "it was deprecated".  I didn't say that there were
not arguments for keeping it for special circumstances, nor that
there were not other tradeoffs.  The WG considered those issues,
deprecated it, and introduced ETRN to deal with the pieces of
the problem that affected the public Internet.  And I was told
to put the text into 5321 that pointed out that using an inbound
email message to trigger a delivery attempt for messages in the
queue might avoid the need for ETRN (or TURN in some cases).  I
didn't make that up although I've seen it put to good use.

> Horses for courses.  That said ETRN is probably more applicable
> here as there is a direct incoming path.
> 
> Backscatter is a problem for any backup MX.

A lot of things are "problems" for any relay SMTP that does not
have access to information about what the delivery server will
do.  Of course, nothing in the protocols prevents the delivery
server from sharing lists of active mailboxes and their status
with the relay server.  If that were done, then backscatter is
no more of a risk with the relay server (or backup MX, if you
prefer) than it would be with the final delivery server.   The
situation is all about tradeoffs; I content that almost any
relay situation is, independent of protocol.

> Note the "backup MX" may be the only MX listed to the public
> and the backup MX can see lower priority MX records.

Sure.  And 5321 (and its predecessors) are quite deliberately
written to not require than an intermediate MX attempt delivery
over SMTP-over-TCP.  It is perfectly acceptable for it to
arrange to get the messages to the delivery system over UUCP, or
rsync, or, for that matter, semaphore flags, even if there are
lower-preference MX records in the DNS (public, private, or
otherwise).

    john

Date: Fri, 02 Sep 2011 01:17:12 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Mark Andrews <[email-masked]>
CC: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

Mark Andrews wrote:

> Horses for courses.  That said ETRN is probably more applicable
> here as there is a direct incoming path.

+1.  It helps support impatience operators who can't wait for the next 
hour :)

-- 
Sincerely

Hector Santos
http://www.santronics.com

To: John C Klensin <[email-masked]>
Cc: Paul Smith <[email-masked]>, Menachem Moystoviz <[email-masked]>, [email-masked]
From: Mark Andrews <[email-masked]>
Subject: Re: Recipient is offline
Date: Fri, 02 Sep 2011 12:44:25 +1000
Sender: [email-masked]

In message <[email-masked]>, John C Klensin writes:
> 
> 
> 
> --On Friday, September 02, 2011 09:32 +1000 Mark Andrews
> <[email-masked]> wrote:
> 
> >> However, if a mail server is online for only a few hours a
> >> day, then I'd  seriously suggest getting a backup mail server
> >> to accept messages during  the downtime (or putting the main
> >> mail server somewhere else with a  better connection), unless
> >> you don't really care about getting mail...
> > 
> > And poll when coming online using TURN after authenticating.
> > 
> > Also if you do this you should ensure that your backup MX can
> > reject non valid addresses or you will generate a lot of
> > backscatter. There are a number of different techniques that
> > can be used to do this.
> 
> That is one of several reasons why TURN was deprecated years ago.

TURN is useful if the is not a direct incoming path.
e.g. Running multiple MTA's behind a firewall/nat that blocks incoming
connections.
ETRN is useful if the is a direct incoming path.

Horses for courses.  That said ETRN is probably more applicable
here as there is a direct incoming path.

Backscatter is a problem for any backup MX.

Note the "backup MX" may be the only MX listed to the public and
the backup MX can see lower priority MX records.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

Date: Thu, 01 Sep 2011 21:57:43 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

John C Klensin wrote:
> --On Friday, September 02, 2011 09:32 +1000 Mark Andrews
> <[email-masked]> wrote:
> 
>>> However, if a mail server is online for only a few hours a
>>> day, then I'd  seriously suggest getting a backup mail server
>>> to accept messages during  the downtime (or putting the main
>>> mail server somewhere else with a  better connection), unless
>>> you don't really care about getting mail...
>> And poll when coming online using TURN after authenticating.
>>
>> Also if you do this you should ensure that your backup MX can
>> reject non valid addresses or you will generate a lot of
>> backscatter. There are a number of different techniques that
>> can be used to do this.
> 
> That is one of several reasons why TURN was deprecated years ago.
> 
> :-(

John,  the natives are coming back!  Just in the last 2-3 weeks, at 
least 5-6 people have setup SMTP for non 24x7 operations. Asking the 
same related questions. The economy has forced many to set up SOHO 
shops and are using the new powerful IPTV high speeds setups.

Just had one today where I think he will need us to be his PORT 587 
SMART HOST to get around PORT 25 outbound restrictions.  We allow ETRN 
here and for the operator to use in local mode to start sending 
specific queues.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 01 Sep 2011 21:04:37 -0400
From: John C Klensin <[email-masked]>
To: Mark Andrews <[email-masked]>, Paul Smith <[email-masked]>
cc: Menachem Moystoviz <[email-masked]>, [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

--On Friday, September 02, 2011 09:32 +1000 Mark Andrews
<[email-masked]> wrote:

>> However, if a mail server is online for only a few hours a
>> day, then I'd  seriously suggest getting a backup mail server
>> to accept messages during  the downtime (or putting the main
>> mail server somewhere else with a  better connection), unless
>> you don't really care about getting mail...
> 
> And poll when coming online using TURN after authenticating.
> 
> Also if you do this you should ensure that your backup MX can
> reject non valid addresses or you will generate a lot of
> backscatter. There are a number of different techniques that
> can be used to do this.

That is one of several reasons why TURN was deprecated years ago.

:-(

   john

Date: Thu, 01 Sep 2011 21:03:56 -0400
From: John C Klensin <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>, [email-masked]
Subject: RE: Recipient is offline
Sender: [email-masked]

--On Thursday, September 01, 2011 16:14 -0700 "Murray S.
Kucherawy" <[email-masked]> wrote:

>...
>> > Because this would mean that any user who turns his SMTP
>> > server on for a couple of hours each day would receive his
>> > mail.
> 
> As long as there's a retry during that window, yes.  But that
> isn't guaranteed.

And, if the sender is doing with 5321 (and 2821 and 1123)
suggest (I'm not going to have a discussion about what is and is
not normative in 5321, at least today), the odds of a
rarely-available receiver getting delivery go down rapidly with
time.  Even if the sender keeps trying for more than the
recommended 4-5 days, incremental back off strategies may make a
short receiver-online connection period an increasingly elusive
target.  

See the discussion in RFC 5321, Section 4.5.4.1, for additional
hints about this or, as others have suggested, seriously
consider an intermediate server that is usually or always on-net
and a mechanism for working with it when your
intermittently-connected host is online.  Those could be as
simple as having the intermediate server know the typical
schedule for the intermittent host being online as well as
mechanisms like ETRN or pinging the intermediate with some sort
of message.

    john

> 
>> I'd say that most probably try for 3 days (gives admins
>> chance to fix their server after a weekend if it broke on the
>> Friday evening). Busy ones like gmail could try less because
>> they have so much mail, and don't care as it's a free
>> service. As long as they generate an NDR it's compliant with
>> the standard.
> 
> After some little-known sysadmin trickery I learned a long
> time ago[*], I managed to figure out that they do indeed try
> for at least a couple of days, though the frequency of retries
> is not constant.  They certainly don't give up after only a
> few seconds.
> 
>> However, if a mail server is online for only a few hours a
>> day, then I'd seriously suggest getting a backup mail server
>> to accept messages during the downtime (or putting the main
>> mail server somewhere else with a better connection), unless
>> you don't really care about getting mail...
> 
> I concur, especially if it's one that lets you do ETRN
> (RFC1985) or some other demand-based delivery system.
> 
> -MSK
> 
> [*] I asked them.
> 

Date: Thu, 01 Sep 2011 20:27:49 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

Paul Smith wrote:

> Well, the sender should retry, but there is no rule about how often or 
> for how long it should retry, just a vague suggestion (not even 'SHOULD')
> 
> If gmail gives up after a few seconds, then I'm a bit dubious. I'd 
> expect it to try longer than that, I'd have thought for at least a day 
> (unless it gets a permanent error report back from somewhere).

Well, I have seen it number of time with non 24x7 SMTP customers 
complaining about not getting mail and we found out that the senders 
did retry - to a 2nd MX the customer had no idea was active, like his 
ISP MX and from there it went no where.

So its possible Google tried to connect to his MX, failed, tried the 
2nd MX and the transaction was done.  Bob will never see it. :)

-- 
Sincerely

Hector Santos
http://www.santronics.com

To: Paul Smith <[email-masked]>
Cc: Menachem Moystoviz <[email-masked]>, [email-masked]
From: Mark Andrews <[email-masked]>
Subject: Re: Recipient is offline
Date: Fri, 02 Sep 2011 09:32:07 +1000
Sender: [email-masked]

In message <[email-masked]>, Paul Smith writes:
> 
> On 01/09/2011 15:35, Menachem Moystoviz wrote:
> > Suppose one person (Let's call her Alice) wants to send an email to
> > another (Let's call him Bob).
> > Now suppose Bob's server is online for 4 hours every day.
> > According to section 4.5.4.1, Alice's server (or whatever server is
> > closest to Bob)
> > should retry sending the mail every half hour or so for 4-5 days,
> > meaning that Bob will eventually get the mail.
> > However, experience with Google's Gmail service has showed me that
> > usually, the server gives up after several seconds.
> > Is my sample biased or is this the norm, in violation of the RFC?
> > Moreover, will Bob receive the mail in my example?
> > Because this would mean that any user who turns his SMTP server on for
> > a couple of hours each day would receive his mail.
> >
> 
> Well, the sender should retry, but there is no rule about how often or 
> for how long it should retry, just a vague suggestion (not even 'SHOULD')
> 
> If gmail gives up after a few seconds, then I'm a bit dubious. I'd 
> expect it to try longer than that, I'd have thought for at least a day 
> (unless it gets a permanent error report back from somewhere).
> 
> I'd say that most probably try for 3 days (gives admins chance to fix 
> their server after a weekend if it broke on the Friday evening). Busy 
> ones like gmail could try less because they have so much mail, and don't 
> care as it's a free service. As long as they generate an NDR it's 
> compliant with the standard.
> 
> However, if a mail server is online for only a few hours a day, then I'd 
> seriously suggest getting a backup mail server to accept messages during 
> the downtime (or putting the main mail server somewhere else with a 
> better connection), unless you don't really care about getting mail...

And poll when coming online using TURN after authenticating.

Also if you do this you should ensure that your backup MX can reject
non valid addresses or you will generate a lot of backscatter.
There are a number of different techniques that can be used to do
this.

Mark

> For a single MTA I'd say that it has to be online for 'pretty much all 
> the time'. Offline for an hour or two for maintenance over a few weeks 
> should be OK in my experience (eg 99.5% uptime).
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Thu, 1 Sep 2011 16:14:16 -0700
Subject: RE: Recipient is offline
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

 -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Smith
> Sent: Thursday, September 01, 2011 9:10 AM
> To: Menachem Moystoviz
> Cc: [email-masked]
> Subject: Re: Recipient is offline
> 
> On 01/09/2011 15:35, Menachem Moystoviz wrote:
> > Suppose one person (Let's call her Alice) wants to send an email to
> > another (Let's call him Bob).

Yes, I've seen them around.

> > However, experience with Google's Gmail service has showed me that
> > usually, the server gives up after several seconds.

I'm almost certain that's not the case.  I would be surprised to see Gmail tries and gives up only a couple of times in a short period.

> > Is my sample biased or is this the norm, in violation of the RFC?

The RFC (5321 replaced 2821, by the way), but as Paul points out there's no normative requirement to try a specific number of times or for a specific duration, just some guidelines.

> > Moreover, will Bob receive the mail in my example?

More than likely.

> > Because this would mean that any user who turns his SMTP server on for
> > a couple of hours each day would receive his mail.

As long as there's a retry during that window, yes.  But that isn't guaranteed.

> I'd say that most probably try for 3 days (gives admins chance to fix
> their server after a weekend if it broke on the Friday evening). Busy
> ones like gmail could try less because they have so much mail, and don't
> care as it's a free service. As long as they generate an NDR it's
> compliant with the standard.

After some little-known sysadmin trickery I learned a long time ago[*], I managed to figure out that they do indeed try for at least a couple of days, though the frequency of retries is not constant.  They certainly don't give up after only a few seconds.

> However, if a mail server is online for only a few hours a day, then I'd
> seriously suggest getting a backup mail server to accept messages during
> the downtime (or putting the main mail server somewhere else with a
> better connection), unless you don't really care about getting mail...

I concur, especially if it's one that lets you do ETRN (RFC1985) or some other demand-based delivery system.

-MSK

[*] I asked them.

Date: Thu, 01 Sep 2011 17:10:12 +0100
From: Paul Smith <[email-masked]>
To: Menachem Moystoviz <[email-masked]>
CC: [email-masked]
Subject: Re: Recipient is offline
Sender: [email-masked]

On 01/09/2011 15:35, Menachem Moystoviz wrote:
> Suppose one person (Let's call her Alice) wants to send an email to
> another (Let's call him Bob).
> Now suppose Bob's server is online for 4 hours every day.
> According to section 4.5.4.1, Alice's server (or whatever server is
> closest to Bob)
> should retry sending the mail every half hour or so for 4-5 days,
> meaning that Bob will eventually get the mail.
> However, experience with Google's Gmail service has showed me that
> usually, the server gives up after several seconds.
> Is my sample biased or is this the norm, in violation of the RFC?
> Moreover, will Bob receive the mail in my example?
> Because this would mean that any user who turns his SMTP server on for
> a couple of hours each day would receive his mail.
>

Well, the sender should retry, but there is no rule about how often or 
for how long it should retry, just a vague suggestion (not even 'SHOULD')

If gmail gives up after a few seconds, then I'm a bit dubious. I'd 
expect it to try longer than that, I'd have thought for at least a day 
(unless it gets a permanent error report back from somewhere).

I'd say that most probably try for 3 days (gives admins chance to fix 
their server after a weekend if it broke on the Friday evening). Busy 
ones like gmail could try less because they have so much mail, and don't 
care as it's a free service. As long as they generate an NDR it's 
compliant with the standard.

However, if a mail server is online for only a few hours a day, then I'd 
seriously suggest getting a backup mail server to accept messages during 
the downtime (or putting the main mail server somewhere else with a 
better connection), unless you don't really care about getting mail...

For a single MTA I'd say that it has to be online for 'pretty much all 
the time'. Offline for an hour or two for maintenance over a few weeks 
should be OK in my experience (eg 99.5% uptime).

From: Menachem Moystoviz <[email-masked]>
Date: Thu, 1 Sep 2011 17:35:18 +0300
Subject: Recipient is offline
To: [email-masked]
Sender: [email-masked]

I apologize in advance if I have violated any norms or rules with this email
After having read the ESMTP RFC(2821) and searching for this problem
on Google, I still don't have an answer.
Suppose one person (Let's call her Alice) wants to send an email to
another (Let's call him Bob).
Now suppose Bob's server is online for 4 hours every day.
According to section 4.5.4.1, Alice's server (or whatever server is
closest to Bob)
should retry sending the mail every half hour or so for 4-5 days,
meaning that Bob will eventually get the mail.
However, experience with Google's Gmail service has showed me that
usually, the server gives up after several seconds.
Is my sample biased or is this the norm, in violation of the RFC?
Moreover, will Bob receive the mail in my example?
Because this would mean that any user who turns his SMTP server on for
a couple of hours each day would receive his mail.

Thank you in advance,

Menachem D. Mostowicz (Gesh)