Subject: [ietf-smtp] Reject messages on backup mail exchangers when primary MX is online

From: Evert Mouw <[email-masked]>
Date: Sat, 23 Feb 2013 18:57:43 +0100

Just an idea... maybe it's silly or not worth investigating, but I
thought it wouldn't hurt to ask for comments.

Thanks, Evert

# Reject messages on backup mail exchangers when primary MX is online

---

TODO:

- Cascading? Like: test all MXs with lower preference number.
- Check with [RFC 5321].
- Error messages: correct numbers? Ask for opinions.
  Negative replies can be permanent (5xx codes) or transient (4xx codes).
- Discuss in newsgroup.
- Test.
- Write RFC.

---

Author: Evert Mouw <[email-masked]>

History:

- 2013-02-06 first version

## Problem

Multiple incoming mail servers (Mail eXchangers; MX) may be configured
for a DNS domain. The MX with the highest priority, that is, the
*lowest* preference number, MUST be contacted ([RFC 5321]) by the
mailserver of another domain if that other mailserver wants to send mail.

The MX with the lowest priority number is therefore called the *primary*
MX. The other MXs are *backup* servers. The backup servers will be used
by the sending party if the primary MX cannot be reached for whatever
reason. In practice, reasons might, for example, be related to network
configuration errors or hardware failure.

Spammers have found that sending spam to both the primary MX and one or
multiple backup MXs often works great to increase the number of spam
messages being delivered. Often, backup MXs are less well configured to
stop SPAM. Even if they are well configured, clever spam messages might
be delivered by both the primary and the backup MXs, resulting in
multiple spam messages for a receiver.

Many mail administrators have responded by removing the backup MXs
alltogether. The added costs of more spam, electricity and maintenance
cost does not rationalize the availability of a backup MX. When the
primary MX is offline, than the mails for the domain will be queued by
the mailservers of the sending parties.

This has major drawbacks. First, it places the costs of the storage and
retries to the sending party while that party is not responsible for the
downtime of the receiving mailserver. Second, when the primary MX is
offline for too long, messages might be lost. Third, messages might be
delayed for a long time, even after the primary MX did come back online.

The proper way to address this issue is to deny the use of a backup MX
when the primary MX is online.

## Bad solutions

Some administrators run a periodic script (e.g. cronjob) on the backup
MX to test if the primary MX is online (e.g. netcat to port 25 of the
primary MX). If the primary MX is offline, they dynamically add a
firewall rule to allow incoming port 25 traffic, or they start the SMTP
daemon. Then the primary becomes online again, they block incoming 25
traffic and flush the queue and stop the SMTP daemon.

This causes "mailserver unreachable" errors on legitimately configured
mailservers when the primary MX is online but not reachable due to some
network error on the side of the sender.

It also is a bad solution for multiple-domain mail exchangers.

## One interesting solution

An alternative implementation uses the ETRN command. The primary MX
sends periodically an ETRN request to the backup MXs. The backup MXs
will only become active when they did not receive such an ETRN request
for a preconfigured period of time (e.g., 5 minutes). See the
[hMailServer discussion].

## Proposed solution

On an incoming message, a backup MX should contact the primary MX to
determine whether it is online (e.g., using HELO / EHLO). If it is
online, then the backup MX should react with an error message 551,
indicating that the sending party should try the primary MX first.

	SMTP Error 551 <domain> Try the primary MX first while it is online;
please try <primary MX>

If the sender continues trying such requests, then optionally the backup
MX should periodically blacklist the sender, e.g. by rejecting it with
an error message 550 571 indicating that the sender has too often tried
to mis-use the backup MX.

	SMTP Error 550 5.7.1 You tried me, a backup MX, too often while the
primary MX is online.

Going even further, the offender could be placed on a public blacklist
such as [SpamCop].

## Gotcha's

### The preference debate

There is a so-called [preference debate] on how a sender should react
when the primary MX is not accepting mail of is offline. Some
implementers interpret the RFCs as stating "immediately try a MX with a
higher preference number, while others interpret it as "wait some time,
and then try a MX with a higher preference number". Also, one could
interpret it as "wait some time, and then try them all again, in order".

Some mail servers will wait to try the backup MX for some time after not
being able to deliver to the primary MX. So the following sequence could
happen:

1. Sender tries to contact primary MX and fails; queues message.
- Sender waits for some time.
- Primary MX comes online again.
- Sender tries backup MX.
- Backup MX replies with 551 because the primary MX is back online.
- Sender gives up.

Of course, the sender should not give up.

Some domains use [Nolisting] (Poor Man's Greylisting) and have a dummy
primary MX. Although the claim is made that Nolisting is RFC compliant,
I suggest that it is not in the spirit of the RFCs to list a false
primary MX in the DNS *on purpose*. However, my proposal does not
conflict with Nolisting because when the backup MX tests for
connectivity with the primary MX, it will fail to connect and thus
accept incoming mails. In practice, most MXs listed as backup MX in the
Nolisting approach will behave as if they were a primary MX.

---

[RFC 5321]: http://tools.ietf.org/html/rfc5321
[preference debate]:
http://en.wikipedia.org/wiki/MX_record#The_preference_debate
[SpamCop]: http://www.spamcop.net
[Nolisting]: http://nolisting.org/
[hMailServer discussion]:
http://www.hmailserver.com/forum/viewtopic.php?f=2&t=19439
[RFC 2821]: http://www.ietf.org/rfc/rfc2821.txt

---

From: "Alex van den Bogaerdt" <[email-masked]>
Date: Sat, 23 Feb 2013 19:32:17 +0100

> # Reject messages on backup mail exchangers when primary MX is online

Online for you maybe, but what if I cannot connect to it? Think about 
routing problems for instance. Am I not allowed to use the backup in such a 
case?

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Sat, 23 Feb 2013 20:27:50 +0100

Hi, Evert,

> multiple spam messages for a receiver.

Do you have any research results that could underpin this problem 
statement? Given the fact that most major ESPs have multiple MX records, 
I would expect that they would have minimized the number of MX records a 
long time ago, if this was really a big problem. Some figures:

outlook.com has 4 MXs, equal priorities
gmail.com has 5 MXs, different priorities
Postini use 4 MXs for their customer domains, different priorities
MessageLabs use 2 MXs, different priorities
...

If backup MXs are less well configured than their primary MX, the mail 
admin is the one to fix this problem. If they are not competent, chances 
are small they will read amendments to RFC5321.

/rolf

---

From: Evert Mouw <[email-masked]>
Date: Sat, 23 Feb 2013 20:34:01 +0100

Ales, Keith:

Thanks for your comments.

I speculated that a backup MX would be placed on another network, so
many of the NAT / routing problems would result in the backup MX also
not able to contact the primary MX. That would then "activate" the
backup MX. Routing problems in the senders' network would indeed cause
the backup MX to behave rude, but then the sender should fix his faulty
network. In fact, the backup MX giving an error that the primary MX is
reachable from the backup MX could help detect routing problems. (This
all assumes network seperation between the primary and the backup MX.)

Note, though, that I agree that routing problems could cause problems
and that the backup MX is a great tool to have an alternate means of
delivery. But nowadays many organisations just give up the backup MX
because of spam, so something must be done.

Maybe the idea could still be useful when some restrictions on its use
are introduced? Like MX network seperation and maybe a retry period or
just a warning ("primary is online") or, in a very softhearted manner,
accepting the message but marking it as possible spam.

Thanks for your thoughts.

Evert

---

From: "Carl S. Gutekunst" <[email-masked]>
Date: Sat, 23 Feb 2013 11:36:32 -0800

>> stop SPAM. 
> statement?

I'm only one data point, but my experience is the exact opposite. One of 
my honeypot systems has 5 MX records, the first four of which are 
directed to Postini, and the fifth that goes straight to the mailbox. 
I've been frankly surprised at the negligible amount of spam received by 
that 5th MX host. Even a sustained directory harvest attack (concurrent 
to an attack on my ssh server) last week didn't reach that 5th MX host.

<csg>

---

From: Evert Mouw <[email-masked]>
Date: Sat, 23 Feb 2013 21:23:20 +0100

Hi Ned,

> That's only true if you let it be true. There is no reason for the insertion
> of a secondary to interrupt the flow of information to the primary, so
> the information available once the message does get to the primary should
> be the same, resulting in the same accept/reject/discard decision being
> made.
> 
> True, it requires a certain amount of finesse, and some standards in this area
> would definitely help, but there is nothing intrinsicly impossible to do here.

Actually that is an interesting idea. The backup MX could act as an SMTP
proxy as long as it can maintain a connection with the primary. That
would also sort out your observation on address checking.

> Not even close. See above. At most the role of a check of the primary would
> be the adjust spam filters and perhaps some rate limiting.
> 
> I'm not going to bother to respond to your specific proposals since IMO the
> entire goal here is wrongminded.

Well, you seem to agree with the others on this list, so what can I say?
It was just an idea... It's hard to get rid of the spammers ;-)

It's too unfortunate that we have to calculate in that routing problems
could result in the primary MX not being reachable from the sender,
while a backup MX will still be reachable. I believe that such a
situation would be highly unlikely in many cases, but I agree that a
standard should cover all cases and be consistent with other standards.

Still maybe the "let the backup MX act as an SMTP proxy to the primary
MX when the primary MX is online" idea is useful, as is the idea to deny
backup MX use in some cases, but all of you convinced me that this
should not be in a standard.

Thanks for the feedback, all of you.

Evert

---

From: "Alex van den Bogaerdt" <[email-masked]>
Date: Sat, 23 Feb 2013 21:42:43 +0100

> not able to contact the primary MX. That would then "activate" the backup 
> MX.

I don't think life is that easy.  A can reach MX2, MX2 can reach MX1, does 
not mean A can reach MX1. The link between MX1 and MX2 may be accepting only 
"local" traffic (think east coast to west coast but not intercontinental), 
and not route A's traffic. If the link between A and MX1 is disrupted, for 
whichever reason, it's traffic does not reach MX1

Ofcourse the Internet is way more complex than this simple triangle I just 
depicted, but I think it proves my point.

> Routing problems in the senders' network would indeed cause
> network.

Another assumption I do not agree with. The problem may very well be in an 
intermediate network, peering agreements, and so on. It's not the fault of 
A, MX1 or MX2, neither can fix it. Then what?

> In fact, the backup MX giving an error that the primary MX is

That I'd consider to be feature creep.

> because of spam, so something must be done.

To 'prove' that A can reach MX2 and not MX1, even if MX2 can reach MX1, you 
would need to cooperation of A. Spammers won't cooperate, so you would need 
to rely on trustworthy hosts near A, on the same network or at least using 
the same peering.

Maybe the biggest mail providers can do that, but that's about it.

> are introduced? Like MX network seperation and maybe a retry period or

mailhosts on the same network should, IMHO, just use a failover 
configuration and appear to the rest of the Internet as one single MX. 
Probably as part of a load balancer.

> accepting the message but marking it as possible spam.

Combined with some bayesian logic perhaps. No need to mark all mail from MX2 
as spam if little to no mail comes via MX1.

my 2 cents again

---

From: "Alex van den Bogaerdt" <[email-masked]>
Date: Sat, 23 Feb 2013 21:44:22 +0100

> should not be in a standard.

it has been a while since I looked at this, so maybe my memory is blurred 
and wrong, but have a look at postfix

---

From: Evert Mouw <[email-masked]>
Date: Sat, 23 Feb 2013 22:03:08 +0100

Hi Alex,

> it has been a while since I looked at this, so maybe my memory is
> blurred and wrong, but have a look at postfix

Thanks. One could set up a Nginx SMTP proxy with XCLIENT turned on, and
use Postfix with XCLIENT support. Looks promising.

Evert

---

From: Keith Moore <[email-masked]>
Date: Sat, 23 Feb 2013 13:32:48 -0500

Bad idea.

- it changes the well-established semantics of MX records
- it assumes that the net is completely interconnected, but it's not.   
i.e. if the SMTP client tries to send to the primary MX and fails, it's 
not necessarily because the primary is down.  it might well be because 
the primary is unreachable due to some combination of NAT brain damage, 
firewall, link failure, routing failure.

individual domains can do what they like with their backup MXes, for 
better or worse, and live with the consequences.  but a general 
recommendation to turn off backup MXes when the primaries are up seems 
overbroad at best.

Ketih

On 02/23/2013 12:57 PM, Evert Mouw wrote:
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: Ned Freed <[email-masked]>
Date: Sat, 23 Feb 2013 11:34:55 -0800 (PST)

> mailserver of another domain if that other mailserver wants to send mail.

That MUST necessarily has to be interpreted in the context of the SMTP
client. That context may be entirely different from the context the 
mail exchanger is operating in. More specifically, it's entirely possible
for a mail exchanger providing secondary MX to be able to see the primary 
MX but for the client connected to the secondary not to be able to. There
are also a myriad of configurations that could lead a client to fail on sending
to a primary in such a way that it will fall back to the secondary. Indeed,
there are many sites out there that depend on such configurations.

As such, a check to see if the primary is visible from the secondary and
if it is failing the mail is *NOT* justified by the cited text. That doesn't
mean it's illegal to do - you can reject mail on the basis of flipping a coin
if you want to - but if you're looking to the standards to actually condone
this, you can forget it.

> configuration errors or hardware failure.

Which can be and often is asymmetric in nature.

That said, there is a real problem here, and that's the limits on the
ability to perform address checking on the secondary. This can result in
mail to invalid addresses being accepted when it shouldn't be.

This is a considerably harder problem, but it also can be addressed. And
once again some standards would help.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Sat, 23 Feb 2013 18:22:47 -0800 (PST)

> would also sort out your observation on address checking.

And when it can't it needs to be prepared to pass as much of the context of the
original connection to the primary as possible later when the primary becomes
available again. The IP address is especially important here, of course.

Again, it would be quite helpful if there was a standard for doing this.
Postfix has XCLIENT, we have something called XPEHLO, and there are
probably others out there.

I would certainly be willing to work on this if there is sufficient interest.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 24 Feb 2013 02:46:50 -0000

># Reject messages on backup mail exchangers when primary MX is online

On today's Internet, are backup MXes still useful?  Any decent mail
client should retry for at least a couple of days, and it's been a
very long time since my main MX was down for longer than that.

For a busy mail system, multiple mail hosts at the same priority make
plenty of sense, but a backup that accepts the mail, then redelivers
it to one of the main hosts later?  Really?

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Sat, 23 Feb 2013 18:52:34 -0800 (PST)

> very long time since my main MX was down for longer than that.

For many the question isn't how long it retries, but how often. A host with
intermittent connectivity problems may want to use a secondary MX to
in effect lower the retry interval.

> it to one of the main hosts later?  Really?

I'm not exactly a big operation, and I operate two of them. So: Yes. Really.

				Ned

---

From: Russ Allbery <[email-masked]>
Date: Sat, 23 Feb 2013 19:04:23 -0800

> very long time since my main MX was down for longer than that.

In the event of some sort of serious mail problem, such as a system crash,
I prefer to have all my queued mail on a host that I control.  I can then
redirect it somewhere else, hold it for arbitrarily long periods, or play
other tricks with it.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: hector <[email-masked]>
Date: Sat, 23 Feb 2013 23:18:23 -0500

It really doesn't have anything to do with "backups" per se as far as 
the sender is concern.  The available machines to deliver mail to is 
just that - a farm of available machines.  It can be a group of 
individual servers and machines - a inbound farm.  How the receiver deal 
with it is relevant to the sender who as far as its concern, the 
delivery was done. It finished the job.

Whats difference I believe today is how we went from scaling out (more 
machines) to scaling up (less machines, more CPUs), so under today 
systems, a host can have plenty of receivers read to take calls all in 
one or two machines.  The problem is still if one goes down (for 
whatever reason) is there another IP address available.  That is why the 
recommendation, just like it is for DNS, it is more ideal reliability 
wise to have two different network paths to do calls.  But the system 
has been robust with the built-in retries we have with can span days, so 
even the single machine/multiple cpu host machine are capable to 
handling mail as good as the rest.  We are also more reliable with the 
network unlike older days where there higher instances of connection 
drops and network delays.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sat, 23 Feb 2013 20:26:47 -0800

I'd support work like that, though probably by working on the document more
than the implementation.

I think the name should be OCLIENT in line with DSN's ORCPT, but that's a
minor point.

-MSK

---

From: hector <[email-masked]>
Date: Sun, 24 Feb 2013 00:00:57 -0500

> standard should cover all cases and be consistent with other standards.

There is really nothing primary vs backup as far as the mail sender is 
concern.

There is nothing that says one is better than the other.  All it is is a 
pool for the sender to use.  The preference is just that, a "hint" for 
the sender to begin with.   The sender concern is delivering the mail to 
the destination - period.  How the host farm of receivers is 
inconsequential.

Also consider the advent of GreyListing Servers where the first attempt 
is automatically rejected with the expectation there would be a 2nd 
attempt by the client.   There is not concrete rule that says what each 
MX  expanded set of IP means to the sender other than they represent a 
pool of available servers.

If a host has its own "special" rules for filtering/moving data 
differently, then thats on it only.  Senders has no idea or guideline 
there other than the preference order but that should NOT make a 
different what the actual order of machine calls is.  They all must 
ultimately do what is expected, delivery/relay the message to the target 
address.

I think the "backup" concern was basically a secondary connection 
concept - a 2nd place to go just in case the first machine was down or 
that connection was failing.

Personally, what would have been interesting explore a few years back is 
a system that advertises host availability for different classes of 
senders (content).  Private versus Public, e.g.  ISH or International 
SPAM Hour perhaps to help in better delivery, lesser overhead and even 
open the door to allow spammers to exist (under friendly terms).

So its possible to have a discovery method where a site can define 
"corporate" MX1 machine(s) and also define  the PUBLIC/SPAM MX2 
machine(s).  We can probably do this under the current MX framework by 
providing range definitions to the preference numbers and ranges.

Nonetheless, even with advanced interpretations and methods,  we still 
need to support the basic idea of pools of machines or receivers to 
deliver mail to.  That is all the MX provided. Anything special on the 
receiver end was not important to the sender.

--
HLS

---

From: "Robert A. Rosenberg" <[email-masked]>
Date: Sun, 24 Feb 2013 03:30:43 -0500

At 11:34 -0800 on 02/23/2013, Ned Freed wrote about Re: [ietf-smtp] 

>there are many sites out there that depend on such configurations.

There is also the issue of Internet Facing DNS Servers vs Local DNS 
Servers. The Local DNS Server that is being accessed by the mail 
exchanger (ie: Secondary MX) may have a different MX list then the 
one being used by the Mail Client or sending SMTP MTA Server. This 
can mess up any assumption that the MX lists are the same to everyone.

---

From: Evert Mouw <[email-masked]>
Date: Sun, 24 Feb 2013 09:57:48 +0100

Hi Hector,

Thanks for sharing your thoughts.

> the destination - period.  How the host farm of receivers is

That is not consistent with RFC 5321: "MX records contain a preference
indication that MUST be used in sorting if more than one such record
appears". [ http://tools.ietf.org/html/rfc5321 ]  So the receiving party
can expect that the MX with the lowest preference number is tried first
by a well-behaved sender. If you want "just a pool", give them all the
same preference number.

> pool of available servers.

That is true, although personally I don't like GreyListing that much.
[ http://en.wikipedia.org/wiki/Greylisting#Disadvantages ]

> ultimately do what is expected, delivery/relay the message to the target

Yes it MUST make a difference in the actual order of machines tried by
the sender, see RFC 5321. Of course the actual result can diverge from
what is tried.

> that connection was failing.

Indeed. All I really wanted was something to punish senders that do not
even try to send their mail to the MX with the lowest preference number.

Regards,
Evert

---

From: Keith Moore <[email-masked]>
Date: Mon, 25 Feb 2013 00:56:45 -0500

> because of spam, so something must be done.

The existence of spam is not an argument for changing well-established 
functionality that people are using and which basically works.   In 
general, the debilitating effect of spam countermeasures is at least as 
bad as that of spam itself.

If "many organizations" don't want to support backup MXes, and deal with 
the consequences, that's their choice - not an imperative to recommend 
that they be disabled.

Keith

---

From: John C Klensin <[email-masked]>
Date: Mon, 25 Feb 2013 01:58:36 -0500

--On Sunday, February 24, 2013 02:46 +0000 John Levine
<[email-masked]> wrote:

>># Reject messages on backup mail exchangers when primary MX is
>># online
> 
> On today's Internet, are backup MXes still useful?  Any decent
> mail client should retry for at least a couple of days, and
> it's been a very long time since my main MX was down for
> longer than that.
> 
> For a busy mail system, multiple mail hosts at the same
> priority make plenty of sense, but a backup that accepts the
> mail, then redelivers it to one of the main hosts later?
> Really?

I'm glad you live in such a nice world.  I've still got
correspondents in out of the way places for whom the "backup"
(really just "less preferred") MX hosts are routinely reachable
and getting to the primary ("most preferred") host routinely is
not. Many of the long-delay ("interplanetary") setups with
direct transmission windows of only a few minutes or hours a day
work in exactly the same way.

I also periodically use a network setup in which my secondary
mail system has a back channel (not dependent on SMTP over the
public Internet) to the primary one so that, if the primary WAN
connection is down, the messages get to the secondary system and
then to the primary one in close to real time, not after a delay
of potentially several days.   That is a problem with your
analysis even if one ignores the intermittently-connected
location problem: it assumes that waiting a few days to
_receive_ the mail is considered acceptable.  As Ned and Keith
pointed out in different ways, this is all a matter of
configuration decisions (although it would be possible to
establish more general mechanisms for passing information along)
-- if it is important enough to receive the mail, one sets
things up that way, including with multiple ISP connections that
don't fate-share; if it is not, one doesn't.

And, fwiw, when that setup is running, the list and rules about
deliverable addresses (and other antispam information) are
updated on the secondary machine within a few hours of changes
on the primary one so that the rejects can actually be produced
on the secondary one.  That isn't rocket science either and it
actually helps with the spam problem because the secondary
machine can take up some of the load if the primary is
overloaded.  Whether it is worth the trouble or not depends on
configuration tradeoffs as well.

Coming back to the original note in this thread, I believe the
key comment is "The proper way to address this issue is to deny
the use of a backup MX when the primary MX is online."  With the
understanding that the last clause should perhaps be "primary MX
is responsive", this is exactly what the standard requires
today, so, with the understanding that different senders may
have different visibility to the two (or more) servers (and the
servers may have visibility to each other (or not) when the
sender does or does not), nothing else is needed.  Of course, it
would be a dumb idea to configure multiple MXs at the same
preference level if a "backup" setup rather than, e.g., a
load-sharing one, were intended but that is just another
configuration issue.

Against that backdrop and considering assorted race conditions
with network outages and congestion, the proposed "solution" is
not only unnecessary but a genuinely terrible idea, the latter
because it encourages legitimate senders to use long timeouts
while waiting for the primary system to respond which would be
ok except that we are seeing lots of servers trying to force
short timeouts to mitigate DoS attacks.

The problem with the original suggestion is that it assumes an
arrangement of systems and a configuration model that is just
one of many.   Making the various other arrangements work worse,
or not at all, in order to get a marginal (and, IMO,
questionable) improvement for that one scenario would be, at
best, a bad optimization.

     john

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 25 Feb 2013 14:26:48 +0100

> probably others out there.

Do those systems provide for keeping recipient lists in sync?  It is
too easy to abuse of a backup forced to accept any recipient.  OTOH, a
backup could stay disconnected from the primary for quite some time,
possibly transmitting mail to yet another backup which is able to
connect to the primary.  So it may make sense to have a copy of the
mailbox names at the backup MX.  What are the alternatives?

There is a problem of PII leakage if the backup MXes are managed by
different ADMDs.  It should be possible to distribute that data in a
crypted form, so that the operators of a backup MX can only learn a
(cleartext) mailbox name upon accepting it as a recipient --or by
playing a dictionary attack.  Would that be an acceptable solution?

In order for a backup MX to make the same decisions as the primary,
other data has to be shared as well.  For example, black/white lists
and how they're used.  It is commendable to share them anyway.

Standardizing this stuff could (re)introduce the habit of exchanging
backup MX services, rather than outsourcing it all together to some
big 3rd party whose principles of operations remain secret.

Thank you

---

From: Ned Freed <[email-masked]>
Date: Mon, 25 Feb 2013 08:27:45 -0800 (PST)

> Do those systems provide for keeping recipient lists in sync?

That's a totally separate problem. And yes, it is a problem, and there
are various ways to solve it, and it maybe makes sense to standardize one of
them. Or not - it's a fundamentally harder problem to solve, with a lot 
more variabless.

>  It is

Again, you're talking about a different problem. Let's please focus on the
problem at hand, which is the transfer context from a secondary to a primary.

Divide and conquer.

				Ned

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Mon, 25 Feb 2013 22:57:06 +0100

> problem at hand, which is the transfer context from a secondary to a primary.

IMO these two problems are interrelated. The more a backup MX is 
configured with the settings and policies of the primary, the less the 
need for context transfer.

At one side of the spectrum: if configuration of backup MX is 
'identical' to that of the primary (complete address list, DNS 
blacklists, greylisting, SIZE policy, max. number of recipients policy, 
transfer rate etc.) the dirty job will have been done by the backup MX 
and messages that pass the backup MX can be transferred without a 
problem to the primary. There are companies which have offices in 
multiple countries/continents which have the ability to implement 
'identical' backup and primary.

At the other side of the spectrum there is the situation where the 
backup MX has little 'knowledge' or none at all, about the configuration 
specifics of the primary. In that situation the second problem prevails: 
how to transfer context of the original connection from backup MX to 
primary.

In real life, most backup/primary MX setups are somewhere in between 
those two extremes. Which means, both problems will be present to a 
greater or lesser extent:

1. how to provide 'configuration context' from primary to backup MX
2. how to transfer 'connection context' from backup MX to primary

As for the second problem, talking about context transfer, the following 
comes to mind:

- original source IP address
- original EHLO/HELO identity
- RFC5321.MailFrom information
- RFC5321.RcptTo information
- Results of SPF and DKIM verification
- Date/time of receipt at backup MX
- et cetera

Having said that, I still have not seen figures about how big this 
problem really is. And _what_ exactly the problem is. Is it that backup 
MXs attract spam? Do they and if so, how do we know that? Is it that the 
use of a backup MX causes spam to be accepted, that would have otherwise 
been blocked when it had been sent directly via the primary? Is it that 
backup MXs causes backscatter? Let's first try to define the problem(s) 
and then start to think about solutions.

/rolf

---

From: Ned Freed <[email-masked]>
Date: Mon, 25 Feb 2013 14:01:18 -0800 (PST)

> need for context transfer.

Of course they are interrelated operationally. But I've yet to see that
the solution to one is related to the solution of the other in any way.
Given that the information flow in each case is in the opposite direction,
I don't see how they could be.

At most the other problem might inform to some extent the design of a
standardized XCLIENT sort of extension. But that's all.

So I again repeat: Let's divide an conquer here. Rather than getting
sidetracked into the myriad naunces of MX behavior that are at most
minimally relevant.

> As for the second problem, talking about context transfer, the following

I strongly recommend reading up on XCLIENT as a starting point.

> - original source IP address

Definitely needed, but the port should be included.

> - original EHLO/HELO identity

Definitely needed.

> - RFC5321.RcptTo information

Not needed; already part of the transaction.

> - Results of SPF and DKIM verification

Not needed; we have standardized headers for commnunication authentication
results.

> - Date/time of receipt at backup MX

Probably not needed. Received: fields should suffice unless someone can
articulate a reason for needing this information earlier in the SMTP
dialogue.

> - et cetera

The main one you're missing is AUTH information. There are use cases
for XCLIENT beyond those for secondary MXes, which is another reason for
not using that lens to view this work.

Another possibility is using a shared secret for validation of this
information. Our XPEHLO extension does this.

> and then start to think about solutions.

Since when are these sorts of figures needed to justify this work? We have at
least two proprietary extensions that have been deployed to do this. I can
attest to the fact that both are in use in the field and also to the fact that
XCLIENT support has appeared as a requirement on RFPs.

Frankly, I view the backup MX case as one of less interest than that
of SMTP proxies. And as you and others have noted there are other problems
that need solving in the MX space. But why should we insist on solving them all
at once?

				Ned

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Mon, 25 Feb 2013 22:30:21 +0000

> # Reject messages on backup mail exchangers when primary MX is online

Been there, done that.  I implemented it using Sendmail 8.13 and a socket map (the code for which I regrettably can't find at the moment, but it would be a day's work to recreate it anyway).

My conclusion was (I don't do it anymore, now that I use Postfix) that it has its uses, but only if transparency is more important to the backup MX operator than anything else.  Others have explained how reachability problems will cause unexpected/unintended behaviour, and they're right, but as an informal method it helped me out quite enormously.  (In fact, the whole machine was activated by the presence of its hostname in an MX target; anybody could request backup mail service by just setting it up as their secondary, and it would just do the right thing.  They also assumed responsibility for ensuring that the global view of DNS, as seen by that machine, did not include stale entries that might cause the MX to reject.)

The big thing to watch out for is not to disrupt clients too badly.  You want to arrange for probes of each MX is best-preference-first order to occur as swiftly as possible, and you always want to give the benefit of the doubt whenever you suspect you might be dealing with a network problem (DNS transient fail, the greeting is not read back from the machine, etc).  If you hang up the clients too long, you will do much more harm than good, and if you can't accept that certain sites will experience prolonged periods during which you, or the primary MX through you, are unavailable, then you absolutely shouldn't do it.  During my four or so years of this (which saved one small site several weeks worth of mail), I've only ever experienced one problematic mailer (Yahoo's, I think it was) who persistently tried and retried the secondary and was informed that "450 4.5.0 <[email-masked]>… Not while <hostname-of-primary> is up."  If you want to improve, you should start with some sort of grace failover for multiple failed attempts.

Cheers,
Sabahattin

---

From: Mark Andrews <[email-masked]>
Date: Tue, 26 Feb 2013 09:44:41 +1100

> mailbox names at the backup MX.  What are the alternatives?

It isn't that hard to keep namespaces in sync.  I was doing it 2
decades ago with yellow pages and stock MTA.  All the backup MX's
had a copy of the valid names in a yp database which was checked
on rcpt to.  Yes it required a little bit of configuration to set
up.  You added a account and updated the YP database.  That was
pushed / pulled (too long ago to remember which) to all the backup
MXs.

-- 
Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 26 Feb 2013 13:52:16 +0100

> standardized XCLIENT sort of extension. But that's all.

I'm not clear if clients should see any difference.  An SMTP proxy set
up for submission, for network design reasons, say, would tend to be
as transparent as possible.  A backup MX besieged by a foreign relay
might want to ask the client if it did try the primary beforehand.  (I
have no idea what kind of answer can be sought by that question, but
it seems to be similar to what the OP was wondering about.)

> sidetracked into the myriad nuances of MX behavior that are at most
> minimally relevant.

But by blindly tackling xclient alone, we risk to find a solution that
won't fit well within the general problem.

> I strongly recommend reading up on XCLIENT as a starting point.

I just saw http://www.postfix.org/XCLIENT_README.html  More pointers?

>> [...]
> Not needed; we have standardized headers for communication of
> authentication results.

There is some confusion, though.  Google, for example, seems to be
taking off on the trail of X-Original-Authentication-Results:
https://sites.google.com/site/oauthgoog/mlistsdkim

Both RFC 5451 and RFC 6377 mention the possibility to sign an
Authentication-Results field.  However, the easiest approach that an
MTA can take to guarantee the trustworthiness of its own A-Rs is to
remove (or rename) any existing A-R, which breaks the signature if the
field is signed --MTAs not natively supporting DKIM don't care.

> Frankly, I view the backup MX case as one of less interest than
> that of SMTP proxies. And as you and others have noted there are
> other problems that need solving in the MX space. But why should we
> insist on solving them all at once?

I think Rolf just wanted to see how much interrelated those problems
are.  As for the mail flows, they both overlap with the "forwarding
service", as depicted in http://fixforwarding.org/wiki/mail_flows
Shredding some light on that part of the picture might have its own
merits, irrespectively of which slice of the problem is going to be
solved first.

---

From: hector <[email-masked]>
Date: Tue, 26 Feb 2013 12:31:05 -0500

> by a well-behaved sender. If you want "just a pool", give them all the

It all a pool, which is what many do but only from the standpoint of 
preferred machines to reach first. It really has nothing to do with 
filtering is what I am saying, so despite the order of machines, all of 
them must ultimately be equal in terms of sender expectation for 
acceptance and delivery.  It really doesn't matter who is called first 
or last. The host playing games with whatever logic to alter 
classification of senders is entirely up to them. but whether it was 
MX-FIRST or MX-LAST that was tried by the sender, if the mail is 
received, the senders job is done and the expectation for delivery is 
intact.    There should be no difference that MX-LAST receiving the mail 
first is going to be any different than MX-FIRST receiving the mail 
first as preferred by the host system.

> [ http://en.wikipedia.org/wiki/Greylisting#Disadvantages ] 

Been developing and using greylisting for 9+ years in our package and 
everyone loves it.  You have to make it work for your customers with 
advanced fast whitelisting ideas and with support for advanced "Smart" 
MTA ideas like SMTPGREY 
(http://tools.ietf.org/html/draft-santos-smtpgrey-02) its fits in better.

Nonetheless, we like it or not, it is out there and it only because the 
idea of using different MX for different things was not really 
expected.  I sincerely doubt Greylisting would of taken off as it did if 
such was the case. I know I would have never bother to develop for it 
and implement in products,  which was done reluctantly.  A waste of time 
if there was an MX idea that the secondary forced call could be 
problematic.  I would had never supported Greylisting at any level. 
Don't think so, nor do I think would it has been pursued by many.

> the sender, see RFC 5321. Of course the actual result can diverge from
Yes, but I hope we can agree that it ultimately doesn't matter, 
otherwise we will have a real problem out there and I don't think we do, 
in this regard.   Retry strategies are complex.   You have to have a 
Retry concept and having a pool of different paths or just a different 
computer is fundamentally required. Fortunately, with hardware 
advancement, at least among smaller operators or sites, its one or two 
single machine it runs 24x7x365 and if down, at least downtime at 4-5 
days or so.  The super major majority (over 95% in my view) has been 
able to handle that - otherwise, email would never had exploded like it 
did.  :)

> even try to send their mail to the MX with the lowest preference number.
With the current framework, this would hurt the SMTP transport and 
delivery process as we know it today.  It will also alter Queuing 
Strategies, in my design opinion, drastically.   In other words, it 
would "punish" the good guys too, and that include those who have 
advanced queuing methods.

I think the better approach is to "teach" the MTA how mail can be 
received and delivered more efficiently without wasted retries and 
delays.  New SMTP extensions that attempt to inform clients about 
server/hosting attributes.  That will probably mean a new DNS Server 
Policy discovery concept like the old Fidonet NODELIST which provided 
clients delivery times and whats was allowed to be transferred to 
servers.  It even told clients when to use the BACKUP routing path 
guaranteed by FTSC policy to be always deliverable. The SMTPGREY  
proposal attempts to provide some reject intelligence to help the MTA 
reschedule its 2nd try.  We have proof of concept that it indeed works 
in practice.

But this extension to more about reaching a higher success rate with 
minimal attempts, i.e. making it at the 2nd attempt at most.   What you 
are proposing is to maximize success at the 1st attempt.

This is why its a complex queuing issue.  Suppose I have 1000 messages 
for 1000 people at the same domain?  How will you do the sending?  Do 
all go to the first MX?  Do you spread it?  Are you shooting for fast 
throughput? What is the better queuing strategy? Does one fit all?  Does 
all receivers behave the same?   etc, etc.     If your idea was to make 
that MX-FIRST better than MX-LAST,  I think it will drastically change 
how mail is moved.   If one is punished by trying MX-LAST (and how do 
you monitor this btw?), then rest assured  many won't bother trying 
these last MX expanded ips.  They will just try the retries at the 
MX-FIRST sites.

--
HLS

---

From: Evert Mouw <[email-masked]>
Date: Tue, 26 Feb 2013 20:00:48 +0100

> (http://tools.ietf.org/html/draft-santos-smtpgrey-02) its fits in better.
Sure there will be uses and when it helps to avoid spam, good!
> reschedule its 2nd try.  We have proof of concept that it indeed works 
Yes, that sounds reasonable. Maybe it could even be combined with the 
XCLIENT proposal, so a retry will reach the same MX. (I understand that 
XCLIENT and SMTPGREY are really different beasts with different purposes.)
> throughput? What is the better queuing strategy? Does one fit all?
The answer is in the RFC I have quoted earlier for you. All 1000 
messages MUST be tried by the client using the first MX. Hate it or love 
it, but that is what the RFC 5321 is for. And yes, I know, there is no 
way to check / enforce that, unfortunately I would say. Maybe the 
XCLIENT proposal can help in that regard.

Evert

---

From: hector <[email-masked]>
Date: Tue, 26 Feb 2013 18:37:01 -0500

> messages MUST be tried by the client using the first MX. 
I don't think that is says or implies that.
> Hate it or love it, but that is what the RFC 5321 is for. And yes, I 
> know, there is no way to check / enforce that, unfortunately I would 
> say. Maybe the XCLIENT proposal can help in that regard.
>
The answer is not in the RFC.  If you used a 1000 messages on the first 
MX, you will run into trouble - pronto!!!! It would take forever sending 
mail one at at time.  Of course, that first MX can expand to more 
host/IPs  but  if its only one or few IPs and there are more MXs, there 
are strategies in order to distribute more than one or a few at a time.  
Throughput is very important for many. Also you have the Connection 
Sharing method as well, that can get into the Query Strategy mix.   
Fortunately,  Connection Sharing is frown upon even by the authors of 
such logic in their own wares.

Thanks

--
HLS

---

From: "Robert A. Rosenberg" <[email-masked]>
Date: Wed, 27 Feb 2013 00:44:31 -0500

At 20:00 +0100 on 02/26/2013, Evert Mouw wrote about Re: [ietf-smtp] 
Reject messages on backup mail exchangers w:

>it, but that is what the RFC 5321 is for.

Are you defining "First MX" as the first MX in the list (assuming 
that the list is sorted in ascending order of the priority entry) or 
the pool of MXs with the lowest priority value? So long as you spread 
the 1000 messages over the lowest priority MXs (assuming that there 
are more than one MX with the lowest value) you are following the 
RFC. In fact with Threading of the sending, you would flush the queue 
faster if you spread the messages over all the MXs than just single 
stream them through one MX session. IOW: Round-Robin the list of MXs 
with the same lowest priority value.

---

From: Paul Smith <[email-masked]>
Date: Wed, 27 Feb 2013 08:14:55 +0000

On 26/02/2013 23:37, hector wrote:
> The answer is not in the RFC.  If you used a 1000 messages on the 
> first MX, you will run into trouble - pronto!!!! It would take forever 
> sending mail one at at time.  Of course, that first MX can expand to 
> more host/IPs  but  if its only one or few IPs and there are more MXs, 
> there are strategies in order to distribute more than one or a few at 
> a time.  Throughput is very important for many.
Well, there is also 'fairness'. While the sender may want to blast their 
mailing list at every possible MX and every possible connection on those 
MXs for a domain, it's entirely possible the receiver may be more 
interested in the sales order that is also trying to come in, rather 
than the car rental offers or whatever that are clogging up their servers.

'Throughput' should be for the benefit of the receiver, not the sender. 
It's inconsiderate to send using as many threads as possible to a single 
receiver.

-

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

---

From: Evert Mouw <[email-masked]>
Date: Wed, 27 Feb 2013 12:57:55 +0100

On Wed, Feb 27, 2013 at 12:44:31AM -0500, Robert A. Rosenberg wrote:
> the pool of MXs with the lowest priority value? So long as you
> spread the 1000 messages over the lowest priority MXs (assuming that
> there are more than one MX with the lowest value) you are following
> the RFC. In fact with Threading of the sending, you would flush the
> queue faster if you spread the messages over all the MXs than just
> single stream them through one MX session. IOW: Round-Robin the list
> of MXs with the same lowest priority value.

Absolutely. A pool of MXs with the same preference number can (and in
fact MUST) be used at the same time. However, Hector stated on the subject
of multiple MXs with different preference numbers: "The answer is not in
the RFC." And again I have to refer to RFC 5321:

MX records contain a preference indication that MUST be used in sorting
if more than one such record appears (see below).  Lower numbers are
more preferred than higher ones.  If there are multiple destinations
with the same preference and there is no clear reason to favor one
(e.g., by recognition of an easily reached address), then the
sender-SMTP MUST randomize them to spread the load across multiple mail
exchangers for a specific organization.

Just to clear things up.

Regards, Evert

---

From: hector <[email-masked]>
Date: Wed, 27 Feb 2013 07:48:07 -0500

The main point I wish to make is that there is no calling order that 
anyone can make a correlation with for filtering as you are proposing. 
At least, not anything reliable.  For the sake of robustness, your 
system of receivers is not going to break down if there exist a suite of 
different queuing and mail distribution strategies, including one who 
behavior is to distribute the MX loading.  I hope not.

A second minor point is that the RFC does not describe complete queuing 
strategies.  Does the referenced RFC statement also imply that  a sender 
MUST called the first MX at each subsequent delayed re-send attempt?   
What if there are already 4-5 connections on that channel?

Your proposal/suggestion can hurt a working system IMO.  It can't simply 
assume that the first MX will be available at all times or that it will 
be the first IP tried at all times.  The specs may imply the caller MUST 
sort and by implication call the first IP, but it would seem odd to me 
to design anything around that.  It would be so unreliable.  You can't 
enforce it or guarantee that this order is persistent or consistent.

---
HLS

---

From: Evert Mouw <[email-masked]>
Date: Wed, 27 Feb 2013 14:03:29 +0100

> anyone can make a correlation with for filtering as you are

That's why I proposed one in the first place.

> Does the referenced RFC statement also imply
> that  a sender MUST called the first MX at each subsequent delayed
> re-send attempt?

Probably not, a less preferenced MX should be tried after an unsuccesful
delivery attempt, as far as I understand.

> Your proposal/suggestion can hurt a working system IMO.

If that working system is not behaving well (RFC compliant) then I don't
care.

> It can't
> simply assume that the first MX will be available at all times or
> that it will be the first IP tried at all times.

We already agreed on that (see some mails ago) and the proposal was
already rejected on that ground, to which I agreed. Currently, we are
discussing alternatives, such as XCLIENT and so on.

> The specs may
> imply the caller MUST sort and by implication call the first IP, but
> it would seem odd to me to design anything around that.  It would be
> so unreliable.  You can't enforce it or guarantee that this order is
> persistent or consistent.

Yes, and that is unfortunate and not easily fixable (there might be some
clever tricks though). Also, for some sites, it might in practice be
very consistent, but in general, there are no quarantees. For some
special cases it might be interesting to enforce RFC compliance.

Regards,
Evert

---

From: "John Levine" <[email-masked]>
Date: 27 Feb 2013 19:16:14 -0000

>It's inconsiderate to send using as many threads as possible to a single 

Oh, please, not this old wheeze again.  What's really inconsiderate is
to demand that everyone in the world artifically delay their mail
because you don't know how to manage your mail server.

If I have mail for some of your users, I expect that they will have at
least some interest in reading it, so the sensible and considerate
thing to do is to deliver it promptly so they can do so.  I have no
way of knowing how your mail system works, so the way to do that is to
connect to it and send the mail.

For reasons that should be obvious to anyone who understands how TCP
works, on the modern Internet, the round trip delay time for each SMTP
command is typically longer than the time it sends to send the
traffic, and mail sessions are so short that no single session will
grow to fills more than a fraction of a typical pipe, so it is a lot
faster to send mail in parallel rather than single threading.  (The
bandwidth is trivial, the total amount used by all mail including all
the spam is barely a blip compared to video streaming and p2p.)

So to deliver that mail, I open some mail connections and send the
mail.  If for whatever reason your mail system can't or won't accept a
mail session, reject or soft fail it, and any normal SMTP sender will
back off.  My mail software can deal with rejection, it's software,
not a person.  

On my mail system there is a global SMTP session budget. It accepts
connections whever it's not at the limit, rejects them when it's over,
without worrying about which senders are using which sessions.  That
works great, sheds load quite nicely.

I realize that in some circles full speed mail delivery is considered
anti-social, so sending mail software has developed elaborate
heuristics to try to game the receivers and ramp up to whatever number
of threads won't quite get them throttled.  But that's religion, not
technology.

R's,
John

---

From: hector <[email-masked]>
Date: Wed, 27 Feb 2013 18:19:55 -0500

Maybe I missed the goal, problem to fix?

- Rejecting bad guys,
- Reducing the # of computers one needs, scaling up instead of scaling out?

--
HLS

---

From: hector <[email-masked]>
Date: Wed, 27 Feb 2013 20:39:05 -0500

On 2/27/2013 3:14 AM, Paul Smith wrote:
>> first MX, you will run into trouble - pronto!!!! It would take 
>> forever sending mail one at at time. Of course, that first MX can 
>> expand to more host/IPs  but  if its only one or few IPs and there 
>> are more MXs, there are strategies in order to distribute more than 
>> one or a few at a time.  Throughput is very important for many.
> Well, there is also 'fairness'. While the sender may want to blast 
> their mailing list at every possible MX and every possible connection 
> on those MXs for a domain, it's entirely possible the receiver may be 
> more interested in the sales order that is also trying to come in, 
> rather than the car rental offers or whatever that are clogging up 
> their servers.
>
> 'Throughput' should be for the benefit of the receiver, not the 
> sender. It's inconsiderate to send using as many threads as possible 
> to a single receiver.
>
>

Hi Paul,

I agree a good throughput strategy includes an awareness of the 
resistance, the friction, the connectivity factors (such as limits) of 
the other end. The goal is always a smooth working software machine and 
communications system, no bumps!  :)

Since a receiver network can load balance, set limits, DoS controls, 
etc,  a more complex, prudent  scheme does include senders using the 
larger and/or entire MDA MX spectrum in the way they need it to work 
better for optimized and throughput reasons.    No one want constant 
overhead (retries) and resistance (rejects) with receivers.

One related case scenario that I was noticing last year with a 
newsletter customer,  was that their mailing list addresses with a wide 
spread of different domains, most seem to be hosted by the same vendor 
perhaps for AVS filtering.

So in other words,  you have 100 distinct domains which to us humans 
only, their domains names may suggest they are part of the same trade 
business/group, hence maybe explaining how they came about to use the 
same vendor hosting service.

But now what you may have in a typical single domain issue regarding MX 
and retries, it is now multiplied by a 100 because they are all served 
by the same hosting service with its pool of MXs allocated per customer 
domains.    The pattern was very clear.  The once successful fast 
parallel (concurrent) processing and connection for many domains 
providing  high throughput was now hampered with greater amount of 
hitting a new observed hosting vendor MX connection limits and also 
client IP limits.  It appeared to be an integrated  greylisting-like 
behavior but it also had a behavior where calling at a certain time 
(like after 5pm business hours) appeared to improve the connection and 
acceptance throughput.

I recall the only thing we needed to do was to make sure we provided the 
options for limiting and spreading the outbound mail MX expanded (and 
sorted by preference) IPs.   We also had to create Domain group profiles 
where you can lump domains into a profile with certain outbound, retry 
scheduling strategies.

Overall, in my experience, following RFC 5321 MX logic and let them fly 
(in parallel) worked well. For the larger loads, there might be a lot of 
stress as the race starts, but it tends to smooth out and eventually the 
mail gets there.  The Receivers are really good and robust about getting 
to much work.  The software rejects, the software adjust, etc.

--
HLS

---

From: Ned Freed <[email-masked]>
Date: Wed, 27 Feb 2013 17:28:34 -0800 (PST)

> I'm not clear if clients should see any difference.

Clients? I don't think you understand the nature of this extension if you
think it's about what the client sees. XCLIENT communicates information from
the originating client to the final destination server, not the other way
around.

> An SMTP proxy set

None of this has anything at all to do with XCLIENT.

> won't fit well within the general problem.

The general problem you appear to be looking at doesn't appear to have anything
to do with the problem at hand.

> I just saw http://www.postfix.org/XCLIENT_README.html  More pointers?

That's sufficient.

> https://sites.google.com/site/oauthgoog/mlistsdkim

So? It's hardly news that standards get ignored - sometimes for valid reasons,
often for no good reason at all - in the field.

I have no idea why Google choose to do this, and I frankly don't feel the need
to spend any time looking in to it further.

> are.

Nothing you or Rolf has said has demonstrated that they are.

>  As for the mail flows, they both overlap with the "forwarding

And I quite simply disagree.

				Ned

---

From: SM <[email-masked]>
Date: Wed, 27 Feb 2013 17:52:19 -0800

>often for no good reason at all - in the field.

There are standards and there is operational reality.  The above sums it all.

Regards,
-sm

---

From: "John Levine" <[email-masked]>
Date: 28 Feb 2013 04:15:01 -0000

>that need solving in the MX space. But why should we insist on solving them all

Seems like an interesting direction to go.  I see two somewhat
different scenarios, live proxy, in which the proxy runs the inbound
and outbound SMTP sessions at the same time (what filtering spam
proxies typically do), and sequential proxy in which the proxy runs
the inbound session, then does the outbound session.

I realize that a lot of this goes on already, but are the people who
do it likely to be interested in making their stuff more plug-and-play?

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Thu, 28 Feb 2013 10:49:08 +0100

On 02/28/2013 02:28 AM, Ned Freed wrote:

[...]

> I have no idea why Google choose to do this, and I frankly don't feel 
> the need to spend any time looking in to it further.

As far as I can see, I've seen at least three issues coming along within 
this thread, so I suggest to split up this thread into multiple threads 
to prevent confusion. I have identified the following issues:

 1. the OP started this thread by describing a problem that, in his
    opinion, exists when backup MXs are abused by spammers to circumvent
    AS measures implemented on the primary MX. So if this problem
    exists, this is problem #1.
 2. later on, this thread turned in another direction:

    to which Evert answered:

>     Actually that is an interesting idea. The backup MX could act as
>     an SMTP proxy as long as it can maintain a connection with the
>     primary. That would also sort out your observation on address
>     checking.

    I assume, Ned, that the problem you want to address, is the lack of
    standards in the area of communication between an SMTP proxy and an
    SMTP server, where the context of the original connection from SMTP
    client to SMTP proxy need to be transferred to the SMTP server, is
    that right? If not, please describe what in your view is the problem
    at hand. And whatever the problem description is, it is problem #2
    which is distinct from the original posed problem.
 3. another item was the problem that backup MXs and primary MXs often
    do not implement the same (AS/AV) policies and the problem of
    providing the backup MX with address information that is available
    to the primary. It's nice to discuss it here (I have no suggestion
    for another venue), but I wonder whether this topic belongs on the
    ietf-smtp list.

Please add to this list if you think it's not complete.

/rolf

---

From: Randall Gellens <[email-masked]>
Date: Thu, 28 Feb 2013 02:50:33 -0800

At 7:16 PM +0000 2/27/13, John Levine wrote:

>  anti-social

The server I use has a global maximum receive sessions, a separate 
maximum from the same host, and a reserve that is maintained for the 
use of white-listed hosts.

This has nothing to do with any concept of what's polite versus 
anti-social, but rather to ensure that a spammer can't monopolize all 
inbound connections.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly selected tag: ---------------
Maybe Computer Science should be in the College of Theology.  -
                                              -R. S. Barton

---

From: hector <[email-masked]>
Date: Thu, 28 Feb 2013 07:18:10 -0500

Problem(s) to solve:

- (Detecting and) Rejecting Bad Guys,
- Reducing Computers for Reception (turning off backups when primary is up),
- Uniformity and  Consistent Filtering Methods across MXes,
- RFC Error with implications that different MX and its order has a 
direct correlation to quality of mail acceptance.

I'll be interested in correcting the possible RFC "Error" first. Later, 
a possible BCP for reducing the TCO without losing scale. That may 
include good suggestions for hosting on other sites (an integrated 
cloud) and ideas to communicate authorized hosting information.

--
HLS

On 2/28/2013 4:49 AM, Rolf E. Sonneveld wrote:
> As far as I can see, I've seen at least three issues coming along 
> within this thread, so I suggest to split up this thread into multiple 
> threads to prevent confusion. I have identified the following issues:
>

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Thu, 28 Feb 2013 15:33:17 +0100

> - (Detecting and) Rejecting Bad Guys,

oh, please, let's not try to solve all problems of the world here.

> - Reducing Computers for Reception (turning off backups when primary 

What problem are you trying to address?

> - Uniformity and  Consistent Filtering Methods across MXes,

falls in category problem #3.

> direct correlation to quality of mail acceptance.

Can you elaborate on this? What error are you referring to?

>
> I'll be interested in correcting the possible RFC "Error" first. 
> Later, a possible BCP for reducing the TCO without losing scale. That 
> may include good suggestions for hosting on other sites (an integrated 

/rolf

---

From: hector <[email-masked]>
Date: Thu, 28 Feb 2013 12:27:38 -0500

> What problem are you trying to address?

I was extracting some of the issues within  the threads, as you asked 
for more. :)    The OP proposed to "punish" the MTA who didn't follow 
the MDA's MX order.   I guess reducing the MTAs down to the primary only 
would be one way to make the others unavailable.  Not necessarily 
punishing, but would just make it harder to deliver. If the server used 
a permanent fail like a 550, this will hurt deliver by stopping 
attempts. If it used a temporary 450, then a retry is scheduled. But 
this will not stop it using a non-primary MX the next time.  So nothing 
gained.   Therefore, to follow such an idea, you can only do one thing; 
reduce the computers and perhaps that is the genesis of all this - 
having one computer with multi-cores, enough to handle huge amount of 
worker receiver threads, 25, 50, 100 or more using advanced IOCP 
technology.  Thats called scaling up where scaling out would be like the 
old days with farms of multiple computers.

> Can you elaborate on this? What error are you referring to?
Sure,  I can see where the MX sorting and preference semantics starting 
with 2821 can provide the idea that filtering can be done at the 
secondary computers, hence different quality levels for acceptance.  
That  means if this was the case, part of the semantics, it would 
drastically change the sender strategies and one perhaps where the first 
MX would always be the only one attempted - why bother with the secondaries?

Overall, I don't think it was the intent to do filtering at the 
secondary MXs.  I don't really think its and error per se because this 
is made clear in section 5.1 paragraph 3 by indicating a purpose for the 
list of addresses - "To provide reliable mail transmission..."

    When the lookup succeeds, the mapping can result in a list of
    alternative delivery addresses rather than a single address, because
    of multiple MX records, multihoming, or both.  To provide reliable
    mail transmission, the SMTP client MUST be able to try (and retry)
    each of the relevant addresses in this list in order, until a
    delivery attempt succeeds.  However, there MAY also be a configurable
    limit on the number of alternate addresses that can be tried.  In any
    case, the SMTP client SHOULD try at least two addresses.

If there is ever a new 5321, it may be something to clarify.

Thanks

--
HLS

---

From: "John Levine" <[email-masked]>
Date: 28 Feb 2013 18:35:15 -0000

>anti-social, but rather to ensure that a spammer can't monopolize all 

Seems perfectly reasonable.

Do you have any idea how often it blocks a spam blast rather than just
some list with a lot of legit recipients?

R's,
John

---

From: Randall Gellens <[email-masked]>
Date: Thu, 28 Feb 2013 14:53:45 -0800

>  some list with a lot of legit recipients?

For my own case, no, since I don't monitor the statistics closely 
enough.  If you're curious, I can forward your question to the 
mailing list for users of the software (EIMS).  Likely one of the ISP 
or corporate users keeps closer watch and would know.

-------------- Randomly selected tag: ---------------
Information is the currency of democracy.
    --Thomas Jefferson

---

From: "John R Levine" <[email-masked]>
Date: 28 Feb 2013 18:36:05 -0500

> For my own case, no, since I don't monitor the statistics closely enough.  If 
> you're curious, I can forward your question to the mailing list for users of 
> the software (EIMS).  Likely one of the ISP or corporate users keeps closer 
> watch and would know.

It'd be interesting.

These days spam tends to come from botnets or snowshoe, so it's very
unusual for a very large blast of spam to come from a single source at 
once.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Ned Freed <[email-masked]>
Date: Thu, 28 Feb 2013 19:21:30 -0800 (PST)

>     exists, this is problem #1.

This is a general area, not a specific technical problem.

>     that right?

No, not really. I want to address the general problem of passing connection
context through relays, proxies, secondary MXes (which can operate either
as relays, proxies, or something in between), HTTP web service mail
submission interfaces, and probably some variants of IMAP submit.

I fail to see why all of these things, which deal in essentially the
same information, should require separate standards.

>     If not, please describe what in your view is the problem

On the contrary, there is substantial overlap. A key aspect of the
circumvention of the AS solution on the primary once the message makes
it there is the lack of original connection context. Havi

>     for another venue), but I wonder whether this topic belongs on the

Your #2 and #3 have the essentially the same solution. As such, I do not much
value in pursuing them independently. However, I do see value in pursuing the
problem of the primary providing information to make the secondaries life
easier, e.g., by improving its ability to validate addresses.

				Ned