Subject: Re: A new SMTP "3821" [Re: FTC stuff...........]

From: David Woodhouse <[email-masked]>
Date: Thu, 02 Dec 2004 01:13:00 +0000

On Fri, 2004-11-26 at 09:16 +0000, Chris Haynes wrote:
> Let me focus on the heart of the debate, and I'm doing my best to be fair to
> both sides here, and to use neutral language.

I think this is very useful; thanks. But I think you've slightly
misrepresented the 'conservative' position.

> Still in the spirit of 'neutral language' , let me use the terms 'conservative'
> and 'progressive' for the two positions I am aware of.
> 
> The conservatives look at the above example and say
> "SPF is breaking the mail system. Forwarding has worked perfectly well in the
> past; it is SPF which is changing; it is SPF which is wrong.  SPF should not be
> deployed".

That's not quite how I'd phrase it. I'd say it was more like this:

"SPF is breaking the mail system for no good reason. Forwarding has
worked perfectly well in the past; it is SPF which is changing; it is
SPF which is wrong. SPF should not be deployed because there are better
ways of achieving the same thing, without the need for such changes."

> Now, continuing in my attempt to be fair, let me withdraw any implication that
> the conservatives are against any change whatsoever.  I'm using 'progressive'
> and 'conservative' purely in the context of this one topic.

Indeed. If a change is absolutely necessary to make progress, I don't
think many people would be arguing against it. See the fairly widespread
support for closing open relays, for example. The problem in this
particular case is that the 'progressives' are trying to enforce change
while the 'conservatives' see ways that the same goals can be achieved
without it.

In particular, it's much easier to justify changes which can be limited
to participating endpoints, rather than changes which affect
uninterested third parties who may encounter the mail in transit.

> How, and from where, would one get an 'authoritative' ruling about the
> conformance or non-conformance of  forwarding (using the origin's MAIL FROM with
> a new RCPT TO) to the extant SMTP architecture?

Actually, I don't think we need such an answer. We know the extent of
existing practice; what was intended in 1982 isn't really relevant any
more.

The important question, in my mind, is 'Do we _need_ to change it?'. And
unless all the alternative schemes like CSV/DK/SES are shown to be
unworkable, I cannot really see how anyone could answer 'yes' to that
question.

Meng Weng Wong has previously said, "When banks start DomainKeys or
S/MIME signing all outbound mail, I promise to give up SPF and Sender
ID".¹  I'll follow that with my own promise:

When everyone has given up on DomainKeys, IIM, SES, CSV, BATV and the
similar proposals which don't need to change the world, I promise to
publish an SPF record and make my forwarding hosts cope with the new
requirements of SPF.

-- 
dwmw2
¹ http://www.imc.org/ietf-mxcomp/mail-archive/msg04998.html

---

From: Dean Anderson <[email-masked]>
Date: Thu, 2 Dec 2004 15:52:33 -0500 (EST)

> ways of achieving the same thing, without the need for such changes."

I think there was more to it than this:

        1) Abuser can forge addresses at domain
        2) Abuser can use stolen credential
        3) DNS cache problems (more records per domain, same cache size)
        4) DNS load (more records per domain)
        5) Ongoing Maintenance issues
        6) Migration issues
        7) IP Renumbering issues
        8) Lost non-spam emails
        9) Lack of universal compliance.*
	10) Not a basis for trust/reduced filtering
	11) Makes forgery blowback problem _much_ worse
	12) Patent issues
	13) spam-profiteering / charges for SPF services

I think I left some things off that list.  I'd say that "After more than a
year of intense technical analysis by 2 IETF working groups, in then end,
SPF didn't achieve any of the stated goals, and made some problems such as
'blowback' much worse."  Perhaps this is aptly summarized as "SPF is
breaking the mail system for no good reason." I guess I'd take that as an
executive summary.

Also, I've been following the source routing discussion, and would note
that there were good reasons to have source routing (interaction between
incompatible networks), and good reasons not to have source routing and
use direct connection instead.  These haven't changed.  Source routing 
really implies the attempt to partition the internet into a spam/abuse 
part and a non-spam part. Such a partition is unrealistic.

I found it mildly ironic that the same people who said open relays were
bad and that source routing _MUST_ be disabled, are now advocating source
routing as a necessary solution.  Anticipating that they next require SMTP
AUTH as mandatory, that will also not reduce spam, because EVERY spammer
is authorized via SMTP AUTH, either by a stolen credential or by a
disposable account.  So the situation is still unchanged.  To use a
farm-colloquialism: "You can't dam a river with a wire fence"  Painting
the fence a different color won't help. Nor will changing from wood
fenceposts to steel fenceposts.  A fence just won't work, and a solid wall
is impossible (wrt spam), and wouldn't work either, because unlike water,
a spammer can choose to cross the solid wall.

		--Dean

-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 344 9000

---

From: David Woodhouse <[email-masked]>
Date: Thu, 02 Dec 2004 23:47:49 +0000

> I think there was more to it than this:
  <...> 
>  I'd say that "After more than a year of intense technical analysis by 2 
> IETF working groups, in then end, SPF didn't achieve any of the stated
> goals, and made some problems such as 'blowback' much worse."  Perhaps
> this is aptly summarized as "SPF is breaking the mail system for no
> good reason." I guess I'd take that as an executive summary. 

Well yes, but I was trying to follow Chris' lead -- be fair to both
sides, and use neutral language :)

-- 
dwmw2

---

From: Dean Anderson <[email-masked]>
Date: Thu, 2 Dec 2004 19:35:22 -0500 (EST)

> sides, and use neutral language :)

How about something like:

A large and diverse group of people came together through two IETF working
groups and together put in a tremendous amount of work on the problem for
more than a year. Unfortunately, their best efforts were unable to
overcome a dozen or so critical problems.  The effort should not be viewed
as a complete loss.  Thomas Edison is reported to have said, when asked
about the large number of failures in trying invent the lightbulb, that he
didn't fail at all, but just discovered a lot of ways not to make a
lightbulb.  So, to take the most positive view: "We've just found another
way that won't help with the spam problem"

		--Dean

---

From: "Chris Haynes" <[email-masked]>
Date: Fri, 3 Dec 2004 09:30:44 -0000

> 13) spam-profiteering / charges for SPF services

I've been trying to act as a neutral observer, collating lists of strengths and
weaknesses of all the offerings in the anti-spam, anti-forgery space in as
neutral and fair a way as I can.

Once again, in the spirit of attempting to be 'fair' to all, can I just please
ask for clarification of the above list of concerns.

When this thread started we were referring to the problems related to forwarding
which are inherent in (the original) SPF(-Classic).

For the benenfit of other readers, may I just remind people that there have been
three proposals in the "SPF" space, but only one was formally submitted to
MARID: the Sender-ID one containing an amalgum of SPF and Sender-ID/PRA.

The above list looks to me like a list of the concerns with that combined
effort - and I'm concerned that problems with Sender_ID may here be unreasonably
associated with SPF-Classic.

The concerns I have previously seen associated with SPF-Classic, and are also
present in Sender-ID are:

It is my understanding that only the above concerns should be associated with
"SPF".

The ones which appear to me to have been added by PRA (derived from Caller-ID)
in Sender-ID are:

And ones which probably represent universal engineering concerns:

2) Abuser can use stolen credential
5) Ongoing Maintenance issues
6) Migration issues
9) Lack of universal compliance.

although of course the way and degree to which these concerns mainifest depends
on the exact scheme in question.

Chris Haynes

---

From: David Woodhouse <[email-masked]>
Date: Fri, 03 Dec 2004 11:19:11 +0000

> 1) Abuser can forge addresses at domain

This is common to PRA and SPF. Anyone with an account on one of the
'authorised' boxes (or subnets, if dynamic IP is in play) can sand mail
which passes. There isn't even an option to require that the mail comes
from a port < 1024, which is odd.

> 11) Makes forgery blowback problem _much_ worse

Not sure about this one. I don't actually see why this is the case.

> 13) spam-profiteering / charges for SPF services

I don't understand this one either, but I don't think it's specific to
PRA. 

> on the exact scheme in question.

In particular, number nine does vary to a vast degree; so much so that
it should be considered a separate problem for some of them. There's a
big difference between a scheme which requires _universal_ compliance,
and a scheme which requires only the sending and receiving parties to
participate, without any need for the rest of the world to change.

-- 
dwmw2

---

From: Dean Anderson <[email-masked]>
Date: Fri, 3 Dec 2004 15:40:03 -0500 (EST)

> from a port < 1024, which is odd.

Agree with all except the "port < 1024" part. Back in the old days, when
computers were significant resources, this was probably implied a person
(eg system admin with root) responsible for a valuable asset. But today it
can be assumed that "port < 1024" does not imply anything.

> Not sure about this one. I don't actually see why this is the case.

"Blowback" is the bounces that you get when a spammer forges your address 
as the From: address. Much of the spam run will be delivered, but what 
isn't, will be bounced.

ISP customer Spammer posts mail to that ISP's relay with a forged return 
address. The recipients reject the message due to SPF. Now the relay sends 
a bounce to the "from: address".   This message is from the Relay, which 
will be valid.  So now instead of a a few bounced messages, you get a 
bounce for every message blocked.

Of course, if the target is harrassment, the spammer can just exchange the 
to: address and the from: address and use some ISPs relay.  You can't 
block the ISP's relay without affecting the rest of the email.

This is no difference than a reverse synflood DDOS attack using a
wellknown router.  Syns are sent to your upstream router, with your IP
address as the source. The router responds with a SYN ack. This can easily
be made to saturate a customer link.  The customer can't block the
upstream router.

> I don't understand this one either, but I don't think it's specific to

Patent issues item refers to the patents that apply to SPF and Sender-ID. 
Several organziations said they won't deploy anything that is patented.

Spam-profiteering was tangentially discussed. Basically, an ISP such as
AOL or MSN can charge fees or prevent altogether the unbunding of IP
access and email services.

> participate, without any need for the rest of the world to change.

Right. Except that there is typically 3 or more parties to email: The
sender, the relay operator, the mailbox operator, and the recipient.

Secondly, if only a subset of the net deploys SPF, one cannot block mail
based on the lack of an SPF record.  And one cannot simply perform less
testing if an SPF record exists since that would encourage spammers to use
SPF-using ISPs (and in fact, it was found that most of the early deployers
of SPF were spammers). One can only do blocking if SPF deployment is
universal, and that is unlikely.

It is not the case that SPF can be deployed merely with respect to the 
sender and recipient.  There was significant discussion on this point with 
Alan Dekok.

		--Dean

---

From: "Alan DeKok" <[email-masked]>
Date: Fri, 03 Dec 2004 20:40:29 -0500

> bounce for every message blocked.

  Is this a *new* attack, or is it an old attack with changed cost?

  So far as I can see, this exact same attack can happen when a server
rejects messages from such a relay, independent of SPF.  Maybe there
are other methods of MAIL FROM validation which don't have this
"blow-back" property.  But I have a hard time seeing how any of them
solve the problem of relays which accept mail that they cannot
deliver, and that they cannot bounce to the correct entity.

> sender and recipient.  There was significant discussion on this point with 

  It can be deployed any way anyone wants.  Whether there will be side
effects is another story.

  Yes, doing MAIL FROM validation will have side effects on relays and
others who use "MAIL FROM [email-masked]" without example.com
knowing.  But to be pedantic, that's the whole *point* of MAIL FROM
checking: to know who is using your domain name in MAIL FROM, and to
control their use of that name.

  If MAIL FROM validation doesn't allow the domain to control the use
of it's name by an MTA, then MAIL FROM validation is not taking place.

  EHLO validation is orthogonal to MAIL FROM validation, for the
simple reason that they're separate fields, and there's not
requirement for the same domain name to appear in both.

  Alan DeKok.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 04 Dec 2004 08:00:03 +0100

Alan DeKok wrote:

> I have a hard time seeing how any of them solve the problem
> of relays which accept mail that they cannot deliver, and
> that they cannot bounce to the correct entity.

SPF does this either directly or indirectly:

If a zombie uses a forged MAIL FROM protected by SPF, and the
receiver checks the sender policy, then it rejects this spam.
The zombie won't create a bounce, it's a minimal SMTP script.

If the zombie sends to an old-style forwarder (= no SPF check),
and the spam is rejected by the next hop, then this old-style
forwarder creates a bounce.  But that's the same what happens
if the real MAIL FROM sends mail to this forwarder, and the
next hop rejects it, because the IP of the forwarder is not
listed in the sender policy.

The old-style forwarder is out of business as soon as the real
sender and the next hop do SPF -all : _all_ mails are rejected
by the next hop, and the sender will stop to send any further
mail to the corresponding forwarded address, until the receiver
has fixed his mail setup.

Therefore all further bounces from this "forwarded address" are
caused by forgeries.  They go to nanas and abuse@forwarder, it
is his problem, if he refuses to fix it he will be blacklisted.

And if the forwarder supports SPF we're back at square one, the
zombie can't send mail to the forwarded address, no bounce, no
problem.

Of course the zombie can use its own throw-away domain with a
sender policy resulting in +all.  But then a potential bounce
won't hit any innocent bystander.

> It can be deployed any way anyone wants.  Whether there will
> be side effects is another story.

"SPF breaks forwarding to third parties" is only the executive
summary.  The truth is that a sender policy simply lists IPs of
border MTAs used for HELO domain resp. MAIL FROM:<user@domain>.

It's IMHO obvious that the receiver can check these IPs only at
one of his border MXs, not later in his routing.  "Forwarding
to third parties" is only one special case of any "later in his
routing".  The essence of SPF is KISS.

> to be pedantic, that's the whole *point* of MAIL FROM
> checking: to know who is using your domain name in MAIL FROM,
> and to control their use of that name.

Ideed.  And it's the receiver who can control it, if he wishes
to control it.  But he can only control it at his border, not
somewhere else in his routing (= later in his forwarding).

> EHLO validation is orthogonal to MAIL FROM validation, for
> the simple reason that they're separate fields, and there's
> not requirement for the same domain name to appear in both.

If the HELO domain is also used in a MAIL FROM:<user@domain> -
not necessarily by the same MTAs - then it makes sense to join
the sets of corresponding IPs in one SPF sender policy for all
receivers using SPF to control their borders.

You could also combine it with other schemes, and if the other
scheme says "bad HELO" skip all SPF tests and reject all mails.

If the other scheme says "good HELO" you could skip SPF HELO
tests (that's not explained in the SPF draft, it's obvious ;-)
but still do SPF MAIL FROM tests.
                                   Bye, Frank

---

From: David Woodhouse <[email-masked]>
Date: Sat, 04 Dec 2004 11:15:46 +0000

> can be assumed that "port < 1024" does not imply anything.

Still it can be assumed that "port < 1024" implies a user with admin
rights for the machine with that IP address. So if an IP address of a
server which also has user accounts is listed, it would make some sense
to indicate that connections should come from a privileged port.

> as the From: address. Much of the spam run will be delivered, but what 

Generally, what isn't delivered should be rejected by the first
recipient it's offered to. It never leaves the spammer's machine and
there's no bounce.

> bounce for every message blocked.

When spammers abuse open relays or ISPs run without any form of antispam
measures, allowing similar abuse of their own relays, this is indeed
possible. It's not caused by SPF alone though -- many forms of spam
checking may cause it, and even the rejection of mail to unknown
addresses at the destination site. This is a problem with any system
which attempts to store-and-forward without an authenticated source
address, and without knowing that its anti-spam policies are at least as
strict as the next recipient.

This is why you should always have MX backups which are capable of
rejecting mail to unknown users at the domain, for example. 

It's also a problem which, at the victim's side, is trivially fixed by
implementing SES -- you instantly stop accepting the bounces to mail you
didn't send.

> sender, the relay operator, the mailbox operator, and the recipient.

Yes, that was precisely my point. A useful scheme would be one which
works properly when it is implemented only by the sender and the
recipient, and nobody in between. In fact SES goes one better than that
-- it actually works to a large extent, allowing you to reject bounces
to mail you didn't send, when implemented solely at the 'sender' site.

-- 
dwmw2

---

From: "Hector Santos" <[email-masked]>
Date: Sat, 4 Dec 2004 10:08:32 -0500

----- Original Message -----
From: "Frank Ellermann" <[email-masked]>
To: <[email-masked]>
Sent: Saturday, December 04, 2004 2:00 AM
Subject: Re: A new SMTP "3821"

> routing".  The essence of SPF is KISS.

This is where possibly a HELO checking methodology works best because at
the transition/route, it is presumed it be trusted by the local network
chain.

> but still do SPF MAIL FROM tests.

Good point on mixed policies.

I'll repost my link to my early work on LMAP validation and trust analysis.
It became the basis for our current SMTP design:

http://www.winserver.com/public/antispam/lmap/draft-lmapanalysis1-2.htm

-- Hector

---

From: Bill McQuillan <[email-masked]>
Date: Sat, 4 Dec 2004 10:28:54 -0800

Okay, I give up!

At the risk of appearing illiterate, I have to ask.

> border MTAs used for HELO domain resp. MAIL FROM:<user@domain>.
                                   ^^^^^
What do you mean by this abbreviation? 

You seem to use it frequently and I have been unable to deduce
its meaning from context.

Please keep the derision to a minimum.  :-)

-- 
Bill McQuillan <[email-masked]>

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 05 Dec 2004 18:21:38 -0500

> recipient it's offered to. It never leaves the spammer's machine and

  As has been pointed out repeatedly, this won't work in the existing
SMTP system, due to certain design and implementation decisions.

  Whether this means the current design is incorrect, or your idea is
incorrect is an argument which will never end.

  For related questions, what happens when I forward email manually,
and it bounces back to me?  Is it legal for me to delete it, or should
I in turn bounce it to the originator who sent it to me?  How does
this scenario differ from the automated forwarding process?  Is it
legal, ever, for a part of the email system to discard a message?  If
so, why?  If not, why not?

> rejecting mail to unknown users at the domain, for example. 

  Ideally, yes.  This can be difficult to do in practice.

  Alan DeKok.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 06 Dec 2004 01:11:21 +0100

> Okay, I give up!

That's the first case where my "DEnglish" forced somebody to 
give up, <shame on="me" /> :-(

> What do you mean by this abbreviation?

"respectively",  Google found a link explaining why that's
a bad idea:  <http://www.languagehat.com/archives/001173.php>

Maybe I should try "or" in these cases, it's shorter ;-)  Bye.

---

From: Bill McQuillan <[email-masked]>
Date: Mon, 6 Dec 2004 01:21:26 -0800

> Maybe I should try "or" in these cases, it's shorter ;-)  Bye.

Thanks for the explanation! I expected to slap my forehead and
say "I should have known that"!

But this is much more interesting. Thanks also for the
LanguageHat reference, I may spend a bit of time there.

---

From: David Woodhouse <[email-masked]>
Date: Mon, 06 Dec 2004 15:32:46 +0000

> incorrect is an argument which will never end.

There's no problem with the current design. In general it's not hard.
Either the spammer is sending mail via the ISP of the zombie machine,
which ought to be doing some kind of check on outgoing mail from their
customers, or often they try to connect directly to an MX host of the
domain to which they're trying to send, which ought to reject anything
the primary MX would reject.

>   Ideally, yes.  This can be difficult to do in practice.

Anything _can_ be difficult to do in practice, if you make enough stupid
decisions along the way. Setting up an MX backup competently can be
_trivial_ to do in practice too.

-- 
dwmw2

---

From: <[email-masked]>
Date: Mon, 6 Dec 2004 10:56:05 -0500

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Alan DeKok
> Sent: Sunday, December 05, 2004 6:22 PM
> To: MXCOMP
> Subject: Re: A new SMTP "3821" [Re: FTC stuff...........]
>
>
>

Indeed, near impossible: There are more then just "invalid recipient" as rejection reason.  Consider
the popular "mailbox full" error.

Terry Fielder
Manager Software Development and Deployment
Great Gulf Homes / Ashton Woods Homes
[email-masked]
Fax: (416) 441-9085

---

From: "Alan DeKok" <[email-masked]>
Date: Mon, 06 Dec 2004 13:11:52 -0500

David Woodhouse <[email-masked]> wrote:
> which ought 

  ... to do a lot of things, but often doesn't.

> to be doing some kind of check on outgoing mail from their

  And what about relays?  Forwarders?

  You're assuming that messages go from source to destination in one
hop.  While this is nice, the current design allows a message to
traverse multiple independent hops, all the while using the same "MAIL
FROM".  This has a serious impact on the "blowback" problem, and any
possible solution.

> _trivial_ to do in practice too.

  Yes, but sharing live information about all of your users with a
backup MX is difficult to do in practice.  Spammers leverage the fact
that backup & primary MX's have different pools of information to
convince the backup MX to deliver messages that the primary might have
rejected.

  If nothing else, the spammers are offloading some of their work onto
the backup MX, and using it to attack the primary.  This attack has
serious consequences for the robustness of the email transport layer.

  Alan DeKok.

---

From: Dean Anderson <[email-masked]>
Date: Mon, 6 Dec 2004 13:01:39 -0500 (EST)

On Fri, 3 Dec 2004, Alan DeKok wrote:

> rejects messages from such a relay, independent of SPF.  

I think it is an old attack, as described when all mail from a relay is 
rejected. Cost is the same as before. Possibly cheaper, since it is 
easier for the abuser to arrange the case where all email will be rejected 
by recipient, but not by forged sender.

> effects is another story.

Err, no. What is meant is that "sender and recipient" is an
oversimplication. There are more parties than just the "sender and
recipient".

> control their use of that name.

Yes, I know that is "the point". However, this isn't possible. Hence the 
conclusion that "SPF is breaking the mail system for no good reason".

> of it's name by an MTA, then MAIL FROM validation is not taking place.

That is correct. MAIL From validation isn't taking place.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Mon, 6 Dec 2004 13:28:36 -0500 (EST)

> to indicate that connections should come from a privileged port.

Just about everyone has admin rights over a machine these days.  Some few 
machines have such rights limited. But even then, that's no indication 
that the admin is responsible or trustworthy.  Not that I guess I feel 
very strongly about it either way.  But, certainly, spammers can get port 
< 1024 if they want. 

Mainly, I'd prefer not to have to remember that IMAP is port 52345. So,
I'd prefer that protocols be assigned low ports. But this is the "listen"
port. I'm not terribly concerned about what port numbers clients connect
from.

> recipient it's offered to. It never leaves the spammer's machine and

This is only true if no relay is involved.  Every spammer has the right
(at least until their account is terminated) to use their ISP's relay, if
the choose. I think at present spammers and viruses send directly because
it easier than figuring out the right relay to use at the ISP.  The relay
information is something that customer gets along with their account
information, and while not secret, its location on a compromised machine
depends on the mail software being used. In other words, the 
spammer/virus/worm operator has to search for it. Much easier to send 
directly.

Ironically, SPF makes this problem much easier for the virus operator, by
identifying the outgoing relays in DNS.  These are the same relays that 
will accept outgoing customer email. 

> strict as the next recipient.

Yes, this is true of traditional address forgery. People forget that there
is no difference between open relays and closed relays, except that closed
relays only accept email from a particular address range.  Spammer always 
has a relay available if they choose.

But with ordinary (non-SPF enhanced)  forgery, the forged recipient
usually only gets _SOME_ bounces.  With SPF, they get _ALL_ messages sent
by the spammer/abuser.  This was one of the problems stated as to be
solved by SPF. SPF doesn't solve it, but makes it worse.

> rejecting mail to unknown users at the domain, for example. 

This doesn't help with forged mail targeted at an existing address, which 
I think is the goal of SPF/Sender-ID. Rejecting mail to unknown users is 
implemented without SPF/Sender-ID. Its the forged mail to real users that 
causes the subject problem.

> to mail you didn't send, when implemented solely at the 'sender' site.

I'm not familiar enough with SES to comment.  Sounds like I should spend 
some time on it, though.

---

From: "Alan DeKok" <[email-masked]>
Date: Mon, 06 Dec 2004 19:13:04 -0500

Dean Anderson <[email-masked]> wrote:
> > But to be pedantic, that's the whole *point* of MAIL FROM
> Yes, I know that is "the point". However, this isn't possible.

  In theory or in practice?  If it's possible in theory, it's possible
in practice.  The only question then is whether the cost is acceptable.

  If it's not possible in theory, I'm curious to know why.

>  Hence the conclusion that "SPF is breaking the mail system for no
> good reason".

  So... am I to be permitted to control the use of my own domain name?
If not, why not?  If so, then I'm free to implement SPF, or anything
other scheme I like.

  Alan DeKok.

---

From: David Woodhouse <[email-masked]>
Date: Mon, 06 Dec 2004 23:34:35 +0000

> very strongly about it either way.  But, certainly, spammers can get port 

I think we're talking at cross purposes here. I'm suggesting that one of
the flaws in the SPF draft -- that anyone with an account on a machine
which is listed can send 'authorised' mail -- could be alleviated by
having an _option_ to say that mail is only authorised if it comes from
a privileged port. I'm not suggesting that one should trust all mail
from a port < 1024 and not accept any from unprivileged ports, because
those with admin rights are 'responsible for a valuable asset' (to use
your terminology).

> it easier than figuring out the right relay to use at the ISP.

And also because ISPs do rate limiting and spam checking on outgoing
mail.

> will accept outgoing customer email. 

Not necessarily. You may need SMTP AUTH and there's no guarantee that
the _incoming_ addresses of the cluster are the same as the outgoing
addresses.

> solved by SPF. SPF doesn't solve it, but makes it worse.

Not really. Hardly anybody actually rejects for an SPF failure anyway.
You can't -- you'd throw away too much valid mail. But you could say the
same for _any_ spam filtering. SpamAssassin makes it worse far more than
SPF does, because _far_ more people will reject for having a high SA
score than they will for SPF. You're right, but I think it's a red
herring. It's going to be the case for _any_ scheme which causes mail to
be rejected by the ultimate recipient, when the relay may have accepted
it.

> causes the subject problem.

It's just another example of the general case -- anything the primary
will reject, the backup should reject too. That's why you should be
running identical spam and virus checking, for example.

You're right, but it's still not specific to SPF; that's all I'm saying.
Give me a break here -- it's rare that I defend SPF, utterly broken as
it is :)

> I'm not familiar enough with SES to comment.  Sounds like I should spend 

Agreed.

-- 
dwmw2

---

From: David Woodhouse <[email-masked]>
Date: Tue, 07 Dec 2004 00:17:38 +0000

> FROM".  This has a serious impact on the "blowback" problem, and any

I'm not assuming that. I'm saying that SPF doesn't make the problem any
worse than _other_ schemes will, if they cause the ultimate recipient to
reject mail which the {backup MX, relay, forwarder} does not reject.

SPF and SenderID have many flaws. This one isn't specific to SPF and
SenderID.

> backup MX is difficult to do in practice. 

That may be your experience; it's not mine.

> serious consequences for the robustness of the email transport layer.

Even to the extent that's true, it's not specific to SPF.

-- 
dwmw2

---

From: David Woodhouse <[email-masked]>
Date: Tue, 07 Dec 2004 00:33:55 +0000

> in practice.  The only question then is whether the cost is acceptable.

Anything's possible in theory, yes. In practice it requires a lot of the
rest of the world co-operating, but yeah, it _could_ happen. Unless SES
and DK and IIM and all the rest are shown not to be workable, I don't
think anyone could claim that the cost is acceptable.

> If not, why not?  If so, then I'm free to implement SPF, or anything

In theory of course you can control the use of your own domain name.

In theory you can also control the use of your own IP addresses. You can
suddenly declare that the core routers may no longer use it on the
packets they forward; that they must use their _own_ IP addresses and
all else is 'forgery'. Good luck in getting them all to implement NAT
for you. :)

There is a difference between theory and practice.

-- 
dwmw2

---

From: David Woodhouse <[email-masked]>
Date: Tue, 07 Dec 2004 00:59:54 +0000

On Mon, 2004-12-06 at 19:55 -0500, Hector Santos wrote:
> If you are using a SPF domain, then you either have your network correctly
> setup for outbound mail or you dont.

The network in which I participate is known as "the Internet". It is not
'correctly' setup for outbound mail, by the definition you're using.

-- 
dwmw2

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 6 Dec 2004 19:55:44 -0500

----- Original Message -----
From: "David Woodhouse" <[email-masked]>
To: "Dean Anderson" <[email-masked]>
Cc: "Chris Haynes" <[email-masked]>; "MXCOMP" <[email-masked]>
Sent: Monday, December 06, 2004 6:34 PM
Subject: Re: A new SMTP "3821" [Re: FTC stuff...........]

> You can't -- you'd throw away too much valid mail.

Not according to our 1 year plus of real production in thousands of sites.

One thing is for sure to come to learn:

Spammers don't complain.   If there are FALSE positives for legit systems,
you will quickly get reports or compliants.   The one segment that are not
complaining is the problem senders/spammers/spoofers.    The reality is the
former is extremely rare and when it is reported, it is all usually due to
getting a network correctly configured which should of been the case in the
first place.

Our view is this:

If you are using a SPF domain, then you either have your network correctly
setup for outbound mail or you dont.   The alternative is to use the relaxed
provisions (softfail and neutral) in which case, it allows for a Receiver to
perform further testing as I illustrated in my previous post regarding the
mix policy.

The main problem I see is that the "advertisement" the SPF people to tell
all people to go ahead and publish a SPF DOMAIN in DNS even if  your own
servers are not SPF ready.

That is half the problem as I see it today as evident with my own private
email account with FastAccess Bellsouth.  They added an SPF domain which
brings the usage of legitimate users usage of Alias Addresses when the mail
hits a SPF server downlink

This is where the SUBMITTER proposal had some promise to assist.  But it
assumes the BellSouth Server was SPF/SUBMITTER ready (as well as downlinks)
and its not. :-)

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 6 Dec 2004 20:07:41 -0500

----- Original Message -----
From: "David Woodhouse" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Cc: "Chris Haynes" <[email-masked]>; "MXCOMP" <[email-masked]>
Sent: Monday, December 06, 2004 7:59 PM
Subject: Re: A new SMTP "3821" [Re: FTC stuff...........]

> 'correctly' setup for outbound mail, by the definition you're using.

Good point.

However, by my definition, if you wish to use SPF by exposing your domain or
network of domains to the SPF ready world, then you better be ready for it.

My point is that you shouldn't use SPF if you don't have your network of
outbound mail machines under its umbrella.

Don't you agree?

Sincerely,

Hector Santos, CTO

---

From: "Alan DeKok" <[email-masked]>
Date: Mon, 06 Dec 2004 21:35:10 -0500

David Woodhouse <[email-masked]> wrote:
> Anything's possible in theory, yes.

  That's the problem, there are things which *are* impossible in
theory.  So if something is impossible in theory, there's no point in
talking about it.  If it's possible in theory, the discussion should
then be cost of implementation.

> In theory of course you can control the use of your own domain name.

  If I can't control the use in practice, then what value is there in
the domain name?  I've registered it and paid for it, but others are
free to use it almost any way they want, without my knowledge or
consent.

  Nice.

  Alan DeKok.

---

From: David Woodhouse <[email-masked]>
Date: Tue, 07 Dec 2004 07:41:53 +0000

> then be cost of implementation.

Well yes, this is true; I had assumed we were speaking of things which
may be achieved by a Turing Machine.

But yes, the discussion should be about the cost of implementation. In
this case there are options which are far cheaper in terms of
interoperability, which I believe should be one of our primary 'cost'
criteria.

> free to use it almost any way they want, without my knowledge or

No, not in _any_ way they want. They can't publish web pages in your
domain, they can't run incoming mail servers for your domains, they
can't publish your DK/IIM/MS keys and they can't run your SES
message&address validation server.

-- 
dwmw2

---

From: "Alan DeKok" <[email-masked]>
Date: Tue, 07 Dec 2004 12:39:16 -0500

> may be achieved by a Turing Machine.

  Which cannot, in theory, perform certain calculations.

  My background is physics, and there's a well-known saying in that
community, which is "if you give us infinite amounts of computing
power, we will need more."

>  In this case there are options which are far cheaper in terms of
> interoperability, which I believe should be one of our primary
> 'cost' criteria.

  But do they do the same thing?  I keep seeing statements like
"proposal X doesn't break forwarding like SPF breaks it".  But when I
look at proposal X, most of the time, it doesn't do MAIL FROM
checking, but something else.  So the comparison is unwarranted.

> [ re: using my domain name ]
>
            ^^
             "almost" is the qualification I used.

> message&address validation server.

  I understand.  Systems use DNS domain names as a way of determining
where to send traffic TO that domain.  But there are no restrictions
on the use of a domain name when traffic is allegedly being sent FROM
that domain.

  For me, that's the crux of the problem.  And it's a hard problem to
solve in the physical world, too.  If you look up the cable company's
address in the phone book and visit there in person, you're pretty
sure it really is the cable company you're talking to.  But if someone
comes to your door and claims to be the cable repair guy, you have few
ways to verify he's really from the cable company.

  In the physical world, these problems are solved by calling the
cable company, and asking them if they send a 6' 250lb guy named
"bob".  If they say yes, then you're likely to let him in.

  Similar approaches should be workable on the net.  e.g. asking a
domain via DNS whether it is really responsible for certain traffic
which is using it's name.

  Alan DeKok.

---

From: David Woodhouse <[email-masked]>
Date: Tue, 07 Dec 2004 18:58:51 +0000

> checking, but something else.  So the comparison is unwarranted.

All proposals are different in their details; the important thing is
what they can _achieve_. Take CSV and SPF, for example. Of course the
details vary, but in practice they do exactly the same thing. Consider:
	MAIL FROM:<SRS0=xx=yy=ox.org=[email-masked]>
	From: [email-masked]

Precisely because SPF validates only one hop, it's achieving no more
than CSV is. The recipient can look up how much they should trust
'forwarder.org' but you can't really tell if the message _really_ came
from you.

DK is similar but it uses the RFC2822 address of the most recent sender.
It's still achieving basically the same thing -- a validated name of
some kind, which you're expected to look up in your reputation database.

SES does actually validate the MAIL FROM: address, and it survives
traditional forwarding.

> "bob".  If they say yes, then you're likely to let him in.

That sounds like a description of SES. You try to send mail from me to
somewhere like sourceforge.net, and they'll call _my_ servers and ask if
I send MAIL FROM:<[email-masked]>. When I say no, they'll reject
the mail you're offering. Try it.

The similar description of SPF would be more along the lines of "you
call the cable company and ask them if the engineer they send will be
coming from the northwest and enter through your rear gate".

> which is using it's name.

"its name". But yes. The question is how you identify that traffic. In
the case of mail, we all agree it would be stupid to identify the
traffic by the MAC address on the packets. Many of us think it's
similarly stupid to use the IP address on the packets. Many of us are
offering other ways you could identify it.

-- 
dwmw2

---

From: "Alan DeKok" <[email-masked]>
Date: Tue, 07 Dec 2004 16:34:00 -0500

David Woodhouse <[email-masked]> wrote:
> Take CSV and SPF, for example. Of course the details vary, but in
> practice they do exactly the same thing.

  They're looking at different fields in the SMTP header, so they're
doing *something* different.

> than CSV is.

  I disagree.  SPF validates the use of a name in MAIL FROM for *any*
hop.  CSV validates that an SMTP client is affiliated with a domain
name it's using in EHLO.

  The validations are orthogonal, because they are validating
different fields which do not have to be correlated.

> the mail you're offering. Try it.

  There are multiple ways of doing this kind of check.  The original
one was RMX, which had it's benefits and limitations.  SES looks to be
a little more general, in that it doesn't have the IP address
restrictions that RMX has.

  The issue SES has that RMX doesn't is that the cryptographic tokens
it uses can be copied.  e.g. Grab tokens from somewhere, and for a
short period of time, use them to send spam to third parties.  While
there are ways to fix this issue, most involve a re-thinking of what
we mean by "sending email".

> "its name". But yes. The question is how you identify that traffic.

  Use of the name in a field in a protocol.  e.g. EHLO or MAIL FROM.

>  In the case of mail, we all agree it would be stupid to identify
> the traffic by the MAC address on the packets. Many of us think it's
> similarly stupid to use the IP address on the packets. Many of us
> are offering other ways you could identify it.

  Cryptographic approaches solve a lot of problems inherent in
IP-based approaches, because they tie authentication to identity, and
not to location.

  Alan DeKok.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 07 Dec 2004 13:23:04 -0800

> SES does actually validate the MAIL FROM: address, and it survives
<snip>

A reference for SES is at http://ses.codeshare.ca/files/ses_proposal.pdf

The CLEAR wg http://www.mipassoc.org/clear/

Are proposing use of BATV that uses a slightly different encapsulation
of this extended information.  Standardizing on this encapsulation is
desired.

A reference for BATV is at http://mipassoc.org/batv/index.html

This different syntax (SES like), as proposed by William Leibzon for
BATV, was addressed by Tony Finch and extracted as follows from the
CLEAR archive:

> Consider a site that does thorough verification of email addresses,
> such that all BATV addresses are known to be valid and only appear on
> messages sent by the address's owner. Say that whether BATV is used is
> configurable per user. (This is similar to the deployment I am working
> on.) The victim has BATV turned on and the attacker has it turned off,
> so the attacker can use untagged return paths. The recipient knows a
> bit about BATV and assumes that a message with a return-path address
> that is tagged is actually from the apparent sender.
> 
> So if the recipient gets a message starting
> 
>         Return-Path: <batv=VALIDBATVTAG/victim@domain>
> 
> they correctly infer that the message was sent by victim.
> 
> However if they get a message starting
> 
>         Return-Path: <attacker+ACTUALLYALOCALPARTSUFFIX/victim@domain>
> 
> and carelessly fail to eyeball the start of the address, they might
> not notice that the message was from attacker not victim.
> 
> This is a minor human factors problem, but it's still a security
> consideration.

As an additional note-

Due to wildcard problems, SPF is unable to defend a domain as an abuser
could exploit existing records to defeat policy statements asserted
within SPF.  With there also being a lack of agreement which domain
referenced within a message should be checked, even assuming a check is
made at every MTA node, there can be no assurance the domain asserted by
the recipient as definitive is valid.  Should SPF ever be used as a
basis for reputation, there will be a great deal of effort needed to
sort out the resulting confusion and damage.

http://mipassoc.org/clear
http://mipassoc.org/mass

CLEAR and MASS are attempting to create relatively safe and less
disruptive solutions for the immediate need to authenticate an
accountable name for the mail channel.  SPF only offers authorization
(unfortunately easily spoofed or defeated), but never could it be said
that such domain has been authenticated as actually sourcing the abuse
detected through the results of SPF.

Although a provider may offer to authorize their servers for a domain
with SPF, the risks associated with respect to miss-applied reputation
will be endured by the domain owner.  The machine that has been
compromised by a security breach is not located by such an SPF based
reputation service.  The name that is important is the name of the
administering domain when dealing with security.

The provider instead should be offering to implement BATV as an
immediate solution for the spam bounce traffic that is really the
driving motivation for SPF I suspect.  This is relatively low risk for
the domain owner.  A few adjustments are still needed for this scheme to
prevent a few glitches.  Hence the need to standardize on the
encapsulating syntax.

-Doug

---

From: Dave Crocker <[email-masked]>
Date: Tue, 7 Dec 2004 14:51:43 -0800

>  > All proposals are different in their details; the important thing is
>  > what they can _achieve_. Take CSV and SPF, for example. Of course the
>  > details vary, but in practice they do exactly the same thing. Consider:
>  >         MAIL FROM:<SRS0=xx=yy=ox.org=[email-masked]>
>  >         From: [email-masked]
>  > 
>  > Precisely because SPF validates only one hop, it's achieving no more
>  > than CSV is. The recipient can look up how much they should trust
>  > 'forwarder.org' but you can't really tell if the message _really_ came
>  > from you.

SPF uses per-message validation.  It can be viewed as validating the latest-hop sending MTA, but the validation is provided by the originating sender.  Hence, SPF requires route registration.  In truth it is validating the originating sender, where the latest-hop MTA is merely a way into that information.

SPF uses per-session validation.  Its validation is based on the latest-hop sender's administration, rather than stretching back to the origin.

These are not small differences.  In fact their semantics, administration and use are entirely different.

d/
--
Dave Crocker
Brandenburg InternetWorking
+1.408.246.8253
dcrocker  a t ...
www.brandenburg.com

---

From: David Woodhouse <[email-masked]>
Date: Tue, 07 Dec 2004 22:59:02 +0000

On Tue, 2004-12-07 at 16:34 -0500, Alan DeKok wrote:
>  SPF validates the use of a name in MAIL FROM for *any*

In the SPF advocates' imaginary world where SRS is common, there's a
_much_ stronger correlation between the two validations than there is in
the real world today.

> we mean by "sending email".

Some do. SES allows the sender to encode a message-digest in the
reverse-path to prevent the replay attack. I probably wouldn't describe
that option as 're-thinking' but I've entertained too many pointless
debates on terminology (in particular 'forgery') to bother to try to ask
precisely what you mean by that word.

>   Use of the name in a field in a protocol.  e.g. EHLO or MAIL FROM.

Right. And it doesn't necessarily matter _which_ name as long as you can
be sure the sending party is really supposed to be using that name.
Which is why I don't agree with you that they're entirely orthogonal.
They're different, it's true; but what they achieve in the long run is
basically the same. 

> IP-based approaches, because they tie authentication to identity, and

Right. The cryptographic approaches make a lot more sense, as far as I
can tell.

-- 
dwmw2

---

From: David Woodhouse <[email-masked]>
Date: Tue, 07 Dec 2004 23:42:44 +0000

On Tue, 2004-12-07 at 14:51 -0800, Dave Crocker wrote:
> SPF uses per-message validation.  It can be viewed as validating the
> latest-hop sending MTA, but the validation is provided by the
> originating sender.  Hence, SPF requires route registration.  In truth
> it is validating the originating sender, where the latest-hop MTA is
> merely a way into that information.
>
> [CSV]¹ uses per-session validation.  Its validation is based on the
> latest-hop sender's administration, rather than stretching back to the
> origin.

You're missing the point of the example you quoted. Look at it again. If
you're receiving that mail from mailhost.forwarder.org to one of your
thousands of users, you have to accept it because it may well be valid.

All you can do is look up 'forwarder.org' or 'mailhost.forwarder.org'
depending on whether it's SPF or CSV you're using. Either way, you only
get to validate the "latest-hop sender's administration".

-- 
dwmw2
¹ Assumed correction -- you actually said 'SPF' again.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 07 Dec 2004 16:29:01 -0800

> merely a way into that information.

The word validation with respect to SPF is a bit vague.  A more accurate
term would be authorization.  The lack of assurance of intervening nodes
making a consistent check or their security removes assurances as to the
originating sender.  It would be better said that SPF validates the
authorizations provided by the originating sender.  Although
authorization may be sufficient to assert favorable accreditations,
authentication is required to hold a domain accountable for abuse.

> [CSV] uses per-session validation.  Its validation is based on the

More specifically, there is no assumption regarding intervening nodes
making specific checks nor assumed security.  CSV authenticates the
domain administering the sending of the mail within the strength of the
underlying IP infrastructure.  This strength permits holding the domain
administering the sending of mail accountable.  The fact that both
schemes form a response based upon the last hop misses two important
points.  One, the strength of any assertion with respect to
accountability.  Two, who is expected to respond to any problem.

> These are not small differences.  In fact their semantics,
> administration and use are entirely different.

Agreed.

-Doug

---

From: "Alan DeKok" <[email-masked]>
Date: Tue, 07 Dec 2004 20:15:06 -0500

David Woodhouse <[email-masked]> wrote:
> ... it doesn't necessarily matter _which_ name as long as you can

  What I mean by "orthogonal" is that SMTP makes no requirements on
the relationship between the values given in EHLO and MAIL FROM.  They
may use similar domain names, or they may use completely different
domain names.  So the fields are "orthogonal", because there is no
dependency relationship between them.

  Now, *validation* of those fields can use correlations between their
values to make decisions:

  e.g. EHLO mta.example.com
       MAIL FROM <[email-masked]>

  Since the recipient discovers for himself that the values of the two
fields are correlated, he may choose to "merge" the field validation.

  e.g. "example.com says this IP really is mta.example.com, and is
permitted to act as an SMTP client, so I'll assume that I don't have
to ask example.com again about the use of it's name in the MAIL FROM
field"

  If the two fields had different domain names, then the validation of
those fields would have to be independent.

  Alan DeKok.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 07 Dec 2004 18:44:46 -0800

On Tue, 2004-12-07 at 16:29, Douglas Otis wrote:

Clarification of a clarification...

> authorizations provided by the [purported] originating sender.  Although

-Doug

---

From: Dean Anderson <[email-masked]>
Date: Tue, 7 Dec 2004 22:20:39 -0500 (EST)

> those with admin rights are 'responsible for a valuable asset' (to use

Not at cross purposes. I understand exactly what you mean. However, for
all practical purposes _anyone_ who wants to send mail has already or can
obtain administrative control over their machine.  Put another way, nearly
all mail is sent from PC's or some sort of appliance-like machine.  The
sender and the "administrator" are the same person.  They can send from
any port they want.  Sending from a port < 1024 means nothing.  

Last night, though, I did think of one reason this isn't such a good idea:  
Most unix/linux machines still enforce a requirement that to open a port
less 1024 one must be root. This is a remnant of the days when machines
were expensive, and it was thought that having "privileged" ports was a
security feature. Presently, many mail systems give up root privilege as
soon as possible to reduce root exploits, and so that mail queueing on
such machines is performed by unprivileged processes. 

Putting your requirement into SPF would necessitate either removing the
ancient "privileged ports" enforcement from the Linux/Unix kernel
altogether, or reducing the security of the system.

Non-linux/unix systems usually don't implement any notion of "priviledged
ports". Even on Unix/Linux systems, its generatlly been more of security 
flaw than a feature.

> And also because ISPs do rate limiting and spam checking on outgoing

This is another myth.  It is extremely difficult (to the point of
practically impossible) to do rate limiting on outbound email for mail
systems of any size.  There was a rate-limiting freeware posted on the net 
some time ago, but it didn't work.

I know of no ISPs that do Spam checking on outbound email, either.  The 
notion of 'what is spam' depends on the recipient, not on the sender.

> Not necessarily. You may need SMTP AUTH and 

SMTP AUTH does not help. That information is already on the compromised 
machine. Or if its a disposable account, then it was given to the spammer.

The spammer/abuser has _ALL_ the privileges and capabilities of the user 
of the ISP.  

> there's no guarantee that the _incoming_ addresses of the cluster are
> the same as the outgoing addresses.

There is no guarrentee other than the economics that ISPs don't buy
unnecessary mail servers.  Mail servers are expensive, and cost money to
operate and maintain. Separating inbound from outbound mail servers makes
sense becaue the inbound mailservers have extra load of the mailboxes, and
creating a separate set of outbound mailservers conveniently helps
distribute load.  But one would be unlikely to put in additional
unnecessary layers.  Economics would soon reverse such a decision.

> You can't -- you'd throw away too much valid mail.  But you could say
> the same for _any_ spam filtering. SpamAssassin makes it worse far more
> than SPF does, because _far_ more people will reject for having a high
> SA score than they will for SPF. You're right, but I think it's a red

Yes: You can't reject bcause not everyone uses SPF.  But the _goal_ of SPF
is to /reject/ forged mail, and thereby prevent forgery.

	SPF rejects forged mail
	We can't use SPF to reject email
	thereore, we have no rejection problem.

This is only true as long as no one uses SPF to reject mail. But if no one
uses SPF to reject forged mail, then it didn't prevent forgery.

If you use SPF to reject forged email, it is now trivial for the abuser to
arrange for 100% of the bounces to hit the forged sender's mailbox.  That
makes the problem _much_ worse. Spamassasin doesn't do anything like that.

Its only "not worse" if SPF isn't used.  And to the extent that SPF is
used, its made worse. Suppose yokel.com starts rejecting mail based on SPF
records. Now, abusers can forge mail to to yokel.com. Yokel.com rejects
the mail, and the target gets 100% of the email.

I saw exactly this sort of abuse of open relays. Abusers would queue up
mail to a domain that blocked open relays. The relay then bounced every
message.  Through testing with different IP addresses and detailed
logging, I found that the abuse was usually performed by people associated
with open relay blacklists.  I've learned a lot over the years about open
relay abuse and abuse detection.  I'll just summarize that the most
effective tool was to block the open relay blacklists. They were the only
ones scanning for open relays.  Abuse only happened after the blacklists
successfully scanned a relay.  Preventing the scans prevented the abuse.  
Finally, the blacklists shut down, and abuse dropped off almost 
completely.

SPF basically turns every relay into an open relay, and allows the same
sort of abuse, only worse, but without the need to scan for open relays.  
If one were to determine authorship of SPF by its effects, one would think
that open relay abusers wrote SPF.

> running identical spam and virus checking, for example.

I agree that its not necessarily specific to SPF.

---

From: Mark Shewmaker <[email-masked]>
Date: Tue, 07 Dec 2004 22:48:34 -0500

On Mon, 2004-12-06 at 18:34, David Woodhouse wrote:
> I'm suggesting that one of
> a privileged port.

I see three types of solutions:

1.  Publish valid outgoing ports that sending servers will use, and
    make sure that your sending mail server users, and only your sending
    mail server users (or programs), use those ports.

2.  Publish lists of user IDs that may send mail from your machine, (or
    publish a mapping of user IDs and valid user names), and set up your
    identd server to answer queries about outgoing smtp servers 
    appropriately.

But in both the above cases, you're configuring ports and user id's that
are allowed to do something, publishing a ruleset, and asking *someone
else* to check the ruleset.

Since you have all the information available yourself, why not just do
egress checking yourself, and not burden the receiver?

To me that's the more reasonable solution, (and it doesn't require you
to expose your internal policies to the outside world):

3.  Make sure that no mail *can* leave your machine that violates your
    policies.

    This means doing egress checking to prevent cross-customer
    forgeries, for example, as well as spf egress checks, to make sure
    that mail is never sent from your machine that would result in an
    spf fail at the destination.

The simplest way of doing #3 is to firewall outgoing connections to
remote port 25's, requiring all code that makes such connections to go
through trusted code instead.  (This assumes that your trusted mail
submission code follows your policies.)

Untrusted code that attempts to make outgoing port 25 connections
directly would then fail, or perhaps you could set up a local
transparent proxy through which trusted code would do the necessary
egress checking.  (I think this makes the most sense, and I expect it to
become easier to do over time.)

Note that if you publish your "port from" rules, you're going to have to
make the same sort of decisions about the above, making sure that
activities that violate your policies are detectable by others, so
others can act upon them.

But if those policy violations made within your own machine are
detectable by others, instead of asking others to enforce your internal
policies, to me it makes more sense to just detect these violations and
enforce them yourself.

-- 
Mark Shewmaker
[email-masked]

---

From: Dean Anderson <[email-masked]>
Date: Wed, 8 Dec 2004 00:08:16 -0500 (EST)

> in practice.  The only question then is whether the cost is acceptable.

Its not possible in theory with SPF.   In theory, SPF can't achieve the 
goal it set out to achieve.  

>   If it's not possible in theory, I'm curious to know why.

We're just going over the reasons, for purposes of summarizing them.

> If not, why not?  

Its not a matter of "permit".  Its a matter of "is it possible to control
abuse of your domain", and more specifically, its a matter of "does SPF
enable control over your domain".  SPF offers no control for about 13+
reasons.

Of course you are permitted to control your domain. You can sue anyone you
want to.  But there is no technical way to make it impossible for someone
to forge your address using the tools proposed by SPF.

> If so, then I'm free to implement SPF, or anything other scheme I like.

You are always free to shoot yourself in the foot.  We can't stop you from
doing so; we can only explain that you are about to do so, before you do
so.

---

From: Dean Anderson <[email-masked]>
Date: Wed, 8 Dec 2004 00:18:09 -0500 (EST)

> reject mail which the {backup MX, relay, forwarder} does not reject.

Err, no. SPF does make the blowback problem much worse.  Other schemes 
create a few percent blowback. SPF enables 100% blowback. Thats much 
worse.

> That may be your experience; it's not mine.

That's a general experience with any large mail system. Its also a problem 
where different vendors provide backup mail services.  Its only not a 
problem on simple, small single server systems.

---

From: Dean Anderson <[email-masked]>
Date: Wed, 8 Dec 2004 00:26:19 -0500 (EST)

> Spammers don't complain.   If there are FALSE positives for legit systems,

That's because spammer's rushed to install SPF records.  Of _course_ they 
don't complain.   That ought to be a signal right there.

That's rather like saying the insurgents don't complain about soldiers
roughing people up, or leaving the WMD sites with the very high explosives
unguarded. 

		--Dean

---

From: "Chris Haynes" <[email-masked]>
Date: Wed, 8 Dec 2004 09:18:17 -0000

> create a few percent blowback. SPF enables 100% blowback. Thats much

I hesitate to (re-)enter this thread, and, once again, I'm trying to be fair and
understand the substantive basis for concerns..

It has been argued in SPF circles, that if you receive a message which the
(purported) sender's policy declares to be a hard failure (-), the message is
_proven_ to be a forgery, an unauthorised re-transmission, or whatever.

Since the purported sender has repudiated the message, the argument goes, the
original SMTP 'contract' to "deliver or bounce" is null-and-void, since whoever
actually injected the message did so without the authority of the domain they
cited.  Therefore it is acceptable to 'silently discard' such messages, and not
send bounces.

Do you have any sympathy with that reasoning, and would it change your view
about 100% blowback?

Or is there some other mechanism within SPF which accounts for your '100%
blowback' concern?

Chris Haynes

---

From: David Woodhouse <[email-masked]>
Date: Wed, 08 Dec 2004 12:33:19 +0000

> understand the substantive basis for concerns..

_You're_ trying to be fair? I actually found myself defending SPF
slightly :)

> cited.  Therefore it is acceptable to 'silently discard' such messages, and not

If SPF (or indeed anything) is checked by a receiving site, it should
cause a _rejection_. Neither 'silently discard' nor 'send bounces' is
acceptable behaviour. If the recipient doesn't like the mail, it
shouldn't accept it in the first place.

Sending bounces is obviously bogus, if you think it didn't come from
that reverse-path in the first place. Silently discarding it is also
bogus since SPF has too many false negatives and you'd be throwing away
valid mail without even letting the sender know you did so.

By rejecting mail at SMTP time, you cause the majority of the spammer
zombies to move on to the next address without comment, and the genuine
forwarder to send a bounce.

> Or is there some other mechanism within SPF which accounts for your '100%

Personally I don't agree with Dean's 'blowback' concerns; I don't see
that SPF makes it any worse than any other scheme which would reject the
same mail. But I'm not going to argue any more in favour of SPF :)

-- 
dwmw2

---

From: "Chris Haynes" <[email-masked]>
Date: Wed, 8 Dec 2004 15:45:53 -0000

> _You're_ trying to be fair? I actually found myself defending SPF

Yes, I noticed - a fascinating spectator-sport!

I was interested to seek Dean's extended views on blowback.

Chris

---

From: "Alan DeKok" <[email-masked]>
Date: Wed, 08 Dec 2004 13:23:46 -0500

"Chris Haynes" <[email-masked]> wrote:
> It has been argued in SPF circles, that if you receive a message
> which the (purported) sender's policy declares to be a hard failure
> (-), the message is _proven_ to be a forgery, an unauthorised
> re-transmission, or whatever.

  i.e. in SPF, the argument is that the discard "should" be done
during message delivery phase, by the sender.  This requires
non-SPF-aware senders to change their behaviour when they interact
with SPF-aware receipients.

  That won't work.

  In other MAIL FROM validation schemes (SES, BATV), the MAIL FROM can
be rewritten so that the forwarder implementing the scheme also
accepts responsibility for the bounces.  This means that any "discard"
decision will be made by the sender, when the recipient rejects the
message.  This may change they way senders work, but that's OK,
because the sender has already changed to implement SES.

  Or, the discard decision is made by the receiver, during the
"bounce" phase, when the sender refuses to accept the bounce.  This
requires no changes on non-SES-aware senders, as it's already a known
failure mode of message delivery.

> Since the purported sender has repudiated the message, the argument
> goes, the original SMTP 'contract' to "deliver or bounce" is
> null-and-void, since whoever actually injected the message did so
> without the authority of the domain they cited.  Therefore it is
> acceptable to 'silently discard' such messages, and not send
> bounces.

  If attackers discover MAIL FROM values which temporarily pass SES
checks, they can re-use them to send messages through non-SES-aware
senders.  When SES-aware recipients discover that the alleged
originating site has repudiated the MAIL FROM value, they will reject
the message, and the sender may bounce it back to the alleged
originating site.

  This failure mode is the same as Dean's "blow-back" argument for
SPF.  (Assuming I've described SES correctly, based on my quick
skimming of the documents.)

  It's also a attack mode of *any* system doing MAIL FROM validation.
Until all sites implement the validation, the attack mode will exist.
It can be mitigated with SES-like schemes, but it's more difficult to
mitigate it with SPF.

  Alan DeKok.

---

From: John Levine <[email-masked]>
Date: 8 Dec 2004 18:47:51 -0000

>re-transmission, or whatever.

If the policy is only "-all", i.e., this domain sends no mail at all,
then the policy is credible.  If the policy is anything else followed
by -all, it's not.  One of SPF's many problems is that it posits a
model of the e-mail world that is a lot simpler than the real world.

In the real world, there are whole lot of remailers and forwarders,
and no matter how desperately some domains might want to argue that
nobody's allowed to forward, etc., those arguments are about as
persuasive as the boilerplate likely found at the bottom of their mail
that says "if we sent you this mail by mistake you must hand deliver
it back to us or go to jail."  People want their bank statements or
whatever, even if the address they give to the bank is their permanent
forwarding address from their alma mater.

I do agree that bounces vs. rejects are an increasing problem, and
I'm having trouble figuring out if there's any way I can do bounces
reasonably for the fraction of the mail here for which I can't tell
if it's deliverable at SMTP time.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
http://www.taugh.com

---

From: David Woodhouse <[email-masked]>
Date: Wed, 08 Dec 2004 19:20:27 +0000

> if it's deliverable at SMTP time.

The only real answer I've found is to make sure that during _normal_
operation, there isn't actually any mail in that category ("can't tell
if it's deliverable at SMTP time"). 

I do recipient verification callouts (with caching, of course) for the
domains for which I am MX backup (and in some cases, also for mail I'm
forwarding). Obviously it's pointless being an MX backup unless you
actually accept the mail while the primary is down; but that's not
normal operation.

-- 
dwmw2

---

From: Douglas Otis <[email-masked]>
Date: Wed, 08 Dec 2004 13:58:32 -0800

> actually accept the mail while the primary is down; but that's not

The spammer often ignores MX priorities specifically to discover SMTP
servers that accept mail for a particular domain without respect to
valid users.  There is still an advantage using BATV even in the case
where there is a bounce generated in this scenario.  The receiving MTA,
if using BATV, should be able to reject a bounce when the MAILFROM is
NULL. This is the reason for BATV.  

It seems the proper etiquette for a bounce would be to ensure a NULL
MAILFROM when issuing a bounce.  This behavior should include virus
detection, vacation notices, and other message related events generated
as a result of a message being rejected and returned.  Unfortunately,
the setting of MAILFROM for a bounce is optional. : (

-Doug

---

From: David Woodhouse <[email-masked]>
Date: Wed, 08 Dec 2004 22:27:22 +0000

> NULL. This is the reason for BATV.  

Yes. That's precisely why I'm _slightly_ less concerned about avoiding
bounces nowadays, since SES and BATV allow the 'victim' of bogus bounces
to reject them. SES just takes it a little further than BATV, by
offering simple ways for the (potential) recipient to verify the source
address too.

> detection, vacation notices, and other message related events
>  generated as a result of a message being rejected and returned. 
> Unfortunately, the setting of MAILFROM for a bounce is optional. :(

For a bounce as defined by RFC2821 it's _not_ optional to use an empty
reverse-path; it's mandatory. For vacation messages it is optional -- in
fact I was utterly dismayed to see that RFC3834 said that an
autoresponse 'MAY' have an empty reverse-path, rather than 'MUST' or at
least 'SHOULD'.

I consider autoreplies (and especially bounces) with non-empty reverse-
path to be an attempted denial of service attack, because of the mail
loop they threaten. I tend to report them to the upstream network
provider of the offender as such, in the strongest possible language.
Especially if they're "responses" to viruses which are known to fake
their sender.

-- 
dwmw2

---

From: Alex van den Bogaerdt <[email-masked]>
Date: Wed, 8 Dec 2004 23:44:32 +0100

On Wed, Dec 08, 2004 at 06:47:51PM -0000, John Levine wrote:

> nobody's allowed to forward, etc., [snip]

This is bogus.  Forwarding does not have to stop.

SPF by itself does not make a distinction between forgeries as a result
of "old-style forwarding" and spam, virus, etc.

People claming that the SPF community wants to ban forwarding are
spreading fud, no matter their reputation.

Alex

---

From: Dean Anderson <[email-masked]>
Date: Wed, 8 Dec 2004 19:42:36 -0500 (EST)

On Wed, 8 Dec 2004, Chris Haynes wrote:

> original SMTP 'contract' to "deliver or bounce" is null-and-void, 

The mailserver getting the rejection code and generating the bounce may
not participate in the SPF "contract modification".  One would also have
to create a special 5xx code to indicate "reject, but don't bounce", and
then upgrade all mailservers everywhere.

And I suspect some people might not accept silent rejection in principle,
and send bounces anyway.

It is probably easier for the recipient mailer to receive but not deliver
(silent discard) But silent discards are also particularly difficult to
debug.  And you wouldn't want to do this unless everyone on the planet had
SPF records.

> since whoever actually injected the message did so without the authority
> of the domain they cited.  Therefore it is acceptable to 'silently
> discard' such messages, and not send bounces.

It isn't necessarilly the case that SPF-rejected email is forged or
unauthorized. It might just be the hosting company isn't allowing
outsourcing of the domain. (or is doing "unreliable" name service to harm
the other company and gain back the outsourced business).  It would not be
acceptable to silently discard these messages, since one would never know
why the message wasn't received. Silent rejection seems to be a bad
approach.

And of course, SPF rejections might happen simply because the IP addresses
of the authorized servers changed, but the TTLs on the SPF records haven't
expired.  This would be new blowback on otherwise good email. This could
go for weeks or months after an IP address is changed.

IP Addresses are frequently changed unexpectedly as a result of abusive
blacklists that can't be taken to court.  Since you can't sue anonymous
blacklists like SPEWS, when they block legitimate business, the only
alternative is to get a new IP address. This, of course, is unplanned.  
No doubt the abusive blacklist would appreciate the ability to create
additional interference.  That should probably be #14 on the list.

> Do you have any sympathy with that reasoning, and would it change your view

Ignoring the just-described problems with silent rejection, silent
rejection could reduce blowback, but it still requires replacement of all
current mailserver software. That would take many years to achieve.

In the meantime, there is 100% blowback. 

I note that recent documents from Comcast show that it would cost $58 
million just to tell their users to change relays, at $9 per support call.
Upgrading all the mailserver software on the planet must be fairly 
expensive. And it might never be really finished.

And whatever system you put into place will still be abused, perhaps 
differently, but still abused just the same.

		--Dean

---

From: David Woodhouse <[email-masked]>
Date: Thu, 09 Dec 2004 01:13:12 +0000

On Wed, 2004-12-08 at 16:48 -0800, kent crispin wrote:
> On Wed, Dec 08, 2004 at 10:27:22PM +0000, David Woodhouse wrote:
> > loop they threaten.
> How about an autoreply from majordomo, asking you to confirm your 
> subscription request?

I'd probably include that, yes -- if responding to a request made by
email, it should go to the reverse-path of the incoming subscription
request, and should have an empty reverse-path if its own. 

This would would prevent the autoreplies from being accepted (and
broadcasted) by other mailing lists.

-- 
dwmw2

---

From: David Woodhouse <[email-masked]>
Date: Thu, 09 Dec 2004 01:18:38 +0000

> of "old-style forwarding" and spam, virus, etc.

Your references to 'old-style' forwarding and 'forgery' will be a lot
more credible when RFC4822 comes out and indicates that forwarding sites
SHOULD NOT behave in the way they've always done by preserving the
reverse-path.

I consider that a prerequisite for any standardisation of SPF; to do it
in the other order is putting the cart before the horse.

-- 
dwmw2

---

From: John Levine <[email-masked]>
Date: 9 Dec 2004 01:37:29 -0000

>This is bogus.  Forwarding does not have to stop.

Hmmn.  I'm not sure how to respond to someone who seems to think that
I wrote the exact opposite of what I did.

I said that forwarding happens, and that SPF records saying "no
forwarding" aren't going to stop it.  Recipients can believe -all at
the time when every forwarder has implemented SRS, that is to say,
never.  Until then, if you actually want to accept the legitimate mail
that people are sending you, SPF is only useful for whitelisting
domains you already know you like.

>spreading fud, no matter their reputation.

Tendentious non-sequiturs don't advance your argument very effectively.

Regards,

---

From: Alex van den Bogaerdt <[email-masked]>
Date: Thu, 9 Dec 2004 10:40:17 +0100

> I wrote the exact opposite of what I did.

I think you wrote:
"Some domains argue that nobody is allowed to forward".

I respond that this is not the case, in other words I don't think it's
true that SPF wants to stop forwarding.

> forwarding" aren't going to stop it.  Recipients can believe -all at

Here you say it again.  SPF does NOT say "no forwarding".
SPF _does_ say: "No spoofing my name".

If you want to forward then that's fine.  Just don't use my name
as the _sender_ of _your_ message.

> Tendentious non-sequiturs don't advance your argument very effectively.

I needed to look this ("non-sequiturs") up and I don't see why you
want to choose this word.  

Alex

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 09 Dec 2004 11:48:07 +0100

John Levine wrote:

> One of SPF's many problems is that it posits a model of the
> e-mail world that is a lot simpler than the real world.

KISS is a feature and no problem.

> In the real world, there are whole lot of remailers and

Remailers (like real mailing lists) have no problem at all
with SPF.  Some arguments "against" SPF from say the EFF
are far beyond surreal.  Any anonymous remailer is by its
own definition automatically "SPF ready".

> no matter how desperately some domains might want to argue
> that nobody's allowed to forward, etc.

These arguments are nonsense, you can forward as much as you
like.  You only have a problem if you check SPF behind your
"border MTA", but the good news is, this is _your_ problem.

> People want their bank statements or whatever, even if the
> address they give to the bank is their permanent forwarding
> address from their alma mater.

Something on their side has to give.  It's not a problem of
the senders, they only publish which MTAs send their mail to
third parties, that's their border.

> bounces vs. rejects are an increasing problem, and I'm having
> trouble figuring out if there's any way I can do bounces
> reasonably

You could try SPF, if the HELO resp. MAIL FROM sender policy
results in a FAIL simply reject the mail.  You're only forced
to bounce if you forward mail and the next hop rejects it.

Otherwise it should be possible to come to a decision about
"reject" vs. "accept" during the SMTP dialogue.  In theory.

BTW, something I've seen in MASS, you said that "most" mail
today contains HTML.  Is that incl. or excl. UBE / UCE ?

If 4 of 5 mails are spam and other unsolicited stuff today,
and 4 of 5 spam mails contain HTML, then 64% of all mail
are both unsolicited and contain HTML.  But it's not clear
how often HTML is used in solicited mail, I doubt that the
majority contains HTML.
                          Bye, Frank

---

From: David Woodhouse <[email-masked]>
Date: Thu, 09 Dec 2004 12:06:13 +0000

> SPF _does_ say: "No spoofing my name".

You're arguing over nomenclature, which is pointless.

Alex, you're right then SPF advocates do indeed phrase it as "No
spoofing my name". But John is also right because what they _mean_ by
that is "no forwarding", since normal forwarding does involve, and
always has involved, the behaviour which SPF advocates now want to call
"spoofing".

The SPF advocates are suggesting that people do something else instead
of normal forwarding, but they haven't really got a coherent and
deployable answer yet as to _what_ else should be done.

-- 
dwmw2

---

From: Alex van den Bogaerdt <[email-masked]>
Date: Thu, 9 Dec 2004 14:05:42 +0100

> You're arguing over nomenclature, which is pointless.

That is your opinion.  For me, the difference is quite relevant.

> always has involved, the behaviour which SPF advocates now want to call

If you know what "SPF advocates" mean to say, why do most (if not
all) disagree with your opinion?

Could it be that you are spreading your opinion, disguised as theirs?

I think that the SPF community is well aware of the fact that there
is a number of people using a way of forwarding that will become
impossible in the future (if and when SPF is deployed).  I don't
think all of them consider this way of forwarding "normal".  Perhaps
most of them agree on this method being "in use", but that doesn't
mean it is THE right way.

There are alternatives. You just don't want to accept them because
YOU think they are unnecessary alternatives.  Newsflash: Others
have opinions to and they may not agree with yours.

Alex

---

From: Douglas Otis <[email-masked]>
Date: Thu, 09 Dec 2004 11:00:47 -0800

> have opinions to and they may not agree with yours.

There are two flaws with respect to this "do not spoof me" assertion. 
One, there are ways to prevent the SPF record from being discovered from
sub-domains negating supposed spoof protections.  Two, there is no way
to know if the recipient of a message from such an SPF-domain will be
forwarding the message.  These forwarded accounts are utilized by a high
enough percentage of the populous as to cause inordinate support issues
should this mail become categorically rejected and perhaps lost.

One of the alternatives offered by SPF for this problem is the use of
the '?all' statement.  In other words, SPF becomes a scheme of graduated
levels of authorization and, of course, with the continuation of
spoofing.  This also means any shared MTA authorized by the SPF record
also allows the SPF-domain to now be spoofed where the message obtains
the highest level of authorization.

Deployable "do not spoof me" solutions that also have a chance of
actually working are being developed within the MASS wg. 

http://mipassoc.org/mass/

-Doug

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 09 Dec 2004 23:10:47 +0100

Douglas Otis wrote:

> there are ways to prevent the SPF record from being
> discovered from sub-domains negating supposed spoof
> protections.

What are you talking about ?  If your domain is an.example,
and your users use MAIL FROM:<[email-masked]> via your MSA,
which says HELO msa.an.example, and if you have SPF sender
policies "v=spf1 a:msa.an.example -all" for an.example and
"v=spf1 a -all" for msa.an.example, where's the problem ?

> there is no way to know if the recipient of a message from
> such an SPF-domain will be forwarding the message.

Yes, the "S" in SPF stands for "sender policy".  As soon as
the receiver does interesting things they are his business.

If the receiver checks SPF at the wrong place resulting in a
FAIL and reject, then he should fix his routing, or find any
other solution for his problems, the sender didn't cause it.

                       Bye, Frank

---

From: Douglas Otis <[email-masked]>
Date: Thu, 09 Dec 2004 15:55:39 -0800

> "v=spf1 a -all" for msa.an.example, where's the problem ?

Those wishing to spoof a domain could add a label that already has a
record such as- 

MAILFROM:<[email-masked]>.

The query for the SPF TXT RR at name_of_inbound_smtp_server.an.example
will fail.  They may even HELO some_existing_record.an.example.  The
fact that IP addresses do not match will not be a problem.  No SPF TXT
records will have been found however.

> other solution for his problems, the sender didn't cause it.

The "all authorized senders are listed" policy causes the problem! 
There is no way to know in advance which recipient forwards mail so, in
essence, this is a false claim that damages the integrity of the mail. 
Unless this damage is intentional with the belief mail forwarding, as it
exists today, should not be used, no SPF record should claim "-all". 
There is no right place for these records.

-Doug

---

From: David Woodhouse <[email-masked]>
Date: Fri, 10 Dec 2004 09:29:09 +0000

> MAILFROM:<[email-masked]>.

Or even MAIL FROM:<SRS0=xx=yy=an.example=[email-masked]>

SPF is by its very nature a hop-by-hop mechanism; it cannot give a true
end-to-end indication of forgery.

The problem is that we need to know with high accuracy which mails are
_spoofed_. SPF can only tell us for sure which mails are _not_ spoofed.
We want a blacklist; we have a whitelist. To go from one to the other is
not a simple negation.

-- 
dwmw2

---

From: Dean Anderson <[email-masked]>
Date: Sun, 12 Dec 2004 19:00:17 -0500 (EST)

> spreading fud, no matter their reputation.

No one is saying the SPF community wants to _ban_ forwarding.  At least, 
I've not heard anyone say that.

Rather, they want to extort^H^H^Hract money for the ability to outsource
or separate parts of the mail system. They can also mess with the service
level of the other provider in ways that aren't easily detectable by the
user.

The blowback issue is different from this.  Blowback happens whenever
anyone _rejects_ emails based on SPF.  A bounce is generated from the
relay to the forged sender.  This enables abusers to arrange 100% blowback
as the abuse vector.

So, SPF rejection can't be allowed.

		--Dean