Subject: Re: A new SMTP "3821" [Re: FTC stuff...........]

From: "Hector Santos" <[email-masked]>
Date: Sat, 4 Dec 2004 22:13:20 -0500

----- Original Message -----
From: "Matthew Elvey" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Sent: Saturday, December 04, 2004 9:34 PM
Subject: Re: A new SMTP "3821" [Re: FTC stuff...........]

> >1) EHLO/HELO validation
> >2) MAIL FROM validation
> >3) RCPT TO validation
> >4) DATA validation, if any
> >5) Mixed Validation
> >
> >
> What is this?

What are mixed validation or Mixed Policies?

In short, it is coupling the validation logic of lets say what a MAIL FROM
validation will produce vs a HELO validation will produce.   It is illogical
to have a "valid" result where another data point is invalid, vice versa.

There are many other examples of mix policies.

> >CSV concentrates on #1, gets lost on SMTP AUTH issues,
> >

> I 'm pretty sure you're confused.  Can you point to what part of the
> spec you're referring to, so we can make it clearer?
> Legit MUAs  only send mail to MTAs they authenticate to.  This is the
> only hop in which CSV suggests SMTP AUTH as *A* solution (not the
solution)

The above was an example of a MIX policy.

CSV presumes to validate the client domain name when it is possible there is
an incoming ESMTP AUTH session, in which case, in my technical opinion, CSV
does not apply.

A pending mix policy may also exist at MAIL FROM and certainly at RCPT TO.

Hope this helps.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 5 Dec 2004 16:48:36 -0500

----- Original Message -----
From: "Matthew Elvey" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Sent: Sunday, December 05, 2004 12:52 PM
Subject: Re: A new SMTP "3821" [Re: FTC stuff...........]

> >an incoming ESMTP AUTH session
> >

> That is false, to my reading of the specs:
> "To validate an SMTP session from an unknown sending SMTP client" is
> from the fourth line of the main spec.
> If there's an incoming ESMTP AUTH session, then there is no "unknown
> sending SMTP client", only a "known sending SMTP client".
>
> What language would you suggest since this isn't clear enough?

The language is fine.

The point is in order to accomplish this, the CSV (and others) state machine
must be based on a delay design mechanism.

Do I need to explain this?

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 5 Dec 2004 21:37:49 -0500

----- Original Message -----
From: "Matthew Elvey" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Sent: Sunday, December 05, 2004 6:03 PM
Subject: Re: A new SMTP "3821" [Re: FTC stuff...........]

> On 12/5/2004 1:48 PM, Hector Santos sent forth electrons to convey:
>
> Yes. I don't know the term and
> http://www.google.com/search?q=%22delay+design+mechanism%22
> brings up no hits.

Ok, fair enough,  I will explain it (see below) but keep in mind, a little
more elbow grease in your googling. i.e, using words like "SMTP DELAY
VALIDATION", etc, would of produced some hits.

This is certainly not the first time this discussion or related concepts
were discussed at some level.

> The only relation of
> "delay" to CSV is that there's a delay when a domain with a good
> reputation goes bad and that info needs to be discovered and propagated.

No, no relationship to mixed validation ideas and issues.

I will try to give you an example.  I will do so with specific CVS examples,
but keep in mind the same or related issues applies to all the proposals.

Keep in mind we are a product developer and the implementation of any new
idea into the system goes thru a very thorough "Software Walkthrough" review
process with many issues to be considered.

In regards to SMTP, the following top design goals were sought:

- Maximum SMTP Compatibility,
- Maximum Spoof protection/Validation at the transaction level,
- Maximum Efficiency and Scalability.
- Minimal to no interference with 2822 mail interpretation.

The last one is based on our long history on online hosting product
experience (second to none) in dealing with US ECPA legal and related
issues.  In short, we don't concern concerns with the administrative or
end-user level mail interpretation issues.  Any new security ideas that will
stop the 'flow" of mail must be maintained at the transaction level for
product liability reasons.  PS: I don't wish to go into this with anyone.

Of course, it should go without saying that any new SMTP concepts should
deal with compatibility issues as well.  Like any other vendor, we don't
wish to "break" their operations.

However, this is very important in the area I feel is very important in
distinguishing problem we are trying to address.  In short, our design
philosophy from the git-go has been the Spoof Problem is one that is a based
on an "Anonymous Final Destination Transaction" (AFDT).

The reason is straight forward:

The world wide SMTP internet email network is predominately based on the
following fundamental principle:

- Trust is required for Relay or Routed transactions,
- No Trust is required for final destination transactions.

In other words, for relay or routed transactions, the client/server
transaction has already negotiated a "trust" relationship.  In practice,
this is traditional done using one or more of the following:

-  IP Relay Tables
-  ESMTP  AUTH
-  POP3 BEFORE SMTP

However, for a AFDT the "network" has worked because the above trust
relationship is not required.  Yet, this is where the exploitation of SMTP
begins and it is the essence of the Spoof Problem that exist today.

So for lack of a better term, "Anonymous Final Destination Transactions' is
where SMTP lacks strength and hence needs new ideas to help protect against
the abuse of the system.

If you don't agree, you might as well stop here.  But if you want to
understand what is meant by delay or mixed policies, then continue reading.

Now, lets review the SMTP state machine:

o connection state
o HELO/EHLO state
o MAIL FROM state
o RCPT TO state:
o DATA state:

Since we are attempting to solve this problem in non-2822 terms, only in
2821 terms, the data points for a transaction are:

CIP - Client IP address (at the socket accept level)
CDN - Client Domain Name (issued at HELO/EHLO)
RP  - Sender Return Path (MAIL FROM)
FP  - Forwarding Path (RCPT TO)

Using the following functional model to depict old (current) vs. new end
point validation concepts:

    result = EPV_OLD(cip, cpn, rp, fp)
    result = EPV_NEW(cip, cpn, rp, fp)

We can now begin to show how all the data points are related and/or mix
policies analysis is required to be considered.

Given what was stated above about AFDT,  no "new SMTP validation logic" is
required until it is known whether the transaction is a route or a final
destination.

Therefore,  one SMTP implementation would to delay all validations until FP
is provided at RCPT TO:

    if FP = Route then
        Result = EPV_OLD(cip, cpn, rp,fp)
    else
        Result = EPV_NEW(cip, cpn, rp,fp)

This says that if the forwarding path is for a route, then we only need to
use traditional, backward compatible SMTP authentication/validation trust
negotiation ideas that already exist and are standards in the market place.

It also says that if FP is not a route but for a final destination, then
this is where we need to apply new validation ideas that are not currently a
standard.

What does this all mean?

Well, from a design standpoint, you have two options:

- An open-ended check for each data point as they are provided, or
- A more efficient check based on one or more of all the possible data
points.

Lets use CSV examples:

Example 1 is the IDEAL "Good Transaction.  All data points are valid.

connection ip:  64.234.45.8
HELO mail.elvey.com
MAIL FROM:  [email-masked]
RCPT TO: [email-masked]

In this case,  you are connecting to our MX server at winserver.com, from
your own valid IP machine, issuing a valid client domain name and a
non-spoofed valid return address.  Its all good.

Example 2 is the bad transaction where you trying not authorized to route
via our server:

connection ip:  64.234.45.8
HELO mail.elvey.com
MAIL FROM:  [email-masked]
RCPT TO: [email-masked]

A few questions now come to mind:

a) At which point do we apply CSV?
b) Can we use CSV to authenticate the route?

If you had follow this so far,  CSV is only needed at  example #1 for a
AFDT.

In example #2,  you need to follow standard SMTP authentication on our
system in order to route.  You have to have a prior relationship, either as
a MUA user or another trusted router.

So unless you apply a delay validation technique, you are forced to use a
open-ended check prior to the FP is known.   This is a very inefficient mode
of operations which has been proven to be the case in practice that require
DNS overhead just like CSV does.

Lets look at Mixed Policies now.

A new example #1.1 is similar to example #1 as a final destination
transaction but with a different sender, not you.  I could of used your
address, but I wanted to highlight the difference.

connection ip:  64.234.45.8
HELO mail.elvey.com
MAIL FROM:  [email-masked]
RCPT TO: [email-masked]

It is an anonymous final destination and a transaction where additional
validation is required.  We are using CSV here and lets assume you are a
member of Doug's new CSV/DNA authentication database.

Lets also assume that the efficient delay validation is used so that CSV is
applied after RCPT TO is provided.  The validation is delayed.  No overhead
in this system.

According to CSV specs, since you are authenticated via CSV/DNA based on the
CDN (HELO client domain) and CIP (client IP address), then we should "trust"
this transaction,  however, the CSV does state that additional validation
may apply depending on the implementation.

Since we have very suspicious sysops and since CSV does not address RP
(return path) validation,  a system may which to add additional logic such
as SPF1 or SENDER ID  to validate the return path domain.  Keep in mind that
SPF1/SENDER ID or LMAP in general  validates just the domain part of the
return path. Not the complete address.

In any case, for the sake of the example, a SPF1/SENDER ID check results as
a FAIL.

You know have a mix policy:

The questions are:

What does it mean?
What validation prevails?
What are the validations weights, if any?
Can the mix policy be use to trump one validation over the other?

Lets look at some of these questions:

Does this mean the CVS authenticated ELVEY.COM  has a spammer user on this
system?

Does this mean the CVS authenticated ELVEY.COM  has a  DNA attribute that
says:

    "[X] Reject if the Sender Path is invalid"

Could it mean that CSV says a validated machine trump any other validation?

Lets go a step further with another mix policy situation:

Lets assume the sysop has implement a CBV (Call back Verifier) instead of
SPF.  The CBV is designed to perform a callback to the return path MX domain
to validate the return address.

Lets assume the CBV returns

        "550 Unknown user: [email-masked]"

What does this say about the trust of an CVS authenticated ELVEY.COM system?

Should ELVEY.COM get blacklisted? or should a report be sent?

So on, and so on.

The point in all this is that NONE of the proposals have taken a very
thorough "Software Walkthrough" and look at all the possibilities.

Delay validation is a requirement in my book.  Not only do you achieve
higher compatibility,  you can save a very significant 35-60% in validation
overhead.

No one proposal can be used unless it covers all bases.  Thus unless someone
comes up with a merger of all the proposals, the advanced SMTP system today
will need a suite and they will have to deal with the mix policy issues at
some level.

I hope this is sufficient Matthew.

Sincerely,

Hector Santos, CTO

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 6 Dec 2004 19:19:33 -0500

Matthew, is a good example of a mix policy with SPF and CBV.   This is just
one of the thousands logged by our system called WCSAP (Wildcat! Sender
Authentication Protocol) which is a suite of many methods configurable by
the admin.

WCSAP is called by our WCSMTP server once the final destination user is
established at RCTP TO:  A number of parameters are passed, but the main
ones are the CIP, CDN,  RP (MAILFROM) and FP (RCPTTO)

18:25:01 version    : 2.01 / 1.61
18:25:01 calltype   : SMTP
18:25:01 state      : rcpt
18:25:01 srvdom     : winserver.com
18:25:01 srvip      : 208.247.131.9
18:25:01 cip        : 68.237.236.230
18:25:01 cdn        : josiah
18:25:01 from       : <[email-masked]>
18:25:01 rcpt       : <[email-masked]>

Data passed and logged

18:25:01 testorder  : FLT RBL SPF CEP CBV

Admin had internal white/black rule based filtering enabled, RBL, SPF,  CEP
(Microsoft Caller ID) and CBV (CallBack Verifier)

18:25:01 sapfilter  : pass (time:63)

It pass the FLT test (time is milliseconds)

18:25:01 saprbl     : testing 230.236.237.68.sbl.spamhaus.org
18:25:01 saprbl     : testing 230.236.237.68.list.dsbl.org
18:25:01 saprbl     : testing 230.236.237.68.bl.spamcop.net
18:25:01 saprbl     : pass (time:15)

It pass the RBL test.

18:25:01 sapspf     : v=spf1 mx ip4:212.80.76.43 ~all
18:25:01 sapspf     : softfail (time:0)

A test for SPF results in a SOFTFAIL.  Admin as setup to continue testing.

18:25:01 sapcep     : test from=seznam.cz
18:25:01 sapcep     : test cdn=josiah
18:25:01 sapcep     : none (time:16)

A test for CEP is done with no result.    Now the final CBV is done with is
a call back to the
return address:

18:25:01 sapcbv     : total mx records: 2
18:25:01 try mx     : mx1.seznam.cz ip: 212.80.76.44
18:25:01 # connecting to 212.80.76.44
18:25:02 S: 220 email.seznam.cz - Email zdarma na cely zivot ESMTP
18:25:02 C: NOOP WCSAP v2.01 Wildcat! Sender Authentication Protocol
http://www.santronics.com
18:25:02 S: 250 ok
18:25:02 C: HELO mail.winserver.com
18:25:02 S: 250 email.seznam.cz - Email zdarma na cely zivot
18:25:02 C: MAIL FROM: <>
18:25:02 S: 250 ok
18:25:02 C: RCPT TO: <[email-masked]>
18:25:02 S: 553 sorry, no mailbox here by that name. (#5.1.1)
18:25:02 C: QUIT
18:25:02 sapcbv     : 553
18:25:02 result     : reject (0)
18:25:02 smtp code  : 550
18:25:02 reason     : Rejected by WCSAP CBV
18:25:02 wcsap finish (1110 msecs)

The address is rejected and the 550 result code is returned to WCSMTP to be
used as part of its response to the RCPT TO command.

The total process took 1.1 secs.

This is a good example of the exploitaiton that has and will continue to
take place of the what I call the "relaxed policies" of SPF1 with its
SOFTFAIL and NEUTRAL result possibilities.

Sincerely,

Hector Santos, CTO

---

From: Matthew Elvey <[email-masked]>
Date: Wed, 08 Dec 2004 12:57:00 -0800

You had claimed:

> The point is in order to accomplish this, the CSV (and others) state

> machine must be based on a delay design mechanism.

This is totally false, and is not supported at all by your 15K post, 
which I have slogged through.

I am familiar with your claim that one MUST wait 'till RCPT TO: to do 
any validation.  It's a false claim.

Certainly doing so might be more efficient. That's not a false claim.

Re. Mixed policies discussion:

1)It's CSV, not CVS. 
2)What part of

> This is out of scope for CSV.  IIRC, you've produced a doc that 
> addresses this scope (or "presumes to", to use your own words) for a 
> certain set of policies; it could could be revised to encompass CSV. 
> CSV does not purport to do the coupling you claim it does not do 
> properly. 

do you not understand?

We are in what's called 'violent agreement'.  We AGREE: CSV must be 
considered in the reality of mixed policies. CSV *ITSELF* does not do 
so, however.
We AGREE: doing the checks that are least costly first (which MAY mean 
delaying validation) is a good idea!

In terms of the specific mixed policy issues you raised:
One only trusts Accredidation and Reputation services that one feels are 
Reputable!  If the USA's DMA runs such a service, I doubt many folks 
will trust it!

>> Should ELVEY.COM get blacklisted? or should a report be sent?
IMO, a blacklist that blacklists based on a delivery attempt *without 
having seen the content of the email, and therefore having no good way 
to know whether it was UBE* should not be trusted; there are legitimate 
reasons for CBV failures, IMO.  So IMO, no.  But again, let me 
re-emphasise, this is not part of the CSV protocol, and does not belong 
in it.  It probably belongs in a BCP around usage.  If a Accredidation 
and Reputation services maintainer feels that there are no legitimate 
reasons for CBV failures, ever, then Yes, the maintainer should ding 
elvey.com in the scenario you describe.  I believe there is consensus 
among abuse desks that email abuse reports that don't include the header 
and body of the email are unlikely to be given much, if any weight.