Subject: [ietf-dkim] DKIM on envelope level

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Thu, 29 Oct 2009 00:19:40 +0100

Hi,

excuse me if this has been discussed before; I was wondering whether 
there has ever been discussion about the usefulness, possibilities, 
caveats etc. of applying DKIM on the SMTP envelope level. I could not 
find an exact reference in the archives of the list; the closest I could 
find is a thread with subject "Signalling DKIM support before DATA" back 
in August 2006. The idea I had in mind is somewhat different from what 
was discussed in that thread: in August 2006 the discussion was about 
signalling during the SMTP dialogue that the header of the 
message-to-be-transmitted will carry a DKIM signature. What I'd like to 
discuss (if this has not been done before) is about using DKIM to 
authenticate the MAIL FROM address/domain.

Idea:

   1. SMTP client connects to SMTP server

   2. SMTP server sends banner

   3. SMTP client starts with EHLO

   4. SMTP server answers with list of EHLO keywords

      New: EHLO keyword DKIM, which consists of the word DKIM followed
      by a unique random sequence generated by SMTP server, e.g.
      250-DKIM 1F64GH996u3YzzXp

   5. If the SMTP client (Edge ADMD MTA) is owner of the MAIL FROM
      domain and detects DKIM SMTP extension support:
      SMTP client uses private DKIM key + MAIL FROM domain (or MAIL FROM
      complete address) + unique random sequence generated by the SMTP
      server and advertised with the DKIM EHLO keyword to generate a
      hash / signature

   6. SMTP client then sends 'MAIL FROM' with envelope From address +
      the above generated hash / signature
      This will require an extension to the MAIL FROM definition.
      Something like:

      MAIL FROM:<user@domainname> DKIM=some_hash_value

   7. SMTP server will use sender's public DKIM key + Envelope From
      information + unique random sequence to verify this hash / signature

   8. If the verification fails: continue as if there had been no DKIM
      extension at all (just like the SMTP dialogue is done now)

   9. If the verification is succesful, the SMTP server can use the
      result for additional actions, along the lines of paragraphs 6.2
      and 6.3 of RFC4871. The advantage here is that we've not yet
      entered the DATA phase.

There are some variants possible like:

a) use MAIL FROM domain or MAIL FROM entire envelope From address
b) make it a per-recipient service, by applying this DKIM extension to 
the RCPT TO, so selectively apply it for some RCPT TO addresses and do 
not apply it to other RCPT TO addresses. Compute the hash / signature 
over MAIL FROM + RCPT TO + Unique random sequence. This might introduce 
some serious complications, not to speak of the extra CPU power required 
to compute all those hashes (if there are many recipients in the SMTP 
session)
c) as an alternative to the the unique random sequence, have the SMTP 
server side publish a DKIM public key in DNS. Use public/private pair to 
sign (SMTP client) and verify (SMTP server)

Is there a use case for b)?
Replay is not possible?
Has c) never been considered before?

I'm pretty sure these things must have been discussed before many times 
and I don't want to go round in circles, so please tell me if this has 
been discusses before.

I see quite clear a number of limitations to the above proposal:

    * the above will (if it works) only work for single-hop SMTP
      connections from one Edge ADMD MTA to another Edge ADMD MTA.
    * hence, the net result of the above is somewhat equivalent to what
      can be achieved by SPF. So is it worth the effort?
      Yet I'm curious to see what the differences with SPF are (there
      are some differences, as SPF is 'validating' IP addresses and DKIM
      in the above scenario is validating MAIL FROM) and whether the
      above has any advantages over SPF or not at all
    * when messages transfer multiple hops and/or are redistributed by a
      mailing list or forwarded (alumni), DKIM on the envelope level as
      described above will not work/help. Are there any figures about
      the percentage of all SMTP connections which are 'direct' versus
      'multi-hop'?
    * the above will require a new SMTP extension. This means it may
      take a long time before a significant part of all Edge MTA's will
      have implemented it, it they will ever implement it.

Please let me know if this is something to explore further, or whether 
it is a complete non-starter. Looking forward to your comments.

/rolf
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Douglas Otis <[email-masked]>
Date: Wed, 28 Oct 2009 17:18:50 -0700

>        have implemented it, it they will ever implement it.

Review the draft:

http://www.ietf.org/id/draft-otis-dkim-tpa-label-03.txt

This draft describes a mechanism that is able to relate various message 
elements, such as Mail-From, SMTP client, DKIM signatures, and Author 
domain with different domains.  It was intended to be used to authorize 
third-party signing domains, but could also affirm Mail From addresses 
as well.

As outbound mail servers become more widely shared among various 
interests, being able to affirm mail by names used in various 
originating elements could offer greater assurance of message 
acceptance.  Provided of course the name is associated with good 
stewardship.  In other words, this could establish an alternative to 
reputations based upon IP addresses.

-Doug

---

From: Steve Atkins <[email-masked]>
Date: Wed, 28 Oct 2009 17:15:00 -0700

>      information + unique random sequence to verify this hash /  

You're asking the SMTP peer to prove interactively that they have access
to a DKIM private key ...

>      extension at all (just like the SMTP dialogue is done now)

... if they can't do so, you accept the entire email ...

>      and 6.3 of RFC4871. The advantage here is that we've not yet

... if they can do so, you accept the entire email.

In either case you accept the entire email, which gives you everything
you need to confirm that the sender has access to the DKIM private
key, just by verifying the DKIM token.

This seems to provide you with no additional information in either
case.

> it is a complete non-starter. Looking forward to your comments.

Probably a non-starter. You'd need to come up with some benefit
for using it before looking into the details. On the surface it doesn't
appear to have any.

Also, anything that doesn't support store and forward is going to
be a hard sell. If you're just doing authentication over one hop
then you could use the existing AUTH keywords that support SASL,
though I doubt there'd be much interest.

Cheers,
   Steve

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Thu, 29 Oct 2009 10:55:58 +0100

Steve Atkins wrote:
> In either case you accept the entire email, 

Not necessarily. Many if not most Edge ADMD MTA's perform all sorts of 
actions after the MAIL FROM phase and before the DATA phase. Think of 
greylisting, call back verification, use of RHSBL, use of local BL and 
WL's, etc. etc. The outcome of a 'dkim=pass' result can be used to make 
other decisions with respect to these actions, or to give another 
'weight' to the outcome of such an action.

> which gives you everything
> be a hard sell.

I'm not sure about that. Look at SPF; in the real world situation it's 
only useful in a one-hop situation and yet half of the Internet is 
publishing SPF information. If (repeat: if) verification of the MAIL 
FROM would have some added value, then it might be worth the effort even 
if only part of all SMTP communications are covered. BTW: are there any 
figures about (on average) the number of multi-hop SMTP sessions versus 
the number of single-hop sessions? Is it 50%? 60%? 70%?

> If you're just doing authentication over one hop

That's an interesting idea!, although I see a lot of problems here...

/rolf

---

From: hector <[email-masked]>
Date: Thu, 29 Oct 2009 08:53:36 -0400

Overall, beside the issues with the technical merits and higher 
complexity and C/S code dependency of your proposal, you are 
attempting to address something that is already addressed.  The 
problem isn't really single hop direct mail transactions.

The problematic issue is multi-hop operations.  Thats the problem. 
There is no acceptable solution on the table to protect against DKIM 
ready multi-hop, forwarders and/or mailing list servers interfering 
with 1st party DKIM signature protection and the proposed solution, 
RFC 5617 POLICY, is intentional ignored because to follow it, breaks 
the multi-hops operations.

In short. SPF has policies like the following that are similar to DKIM 
POLICY:

    HARDFAIL   -->  DKIM=DISCARDABLE
    SOFTFAIL   -->  DKIM=ALL
    NEUTRAL    -->  DKIM=UNKNOWN or No Record

Problem #1

Only DKIM=DISCARDABLE has an explicit handling mandate.  DKIM=ALL does 
not.  So as in SPF=SOFTFAIL, DKIM=ALL leaves receivers in wasteful limbo.

Problem #2

Currently, the list server implementations are ignoring RFC 5617 some 
before of legacy issues, others intentionally, like gmail.com and 
mipassoc.org (this list).  So even if checked at the edge MSA, the 
internal forwarders and remailers are ignoring the possibility of ADSP 
protected domain, breaking the integrity, resigning, removing any 1st 
party signature and forwarding the mail.   This will conflict with the 
final MDA.

So even if your proposal attempts to address this multi-hop issue, 
which it does not, it won't work unless remailers and forwarders also 
support it.

This is the somewhat the same issue with SPF forwarders. However SPF 
does have a optional solution - EHLO/HELO domain SPF checking. 
Something similar is needed for DKIM.

But again, if the integrated software logic doesn't fit, and the 
remailers continue to not support the initial submission checking and 
filter ADSP domains, then we will continue to have the same multi-hop 
issues until another solution is proposed.

Keep in mind even a reputation proposals doesn't work if remailers are 
exempt from checking the initial submissions as well.

In fact Super Idea XYZ will not work if remailers are exempt.

--
HLS

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 29 Oct 2009 09:45:31 -0400

Rolf E. Sonneveld wrote:
> WL's, etc. etc. 

I was just at a session at an industry trade association where the question of 
doing DKIM during SMTP came up.  There were operations folk who very much liked 
the idea of being able to obtain some DKIM benefit during the SMTP session, 
before the dot...

No one suggested modifying SMTP or DKIM specifications.

What /was/ discussed was the possibility of doing a signature that would 
validate before DATA.  This merely requires a signature that does not cover the 
body.

I can't say that anyone sounded hugely enthusiastic about this, but given that 
there was interest in SMTP-time benefit, I think they just needed to think about 
this more.

Having two signatures, with one covering the body and relevant parts of the 
message header, and the other only covering the header, strike me as a plausible 
use of DKIM, worth considering.  I've no idea whether it would provide any or 
enough value-add.  However it is only a stylized use of the existing standard, 
and so the cost of experimenting with it is reasonable.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: hector <[email-masked]>
Date: Thu, 29 Oct 2009 10:32:44 -0400

> the idea of being able to obtain some DKIM benefit during the SMTP session, 

One proposal (via list mail) was suggested via an SMTP extension that 
allows the SMTP server to RAISE the BAR on SMTP clients. For example:

     250-DKIM REQUIRED

But that has a problem with legacy transactions.

Another proposal with a few participants here worked with me called 
"HEAD" described where the header can be pushed first minimizing the 
payload and also helping with 2822/5322 level technologies.

   http://www.isdg.net/public/ietf/drafts/draft-santos-smtphead-00.html

Example:

          S: serverdomain.com, welcome ESMTP v2.0
          C: EHLO mail.clientdomain.com
          S: 250-HEAD
          S: 250-HELP
          C: MAIL FROM:<[email-masked]>
          S: 250 User Ok
          C: RCPT TO:<[email-masked]>
          S: 250 User Ok
          C: HEAD
          S: 354 Start Header input; end with <CRLF>.<CRLF>
             [header block]
             .
          S: 250 Header Accepted. Please Continue with DATA
          C: DATA
          S: 354 Start Body input; end with <CRLF>.<CRLF>
             [message body]
             .
          S: 250 Message Accepted for Delivery
          C: Quit
          S: 250 Goodbye

> validate before DATA.  This merely requires a signature that does not cover the 

HEAD can be updated to support suggest an idea.

The beauty of HEAD is that it would help other payload technologies 
and not just DKIM. But also Sender-id.

> there was interest in SMTP-time benefit, I think they just needed to think about 

I think most people are simply interested in rejecting mail they don't 
have to accept.  But under the current framework that requires a 
transfer of the payload overhead just to get the header+body.    Just 
like David MacQuigg wanted to do with just checking the header for 
DKIM policy:

       http://www.imc.org/ietf-smtp/mail-archive/msg05781.html

This is the direction with many SMTP implementators - accepting mail 
without "thinking" has its consequences. In today worlds, it decreases 
confidence in mail reliability leaving us with "DISCARD" ideas that 
only perpetuate non-notification transactions.

That is why HEAD was put together.

--
HLS

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Thu, 29 Oct 2009 16:48:27 +0100

SM wrote:
> Hi Dave,
> At 06:45 29-10-2009, Dave CROCKER wrote:
> Murray and I discussed the idea of doing DKIM during SMTP last year. 
> There were some flaws in the idea such as how to deal with replay. 

To prevent replay I proposed the use of some unique string/number 
sequence, provided by the _receiving MTA_, which is then used by the 
sending MTA to generate the hash/signature. This restricts the use of 
the hash/signature to only the current SMTP transaction.

> There is also the drawback of getting this deployed as it requires 
> changes to the MTA.

Granted.

> I guess that you are looking at it implementation-wise where we can 
> skip the body() call. This is like having a "l=0".

Dave, can you elaborate on what you have in mind? I'm not sure I 
understand what you mean.

/rolf

---

From: SM <[email-masked]>
Date: Thu, 29 Oct 2009 08:28:57 -0700

Hi Dave,
>the idea of being able to obtain some DKIM benefit during the SMTP session,

Murray and I discussed the idea of doing DKIM during SMTP last 
year.  There were some flaws in the idea such as how to deal with 
replay.  There is also the drawback of getting this deployed as it 
requires changes to the MTA.

Regards,
-sm

P.S. Apologies to the WG Chairs for going off-topic.

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 29 Oct 2009 09:11:12 -0700

First blank line after DATA.

Whether that affords sufficient value-add is an open question to me and probably 
others.

d/

Ian Eiloart wrote:
> 
> 
> --On 29 October 2009 09:45:31 -0400 Dave CROCKER <[email-masked]> wrote:
> 
>>
>>
>> Rolf E. Sonneveld wrote:
>>> Not necessarily. ....
> ....
>> I was just at a session at an industry trade association where the
>> question of  doing DKIM during SMTP came up.  There were operations folk
>> who very much liked  the idea of being able to obtain some DKIM benefit
>> during the SMTP session,  before the dot...
>>
>> validate before DATA.  This merely requires a signature that does not
>> cover the  body.
>>
>> I can't say that anyone sounded hugely enthusiastic about this, but given
>> that  there was interest in SMTP-time benefit, I think they just needed
>> to think about  this more.
> 
>> Having two signatures, with one covering the body and relevant parts of
>> the  message header, and the other only covering the header, strike me as
>> a plausible  use of DKIM, worth considering.  I've no idea whether it
>> would provide any or  enough value-add.  However it is only a stylized
>> use of the existing standard,  and so the cost of experimenting with it
>> is reasonable.
> 
> So, how do you get the headers without the body?
> 
>>
>> d/
> 
> 
> 

-- 

   Dave Crocker

---

From: hector <[email-masked]>
Date: Thu, 29 Oct 2009 12:30:04 -0400

> Whether that affords sufficient value-add is an open question to me and probably 

There is no doubt it has value whether you got it at DATA (up to the 
first blank) or received the entire payload.

The difference is a question of overhead savings which SMTP does not 
allow because a receiver can not DROP at the first blank due to some 
clear DKIM/SMTP verification/policy failure.  To do so creates an 
implicit 451 retransmission problem.

See the discussion in IETF-SMTP about this very issue:

---

From: Douglas Otis <[email-masked]>
Date: Thu, 29 Oct 2009 10:05:19 -0700

On 10/29/09 6:45 AM, Dave CROCKER wrote:
>> Not necessarily. Many if not most Edge ADMD MTA's perform all sorts
>> of actions after the MAIL FROM phase and before the DATA phase.
>> Think of greylisting, call back verification, use of RHSBL, use of
>> local BL and WL's, etc. etc.

If DKIM is to provide acceptance value, so should authorizations of a
DKIM signer.  When a Mail From domain authorizes a different DKIM domain
signing the message, this could serve as a basis for acceptance of the
message.  Conceivably, this could be done prior to acceptance of the
entire message.  After all, authorization can be checked within a single
DNS transaction during validation.

Perhaps knowing the Mail From domain had authorized the signing domain
might grant acceptance prior to validation, but this could lead to an
excessive number of DSNs whenever the authorized signature proves to be
invalid subsequent to acceptance.  The TPA-Label scheme even allows
selective assertion of signing practices that could target a message
signer being spoofed.

> question of doing DKIM during SMTP came up.  There were operations
> folk who very much liked the idea of being able to obtain some DKIM
> benefit during the SMTP session, before the dot...
>
> What /was/ discussed was the possibility of doing a signature that
> would validate before DATA.  This merely requires a signature that
> does not cover the body.

DKIM has split out the body hash from that of the header fields, but
that only permits hashing the message body later.  Not much saved there.

> I can't say that anyone sounded hugely enthusiastic about this, but
> given that there was interest in SMTP-time benefit, I think they just
> needed to think about this more.

In for a penny, in for a pound.  As the prior paragraph suggested, the
current DKIM signature can provide this feature whenever signature
validation is done prior to acceptance and the Mail From domain has
offered authorization.  It seems holding acceptance until DKIM
validation might require hardware assist.  Hardware assist is not
expensive, and could be limited to trusted sources.  Here again, the
TPA-Label approach could play a role in the selection.

-Doug

---

From: Mark Delany <[email-masked]>
Date: Thu, 29 Oct 2009 09:42:06 -0700

On Oct 29, 2009, at 9:11 AM, Dave CROCKER wrote:

I'm guessing the incentive for all of this is to reduce bandwidth,  
otherwise why not just issue the 4XX/5XX after the DATA/. sequence  
rather than invent a new mechanism to issue the same response at the  
envelope end of the transaction?

> First blank line after DATA.

If the proposal is an attempt to reduce SMTP bandwidth, which is  
becoming a vanishingly small part of Internet traffic for most sites  
anyway, then stopping after DATA doesn't help as your OS will have  
likely received a socket buffer full of data, even if the application  
doesn't read it. So it might make you feel good, but it doesn't reduce  
the bytes coming down the line consequentially.

Mark.

---

From: Steve Atkins <[email-masked]>
Date: Thu, 29 Oct 2009 10:18:28 -0700

> Whether that affords sufficient value-add is an open question to me  

DKIM already lets you verify the headers before you've seen the body, if
I recall the algorithm correctly. There's no need for anything to change
there.

Any rejection of a message that's dominated by reputation, rather than
body content, can be performed after you've seen the headers and
before seeing the body.

In this brave new world of multi-megabyte MIME attachments the
resources consumed by the body (bandwidth, temporary storage,
CPU and concurrency) far outweigh the resources consumed by
the headers.

But if you've made a decision by the time you've seen the headers,
what do you do then? There's not any clean, in-band way to respond
to just the headers (leading to all sorts of ugly workarounds like
dropping the connection and then rejecting anything that looks
kinda like the same message before DATA when it's retried).

It would be very, very nice to have an ESMTP extension that
allowed sending just the headers of an email, waiting for a
2xx or 5xx response, then sending the body. And that's the
general solution to this sort of problem, though a little broader
in scope than is really appropriate for the DKIM forum alone.

Cheers,
   Steve

>>> would

---

From: hector <[email-masked]>
Date: Thu, 29 Oct 2009 13:28:31 -0400

> that only permits hashing the message body later.  Not much saved there.

The current framework payoff would be to use POLICY at the data level 
simply because most of its potential benefit DOES NOT require rehashing.

It would be a form of Query Dissemination Filtering Technique - 
removing the obvious as fast as possible.

- DATA analysis after <CRLF>.<CRLF> and before response:

   # ADSP RULE (No Rehash Check)

   if 5322.From Domain ADSP has DISCARD and there
      is no (physical) signature REPLY WITH 551

   # ADSP RULE (Message BH= Rehash Check)

   if 5322.From Domain ADSP has DISCARD and the
      BH= REHASH fails REPLY WITH 551

   # LOCAL POLICY RULE (No Rehash check)

   if  5322.From Domain ADSP has ALL and there
       is no signature and Local Policy
       __XYZ_____ Detected State (SPF),
       then REPLY WITH 551

In other words, to make it scale it is possible to apply these low 
overhead rules at this DATA state. The information is there. BH= would 
be small overhead.

It is possible to do a B= rehash check also:

   # ADSP RULE (Message B= Rehash Check)

   if 5322.From Domain ADSP has DISCARD and the
      B= REHASH fails REPLY WITH 551

But that means you need the entire body and it might not scale for 
many mid to large sites without load balancing, fast hardware and 
highly optimize DNS operations.  Smaller volume sites can afford to 
this though.

---

From: "O'Reirdan, Michael" <[email-masked]>
Date: Thu, 29 Oct 2009 09:55:47 -0400

I was at the same event and I do think this may have some merit.

Mike

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 29 Oct 2009 15:01:46 +0000

....

-- 
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 29 Oct 2009 15:08:24 +0000

> not.  So as in SPF=SOFTFAIL, DKIM=ALL leaves receivers in wasteful limbo.

No, they don't. They both provide useful information. For example, we 
provide very limited whitelisting which requires that senders get an SPF 
pass - to get on our whitelist, senders have to publish SPF records. The 
skip some of our checks when they send mail from the right sender to the 
right recipient through an SPF protected channel, but not otherwise.

DKIM=all gives me useful information, too. If I seem mail from such a 
domain, without a signature, and can't see that it's taken a path that 
might have broken a signature, then I know to treat the mail with 
suspicion. I won't discard it, but I might reject it at SMTP time.

---

From: Michael Thomas <[email-masked]>
Date: Thu, 29 Oct 2009 11:26:34 -0700

> the bytes coming down the line consequentially.

My suspicion is that Mark is exactly right. Besides, DKIM alone is
generally better at determining who's "good" rather than who's "bad".
So you're going to need the whole body for filtering in all normal
cases anyway.

	Mike, bad guys aren't going give you a short
	 header saying they're bad after all. That's
	 the job of RFC3514
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 29 Oct 2009 14:57:34 -0700

> in scope than is really appropriate for the DKIM forum alone.

Seems like the SMTP Clients with mail that would get dropped after the header 
would not be inclined to implement the option.

But perhaps it would still have uptake, presuming bad authors are different from 
bad transmitters...

d/
-- 

   Dave Crocker

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 29 Oct 2009 14:52:25 -0700

> -----Original Message-----
> From: [email-masked] [mailto:ietf-dkim-
> [email-masked]] On Behalf Of Mark Delany
> Sent: Thursday, October 29, 2009 9:42 AM
> To: [email-masked]
> Cc: IETF DKIM WG
> Subject: Re: [ietf-dkim] DKIM on envelope level
> 

Do you mean the server's side of the connection will have buffer data in it?  That would mean the client sent DATA<CR><LF> followed by header/body information, possibly even in the same packet, without waiting for the reply from DATA.  The server could drop the connection outright in that case for violating SMTP rules, even with support for PIPELINING which as I recall requires a synchronization point at DATA.  This wouldn't reduce pipe bandwidth consumption (if that's the goal) but it could reduce resource consumption at MTAs.

It seems to me the idea proposed might tie envelope authentication to a single domain name (either the HELO/EHLO parameter or the domain name in the MAIL FROM parameter, if any.  An ADSP variant could answer the "should be signed" question for that name as well), but then you would still need domain reputation to assert a value or meaning of that name for a filtering decision to be possible.

The other obvious consideration: Spammers are already signing their mail with DKIM, so they could just apply this method as well.  Without widespread deployment of domain reputation, they would still get a "green light" from the DKIM module and maybe even the ADSP one.

As for the data volume, we're still seeing spam campaigns at 200k or more per message.  Compared to general Internet traffic (e.g. streaming video) that's probably kind of small, but it might make a difference to people on the end of smaller links.

---

From: hector <[email-masked]>
Date: Thu, 29 Oct 2009 19:09:52 -0400

> Do you mean the server's side of the connection will have buffer data in it?  That would mean the client sent DATA<CR><LF> followed by header/body information, possibly even in the same packet, wit…

Not always.

If a HALF CLOSE is done

    shutdown(socket,SD_SENT)

there will be some remaining buffer. Half Close is the recommended way 
to shutdown a TCP session.[1]

It could be just QUIT<cr><lf> remaining in the buffer.

You can test this now but creating a DATA shim that receives the 
payload but delays for 3-5 minutes before a response.  The Server can 
wait for a client connection drop and read the remaining buffer.

We use this to resolve the SMTP Specification NO-QUIT-CANCEL found in 
sites that have lengthy DATA filtering scripts.  The client did its 
part of sending the data and is waiting for the server response.  It 
doesn't come, so it issues a QUIT and drops the socket.  This is 
detectable.  The server should see if a QUIT was issued before 
cancelling the transaction

The point is, that the client is not designed to wait in sending 
partial data via SMTP state machine.  The SMTP server can not drop 
because there is an implicit 451 reply code interpretation by the 
client [2].

--
HLS

[1] TCP/IP Illustrated Volume 1, The Protocols. Pages 233, 238.
[2] RFC 5321 Section 3.8.  Terminating Sessions and Connections

---

From: Mark Delany <[email-masked]>
Date: Thu, 29 Oct 2009 17:30:41 -0700

> Do you mean the server's side of the connection will have buffer  
> data in it?  That would mean the client sent DATA<CR><LF> followed  
> by header/body information, possibly even in the same packet,  
> without waiting for the reply from DATA.  The server could drop the  
> connection

I haven't been watching lately, but at some point a popular bot  
technique was to send the whole transaction without looking at the  
responses. After all, what do they care?

It was popular enough to generate the "greet_pause" feature in  
sendmail - as you must know. I don't know whether "greet_pause"-type  
detection is so wide-spread now that spammers have moved away from  
doing it, but I'll bet a lot still don't care and just blast away.

Mark.

---

From: John Levine <[email-masked]>
Date: 30 Oct 2009 05:40:50 -0000

>2xx or 5xx response, then sending the body.

See the CHUNKING extension defined in RFCs 1830 and 3030.

It was defined 14 years ago but I don't know anyone who uses it.

R's,
John

PS: Shouldn't that be spelled CHONGQING now?
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: John Levine <[email-masked]>
Date: 30 Oct 2009 05:16:54 -0000

In article <[email-masked]> you write:
>Whether that affords sufficient value-add is an open question to me and
>probably others.

I must be missing something.

SMTP does not allow you to read the message headers without also reading
the body, unless you're planning to use something like BDAT (RFC 3030)
which sends the message in chunks.

Once the server has responded to DATA, the client starts sending the
message header.  As soon as the DKIM-Signature: header field arrives,
the server can send off a DNS request for the key record.  As soon as
the blank line at the end of the header arrives, the server can compute
the header hash, and as soon as the DNS response arrives, check if the
b= hash signature is valid.

Assuming the b= signature is valid, the server then computes the body
hash as the body arrives, and at the end of data, it can check the
hash it just computed against the bh= body hash.  If the b= and bh=
both validate, congratulations, you have a valid DKIM signature.  If
the b= hash wasn't valid, there's no point in computing the body hash,
although I suspect that in most cases the amount of computing saved
would be insignificant.

Given that you can pipeline the DKIM verification as the message
arrives, and you have to receive the entire message anyway, what do
people want that they can't already do?  If you want the client to
prove who it is before sending the mail, it can use STARTTLS with a
client certificate signed by someone the server knows.

R's,
John

PS: For anyone who's forgotten, if the server disconnects part way
through the data, e.g., after the header, any legitimate client will
treat it as a soft failure and retry later.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: hector <[email-masked]>
Date: Fri, 30 Oct 2009 02:33:07 -0400

> doing it, but I'll bet a lot still don't care and just blast away.

Yup. Bad clients attempting to behave as a RFC 1854 PIPELINING client 
still occurs out there regardless if the SMTP server doesn't even 
advertise Pipelining EHLO response support.  They are blasting away at 
random sites to see what sticks to get their foot the door which is 
basically step one - message acception.  They really don't care if you 
reject/discard before or afterwards.  Furthermore, I venture that most 
mom and pop spammers, doesn't invest too much money in getting SMTP 
clients compliant with SMTP.

In any case, non-PIPELINING servers should count consecutive bad 
command errors and drop the session. Pauses just delays the session 
completion and can affect incoming worker queue designs.

==
HLS

---

From: Ian Eiloart <[email-masked]>
Date: Fri, 30 Oct 2009 10:12:40 +0000

On 29 Oct 2009, at 16:11, Dave CROCKER wrote:

> Whether that affords sufficient value-add is an open question to me  
> and probably others.
>
> d/
>

There's no opportunity to do anything other than drop the connection  
there, is there? Not without modifying the SMTP spec. The only benefit  
is that you don't have to read the body into memory, but bodies are  
limited in size, so I can't think of much advantage.

A DKIM sig that only signed message headers would have a better chance  
of surviving mailing lists redistribution. It'd be available for re- 
use though, wouldn't it?

-- 
Ian Eiloart

---

From: Eliot Lear <[email-masked]>
Date: Fri, 30 Oct 2009 14:49:21 +0100

Dave, Steve, John, all:

Let me add several thoughts along these lines:

NNTP has since day one had the capability to retrieve headers (HEAD), 
and later specific headers (XHDR, et al).  I tend to think that this 
isn't enough bang for the buck to completely rearchitect SMTP around 
such a notion.

Early in the discussion, I thought we were talking about the envelope.  
Validating the envelope seems to me useful, if only because it provides 
a way to reduce the number of bytes sent, and believe it or not, this is 
still a problem in certain parts of the developing world, where 
bandwidth is still expensive.  Right now some solve the problem with 
upstream filtering.  That has its own set of problems that are as much 
political as technical.

John mentioned CHUNKING.  The reason CHUNKING hasn't taken off is that 
it was intended to address the idea that it would be possible to run out 
of resources within DATA, and that this should be signaled.  While 
theoretically possible, I don't know that CHUNKING would have made any 
difference to either end from this perspective.  I don't think it will 
do much in the context of spam either, since it requires a certain level 
of cooperation on the part of the client to CHUNK a header.  Not that 
one couldn't encourage it, but is it worth it?

Eliot

---

From: Doug Otis <[email-masked]>
Date: Fri, 30 Oct 2009 08:31:28 -0700

> upstream filtering.  That has its own set of problems that are as much

Unless most connection are not accepted, a small network can not be 
protected.  Such a system would either depend upon external filtering or 
a combination of IP address reputation, with perhaps the sampling 
questionable connections to leverage IP address good reputation with 
selective inclusion of IP addresses having unknown reputation.

A good defense would likely entail tracking EHLO hostnames looking for 
consistency, simply because bot-nets notoriously provide inconsistent 
information. This approach may block "localhost" hostnames and the like, 
but these typical misconfigurations represent a small percentage of 
legitimate email.

What is left may then be confirmed as desired through other means.  DKIM 
might be such a means, especially when disparate elements can be 
combined to offer unique identifiers that leverage prior trusted 
elements.  This might be an authorization of a mailing list from a known 
 From domain, for example.  The authorization could be established 
through the use of the TPA-Label.  Even EHLO hostnames can be authorized 
by this scheme.

The small network might then be able safely expand their acceptance 
lists and perhaps eventually become fairly independent once the size of 
this list has grown to a few million entries.

-Doug

---

From: John Levine <[email-masked]>
Date: 30 Oct 2009 18:24:09 -0000

>John mentioned CHUNKING.  The reason CHUNKING hasn't taken off is that ...

 ... the problem it was supposed to solve wasn't important.

In case it's not clear, I'm not saying that CHUNKING is a good idea.
But if for whatever reason you want the ability to abandon the
delivery between the headers and the body, CHUNKING lets you do that
without inventing anything new.

Signing the envelope strikes me as one of those things that sounds
nice, but when you try to work out the wa you'd use it in a practical
application, it turns out not to solve any interesting problems.

In particular, if the signature required an extra round trip for an
extra command, the delay would more than wipe out any speed increase.

Also, it is not my impression that the mere data bytes of mail flowing
through networks are a particular problem.  If, for example, you
pipelined DKIM checks so you could tell that you could throw away the
body of an message as it arrived rather than storing it, how much
better would an envelope check be?

R's,
John

---

From: Steve Atkins <[email-masked]>
Date: Fri, 30 Oct 2009 11:45:18 -0700

> better would an envelope check be?

Mailserver concurrency seems to be a significant problem at large  
receivers.
It's not the bytes so much as the ports multiplied by the seconds.  
Receiving the
mail and throwing it away saves you on IO bandwidth and memory, but
it still ties up that valuable delivery slot for the entire duration  
of the
message delivery.

Cheers,
   Steve

---

From: hector <[email-masked]>
Date: Fri, 30 Oct 2009 14:49:14 -0400

Ian Eiloart wrote:

> there, is there? Not without modifying the SMTP spec. 

Right. Its really a non-starter unless an integrated solution is 
endorsed with an SMTP extension perhaps changing the SMTP 451 reply 
code semantics for a dropped connection or at least offers a way for 
the client to interpret a drop differently.

> The only benefit  
> limited in size...

How so? You mean the socket stack buffer?  Under the Windows socket 
stack, the default is 8K.

> of surviving mailing lists redistribution. It'd be available for re- 

At different sites yes.  At the same site, you might have dupe 
checkers like a NONCE concept.

This whole idea of "patching" broken mail integrity during transports 
conflicting against the very nature of what an inherent mail integrity 
protocol provides is so odd to me.

The victims of this are remote receivers outside the "Good" protected 
channel and no protection for the #1 problem - abusive unsolicited 
public anonymous transactions.

Its would be all good if while we are looking for the golden needle in 
a haystack would be also filtering the bad or fake golden needles 
found. I simply don't see how we can cover one side and not the other. 
  This only keeps a status quo - no incentive for legacy bad goods to 
adapt. There is no need to change. Failure is ignored so why would 
anyone bother.

In the past week, we began to collect and analysis incoming DKIM mail.

So far, as expected, a large percentage is spammers and virus messages 
signed are leverage DKIM.  I saw one interesting group of signers:

   d=siouxfallsblog.net;
   d=simivalleymicroblog.com;
   d=shreveportblog.net;
   d=shreveportmicroblog.com;
   d=savannahmicroblog.com;
   d=santaclaritamicroblog.com;

all with slightly changed content blasting a "click me" virus.

No keys, domains don't exist anymore.

That is why I call it a waste if this calculation overheads offer no 
payoff.

--

---

From: Eliot Lear <[email-masked]>
Date: Fri, 30 Oct 2009 19:52:54 +0100

> application, it turns out not to solve any interesting problems.

I can't say, but I do know that many of us toss a whole lot of mail at 
EHLO, some at MAIL FROM:<> and some at DATA.  The idea I was thinking 
about was whether it provides any value whatsoever to at least know that 
you are authentically dealing with a legitimate source sooner, without 
> extra command, the delay would more than wipe out any speed increase.

I would probably modify the MAIL FROM: handling to do this such that no 
new roundtrips were needed.  That having been said, a DNS query would be 
needed, even if it's just to a caching resolver.
> better would an envelope check be?
You and I are looking at two different problems.  I don't think that 
storage of the message is the issue, although there is a cost.  As I 
mentioned earlier, there are some sites that still pay a whole lot for 
bandwidth.

Eliot

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 30 Oct 2009 11:43:13 -0700

> From: Mark Delany [mailto:[email-masked]]
> Sent: Thursday, October 29, 2009 5:31 PM
> To: Murray S. Kucherawy
> Cc: [email-masked]; IETF DKIM WG

I agree that they're probably doing that, but you can save yourself a lot of server-side stuff like temporary storage of a (possibly large) message body, creation of a thread, etc. if you observe at DATA that there's more information pending before you've sent back the 354.  Sure, the bits have already come down the wire and into a kernel socket buffer, but the client can abort early (freeing resources, maybe even shutting down the session) which conserves some user-level server resources.

The same might be true if at DATA the client has failed some kind of envelope signature test like what's been proposed.

---

From: hector <[email-masked]>
Date: Fri, 30 Oct 2009 15:45:28 -0400

> EHLO, some at MAIL FROM:<> and some at DATA. 

These are normally site pattern specific, but for our site, we see

   EHLO/HELO:  10% is rejected or the client drops
   MAIL FROM:  Delayed Verification until RCPT TO is known
   RCPT TO:    51% are rejected with bad RCPT TO addresses.
               Locally Unknown or unauthorized to relay.

Of the remaining that get pass RCPT TO, MAIL FROM is then checked with 
a breakdown:

     1.5%  Local Rules
    29.8%  DNS RBL rejections
     6.6%  SPF rejections
    62.1%  CBV (Call Back Verification) Return Path rejections

The main reason for the MAIL FROM delayed verification was because it 
offered a huge +50% DNS lookup overhead reduction.  FYI, this 
technique is discussed in RFC 5321 Section 3.3:

    ......................................Despite the apparent
    scope of this requirement, there are circumstances in which
    the acceptability of the reverse-path may not be determined
    until one or more forward-paths (in RCPT commands) can be
    examined.  In those cases, the server MAY reasonably accept
    the reverse-path (with a 250 reply) and then report problems
    after the forward-paths are received and examined.
    Normally, failures produce 550 or 553 replies.

> You (levine) and I are looking at two different problems.  I don't 

 > think that storage of the message is the issue, although there is
 > a cost.  As I mentioned earlier, there are some sites that still
 > pay a whole lot for bandwidth.

+1, in the past,scaling out (more machines) was the solution. Today, 
scaling up is more feasible and cost effective due to faster 
multi-core machines, more memory, VM, better integration with the 
backend, to share information among threads, etc.

We find a few issues due to SMTP current design:

   1) SMTP implicit 451 retries for server drops.
   2) More 5322 technologies requiring a payload reception
      for processing.
   3) No "Keep Alive" concepts for lenghty DATA processing.

Because of this SMTP is encourage to accept all transactions for 
delayed processing.  See  RFC 1047 "Duplicate messages and SMTP"

IMO, this is old and based on lower bandwidths where SMTP level delays 
and/or drops were more costly.   Today's hardware makes it more 
feasible to speed up the SMTP transaction.

Overall, since the SERVER is encouraged to not drop connections and to 
allow clients to gracefully end the session with a QUIT, load 
balancing vs connection balancing becomes more of a challenge.

--

Hector Santos

---

From: "John R. Levine" <[email-masked]>
Date: 30 Oct 2009 16:05:25 -0400

> I can't say, but I do know that many of us toss a whole lot of mail at EHLO, 
> some at MAIL FROM:<> and some at DATA.  The idea I was thinking about was 
> whether it provides any value whatsoever to at least know that you are 
> authentically dealing with a legitimate source sooner, without having to send 
> even a whole header.

Well, there's STARTTLS.  Again, it's there, I believe there's pretty broad 
support, but it doesn't seem popular outside of some specialized financial 
environments.

Also remember CSV which Dave and I concocted to allow sites to identify 
the hosts that are supposed to be mail clients.

R's,
John

---

From: Doug Otis <[email-masked]>
Date: Fri, 30 Oct 2009 14:25:08 -0700

> the hosts that are supposed to be mail clients.

John, it was Dave Crocker, John Leslie, and myself, where I had 
independently written a draft similar to Dave's.

It is really a shame too, as this approach would have helped establish a 
name basis for a reputation scheme that could have been applied early in 
the SMTP transaction.  A reputation scheme based upon any authorization 
that leave providers nameless would be wrong and inherently unfair.

The immediate forward reference to identify SMTP hostnames would have 
been effective in mitigating much of the bot-net generated traffic 
without any change to SMTP. Of course, retention of the SMTP client IP 
address would help as well.  Perhaps CSV might be how the IP 
address/hostname gets added to the Authentication-Results header. :^)

Dealing with IPv6 will likely require reputations be based upon a domain 
name rather than upon individual IP addresses.  The remaining question 
would be affirming SMTP hostnames.  Reverse DNS invites long timeouts, 
making this effort fairly expensive from a resource standpoint.

DKIM might help in identifying which SMTP clients are legitimate. 
Allowing DKIM to "bless" other originating domains found within the 
email transaction (via TPA-Labels) could help in spreading the trust 
established.  Perhaps the TPA-Label for a hostname could include a CVS 
requirement flag or a couple of CIDR entries.

Maybe providers will stop avoiding naming their role the in issuing 
email.  I strongly believe autonomous authentication schemes for DKIM 
can greatly increase the value of the service.  When combined with CSV, 
replay issues could be handled in manner compatible with third-party 
re-mailers, such as mailing-lists.

Looking for the low-cost web of trust...

-Doug

---

From: Dave CROCKER <[email-masked]>
Date: Fri, 30 Oct 2009 14:34:18 -0700

John R. Levine wrote:

> Also remember CSV

    <http://mipassoc.org/csv>

d/
-- 

   Dave Crocker

---

From: hector <[email-masked]>
Date: Fri, 30 Oct 2009 18:04:15 -0400

> that leave providers nameless would be wrong and inherently unfair.

Doug, for the same reasons reputation schemes are hard to shallow for 
DKIM as a general, useful, consistent scheme protocol, it was also 
hard to shallow for CSV. It was a Batteries Required concept, and once 
again, doesn't address failure which was what MARID was looking for - 
the result SPF/SENDER-ID.

If you wish to consider policy and enforcement, I might even begin to 
cheer for CSV.  However, its really a day late and a dollar short - 
the SPF standard is widely adopted and has an optional solution at 
EHLO/HELO checking.  CSV would be redundant.

> name rather than upon individual IP addresses.  

Why do we keep talking about a solution with a undefined reputation 
component?  We talk about it so much, maybe it should be written into 
the charter so we can concentrate on developing a open standard 
reputation protocol, then maybe some of the reputation/DKIM ideas can 
begin to make sense.

> Looking for the low-cost web of trust...

Original Domain DKIM POLICY!  Middle ware should keep their fingers 
off. Mailing list should HONOR it.  If the abuse is so low, then it 
wouldn't hurt if forwarders honored policy.  But it would at least 
close the loophole for the presumed "low volume" domain abuse.

--
HLS

---

From: Doug Otis <[email-masked]>
Date: Fri, 30 Oct 2009 17:41:10 -0700

> Doug Otis wrote:

Hector,

> hard to shallow for CSV. It was a Batteries Required concept, and
> once again, doesn't address failure which was what MARID was looking
> for - the result SPF/SENDER-ID.

IP address path registration scales poorly and invites trouble.
Reputations of those offering SPF authorization is a poor tool for
improving the behavior of providers at fault.  In addition, Dan Kaminsky
demonstrated that arbitrary queries enable poisoning of other important
records. By not including message bodies in DSNs and by checking valid
recipients prior to acceptance, this has largely solved the "failure" of
which I think you're referring. CBV represents another ill conceived
effort further burdening SMTP. :^(

SPF/Sender-ID expects MUAs and MTAs to make from 1 to 111 DNS
transactions for each name checked!  Queries whose targets might be
unrelated to anything within email and yet modulated by various email
address local-parts generated from cached records.  As such, SPF remains
a risky tool.  Imagine the DDoS attack enabled by SPF over DNSSEC.

> EHLO/HELO checking. CSV would be redundant.

The SPF protocol suggests these records _might_ confirm the IP address
of the SMTP client.  There is no specific "policy" requirement being
imposed.  In the end, the sending MTA is likely never affirmed by name.
Only that perhaps some feckless domain authorized an MTA by an IP address.

>> Dealing with IPv6 will likely require reputations be based upon a
>> domain name rather than upon individual IP addresses.
>
> component?

IMHO, soon only positive reputations are likely to prevail, but these
should reflect the actions of those in control.  When done by name, the
numbers are manageable and less likely to require reputation assessment
services.  A list a few million will likely be enough to seed a process,
where relational trust can fill in the gaps.

> We talk about it so much, maybe it should be written into the charter
> so we can concentrate on developing a open standard reputation
> protocol, then maybe some of the reputation/DKIM ideas can begin to
> make sense.

If there is a need, many companies are more than willing to provide the
service.  The hope would be to document a process that avoids the need
for this type of service.

> off.

Any transition will likely require exceptions.  Publishing which
exceptions are needed should be made easy for the sending domain, and be
little effort for receivers to apply.  If a TPA-Label authorization of
the Mail From avoids CBV, that would be a win for both. If the TPA-Label
authorization ensures the issue of an important DSN, again a win for
both, at the expense of a single transaction.

> Mailing list should HONOR it. If the abuse is so low, then it

Any effort at imposing ridge rules will lower the reliability of email,
while also increasing support costs.  As such, absolute policy
assertions are likely to prove useful in only a few cases. Nevertheless,
the TPA-Label scheme offers a significant level of flexibility that does
not require any coordination between the involved parties.  After all,
such coordination will never scale.

-Doug

---

From: hector <[email-masked]>
Date: Sat, 31 Oct 2009 08:41:45 -0400

> such coordination will never scale.

I generally work on the basis that steady state operations do scale 
and operate efficiently and smoothly under a common protocol 
expectation with parts that fit under a standard tolerance levels.

However, when we have relax provisions, pars that don't fit, it 
promote complexity, ambiguity, overhead, additional patch work, relief 
values, more wrapper layers and overall higher cost to manage, this 
continues to place more pressure on the system with all kinds of 
potential leaks in the network.

Whether its DKIM itself, ADSP, undefined reputation schemes, TPA, 
DSAP, this subject topic SMTP level DKIM header signature proposal, 
there is something fundamentally wrong with the work if we have a set 
of documents that specifically targets mitigating exploits, yet we 
have other docs that basically offers relax provisions that 
essentially conflict with these protections.  I find it very hard to 
come up with a working solution or compromise.  Clear and consistent 
functional specifications are required first.

For this topic, to me, the fundamental question is whether a SMTP 
transaction can be rejected with a new SMTP level DKIM-related 
algorithm before DATA is reached or partially read.

Any current plug and play 2822/5322 technology trials at SMTP requires 
the entire payload to be read first.  So there is no payoff on 
minimizing data reception.  The only payoff is SMTP level rejection.

However, under DKIM, with remailers being exempt to not pay too much 
attention to original domain intentions, this SMTP level rejection has 
its risk.

So IMV these SMTP/DKIM handling engineering issues need to be settled 
before any wrapper technology can even get a chance at Proof of Concept.

--
HLS

---

From: hector <[email-masked]>
Date: Sat, 31 Oct 2009 09:44:50 -0400

> Mailserver concurrency seems to be a significant problem at large  

+1.   But its an issue with all SMTP software developer/vendor to be 
able to handle it for any receiver.  Bad guys are not prejudice.

Scaling up as opposed to scaling out or both with a lesser of scaling 
out (more threads per machine) is on the raise. Efficiency, 
performance, loading an connection balancing is important.

 > It's not the bytes so much as the ports multiplied
 > by the seconds.  Receiving the mail and throwing it away saves
 > you on IO bandwidth and memory, but it still ties up that
 > valuable delivery slot for the entire duration of the message
 > delivery.

Right. I find the issue is mostly how fast you can rid of the 
lingering clients waiting to be processed.

Depending the socket stack SOMAXCONN value or socket provider limit, 
coupled with connect/load balancing in the SMTP receiver, it is not 
uncommon to see regular spurts of concurrent blasting where the 
non-compliant SMTP client is just sitting there waiting for a "220 ". 
A good way to see this is add a multi-line system policy display at 
the connection server welcome response. The client is not expecting 
the first line to contain "220-" like AOL.COM will show.

I explored ways to accelerate releasing the waiting clients, but SMTP 
prohibits this because a DROP will cause the clients to retry. So the 
only thing you can do is use the connection limit as a way to issue a 
welcome response of

     421 domain.com, Service busy. Try again later

But  as you implied, the smtp "residence time" is still there waiting 
for a client to drop or time out taking up a connection slots.

---

From: "John R. Levine" <[email-masked]>
Date: 31 Oct 2009 13:01:39 -0400

> CHUNKING can in theory be used to send the headers and body separately. But
> it doesn't have to be - the chunk boundaries are aribtary and there's no way
> for a server to say "send me the headers and body in separate chunks please".
>
> Of course it would be possible to devise an extension to do this - it would
> actually be a pretty simple thing to define - but I'm quite skeptical as to
> it's potential for deployment. Remember, for this to work the clients have to
> change, but the benefits accrus to the server. So why would I as a client
> implementor do something that complicates my code and error handling while
> providing me essentially no benefit?

I agree.  Senders would only do this if recipients demanded it, and 
recipients aren't going to demand it.  So we can forget it.

Your note about milter is a good one.  If performance were an issue, MTA 
vendors could rewrite their own code to stream DKIM verification and 
potentially dispose of unwanted messages more quickly.  If it's not worth 
doing that for the mail stream they have now, it's surely not worth trying 
to change SMTP.

R's,
John

---

From: [email-masked]
Date: Fri, 30 Oct 2009 14:21:51 -0700 (PDT)

> It was defined 14 years ago but I don't know anyone who uses it.

Well, FWIW, we implement CHUNKING and have done so for quite a while. A quick
check of my home logs shows that out of the 5071 messages received today,
48 were transferred with BDAT. So about 1% - not a lot, but some.

I'm afraid I can't tell you how many chunks were used in those CHUNKING
transaction - it's not something we bother to log - but my guess would be one
per. Of course our implementation supports reception of multiple chunks, but we
always send the entire message with a single BDAT. Aain, when you know the
length, why bother with multiple commands?

An even more interesting question would be to what extent do server
implementations that support chunking have the ability to report an error based
on early analysis of partial message data. We do perform such analysis once the
header is available, but this is isn't currently tied back to  BDAT so the
result won't show up until the BDAT n LAST comes in. We could change this
fairly easily, but absent not only more use of BDAT in general and multiple
BDATs in particular, what would be the point of making such a change?

Another issue that would have to be addressed before this would work for DKIM
stuff would be how DKIM-MILTER operates. Given that sendmail apparently doesn't
send the message to the milter as it comes in, I wouldn't expect milter-based
implementations to care about early header processing, initiating DNS lookups,
and so on.

				Ned

---

From: hector <[email-masked]>
Date: Sat, 31 Oct 2009 15:25:49 -0400

> implementations to care about early header processing, initiating DNS lookups,

Excellent point Ned.

The same concept applies with most SMTP software with callouts, shims 
concepts at each state completion and before the client is allowed to 
move to the next state.  In addition, the connection handle may or may 
not passable to the sandboxed shim to protect against channel output 
(state machine) corruption.

But even if the milter or shim callout functionally is split by the 
receiver, we still need to address the SMTP explicit 451 drop/retry 
semantics.  Backward compatibility will still be an issue.

I do have an 2006 draft that attempts to deal with the HEAD/BODY issue.

http://www.isdg.net/public/ietf/drafts/draft-santos-smtphead-00.html
http://www.isdg.net/public/ietf/drafts/draft-santos-smtphead-00.txt

If there is interest .........

--
HLS

---

From: Doug Otis <[email-masked]>
Date: Sat, 31 Oct 2009 13:36:26 -0700

On 10/31/09 5:41 AM, hector wrote:
> I generally work on the basis that steady state operations do scale and
> operate efficiently and smoothly under a common protocol expectation
> with parts that fit under a standard tolerance levels.
>
> However, when we have relax provisions, pars that don't fit, it promote
> complexity, ambiguity, overhead, additional patch work, relief values,
> more wrapper layers and overall higher cost to manage, this continues to
> place more pressure on the system with all kinds of potential leaks in
> the network.

When "steady state" includes dropping suspicious messages, or the 
performing of challenge response checks, then there is an area that a 
reduction in process and/or ambiguity would help.

> Whether its DKIM itself, ADSP, undefined reputation schemes, TPA, DSAP,
> this subject topic SMTP level DKIM header signature proposal, there is
> something fundamentally wrong with the work if we have a set of
> documents that specifically targets mitigating exploits, yet we have
> other docs that basically offers relax provisions that essentially
> conflict with these protections. I find it very hard to come up with a
> working solution or compromise. Clear and consistent functional
> specifications are required first.

Since no change ever occurs at once, there will remain areas where 
handling is less certain.  However, the sender is still able to offer 
information that has been made more trustworthy as a result of DKIM. 
This information might affirm relationships between the signing domains 
and a different domains contained within Mail From or EHLO, for example. 
  Perhaps this information might lessen the likelihood of the sender 
then receiving a challenge response check, or of having their message 
dropped without issuance of a DSN.

> transaction can be rejected with a new SMTP level DKIM-related algorithm
> before DATA is reached or partially read.

You are on the right track.  IMHO, name based acceptance will soon be 
how SMTP EHLO acceptance is achieved, where relational information 
offered from trusted DKIM domains can assist in amending this 
information.  There can never be any definite rules in how this is done, 
since all rules will be gamed.  When decisions change from "unknown" or 
"bad-actor", to "unknown" or "good-actor", the process is less easily 
gamed, and tighter thresholds in determining behavior permit maintenance 
of "acceptance list" with minimal exposure.

There would be two aspects of this list. Domain to domain relationships 
that might entail a few hundred million out of which good actor domains 
need to be identified.  And then good actor SMTP hostnames with their 
related IP addresses which will entail a few million.  Hints given by 
SPF are often overly broad and are easily gamed. IMHO, this information 
should be ignored.  To prevent gaming, receivers should track "actual" 
use.  The amount of information involved with "good actors" is actually 
fairly small, so this is not a difficult problem.  Once a receiver has 
been "seeded" with reasonably complete information, amending the 
information on a trial basis with information of affirmed relationships 
should prove practical.

> the entire payload to be read first. So there is no payoff on minimizing
> data reception. The only payoff is SMTP level rejection.

There are other payoffs as well. Avoiding challenge response, or dropped 
messages without DSN would represent other payoffs.

> attention to original domain intentions, this SMTP level rejection has

When there are hundreds of thousands of domains indicating that 
"mailing-list-r-us" should be trusted, then how they go about handling 
their lists with respect to DKIM is less of a concern.  Of course, it 
would help greatly for them to add their own DKIM signature.

> before any wrapper technology can even get a chance at Proof of Concept.

Dave insists that transfer by reference is off the table.  IMHO, SMTP 
and DKIM unchanged can be made to work as described.

-Doug

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 31 Oct 2009 16:57:00 -0700

> for a server to say "send me the headers and body in separate chunks please".

If the server announces chunking support, there would be no penalty if smtp 
clients simply chose to send in this tailored fashion.  So it could be a 
convention rather than a per-session request.

All of which presumes a meaningful constituency in the email receiver world that 
says it wants this.

> change, but the benefits accrus to the server. 

If senders would like to share in the benefit of consuming less bandwidth, and 
possibly having more immediate feedback on their traffic, they might count as 
having a positive incentive.

d/
-- 

   Dave Crocker