Subject: Re: [yam] Interop problem: SMTP submission, STARTTLS, AUTH EXTERNAL

From: Tony Finch <[email-masked]>
Date: Sat, 1 May 2010 21:23:36 +0100

On Fri, 30 Apr 2010, Chris Newman wrote:
>
> I've been dealing with client certificate authentication lately and realized
> we have an interoperability problem with our standards.  For SMTP submission,
> when an end-user wishes to authenticate using a client certificate, the
> protocol sequence to do so is unclear.

There is no specification for how SMTP+TLS interacts with any
authentication or authorization logic. There is no specification
for how SASL EXTERNAL interacts with security layers. Since the
problem is gaps (rather than errors) in the specifications I think
the right solution is:

Option 4: Write a spec for how SMTP + TLS + SASL EXTERNAL should work
together.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 02 May 2010 14:55:10 +0200

Chris Newman wrote:
> 
> Option 1: relevant to YAM -- Clarify 4409 section 4.3 to state that 
> providing client authentication during TLS does not constitute 
> "independently established authentication" [...]
> 
> Pros: [...] Consistent with IMAP+STARTTLS and POP+STARTTLS.

Even if SUBMIT mandates authentication, it inherits the SMTP trust 
model, which is different from that of IMAP and POP. Section 4.3 
just allows such compatibility. An historical trait.

> Option 2: Incompatible change to RFC 3207 (SMTP STARTTLS) when an SMTP 
> client provides a client certificate the server deems valid for 
> authentication purposes, the server MUST enable the SASL EXTERNAL 
> mechanism (advertising it in EHLO and allowing it in AUTH).  If the 
> client issues "MAIL FROM" without issuing an AUTH command in this 
> situation, the server MUST behave as if an implicit "AUTH EXTERNAL =" 
> was issued by the client.

This option may cause difficulties with the paragraph in rfc 4954 
that says

  Restrictions:
      After an AUTH command has been successfully completed, no more
      AUTH commands may be issued in the same session.  After a
      successful AUTH command completes, a server MUST reject any
      further AUTH commands with a 503 reply.

In case an implicit "AUTH EXTERNAL =" had been assumed by the server 
when a previous transaction started, the client cannot repeat it for 
the whole session. For an MTA, non-authenticated transactions are 
allowed, so it may be unclear whether the server should assume it in 
any case.

I note that SMTP-AUTH does not allow a client to query its current 
authentication state, nor to learn what authorizations such state 
implies (e.g. relaying, specifying trusted AUTH parameters.) I think 
the problem you describe results from that design.

I'd agree if rfc4409bis will informatively mention that "AUTH 
EXTERNAL =" may be used by a client to check the status of an 
independently established authentication --in case the server has 
such mechanism. Additionally, section 3.3 may want to mention client 
TLS certificates. Would that bring any good?

---

From: Tony Hansen <[email-masked]>
Date: Thu, 06 May 2010 08:41:16 -0400

I think this is totally an issue with AUTH EXTERNAL and worth mention in 
an update to that spec. However, I don't think it directly affects 4409bis.

Any other people's thoughts?

     Tony Hansen
     [email-masked]

On 5/2/2010 8:55 AM, Alessandro Vesely wrote:
> model, which is different from that of IMAP and POP. Section 4.3 just 
> allows such compatibility. An historical trait.
>
>> Option 2: Incompatible change to RFC 3207 (SMTP STARTTLS) when an 
>> SMTP client provides a client certificate the server deems valid for 
> This option may cause difficulties with the paragraph in rfc 4954 that 
> says
>
>  Restrictions:
> I'd agree if rfc4409bis will informatively mention that "AUTH EXTERNAL 
> =" may be used by a client to check the status of an independently 
> established authentication --in case the server has such mechanism. 
> Additionally, section 3.3 may want to mention client TLS certificates. 
> Would that bring any good?

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 6 May 2010 14:44:53 +0200

> an update to that spec. However, I don't think it directly affects 4409bis.

+1

Arnt

---

From: "John R. Levine" <[email-masked]>
Date: 6 May 2010 10:44:37 -0400

> I think this is totally an issue with AUTH EXTERNAL and worth mention in an 
> update to that spec. However, I don't think it directly affects 4409bis.
>

Agreed.  Honestly, I don't understand what problem AUTH EXTERNAL is 
supposed to solve.  In the software I'm familiar with, any auth-age that 
AUTH EXTERNAL might do happens automatically, whether the client asks for 
it or not.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Thu, 06 May 2010 10:52:34 -0400

--On Thursday, May 06, 2010 08:41 -0400 Tony Hansen
<[email-masked]> wrote:

> I think this is totally an issue with AUTH EXTERNAL and worth
> mention in an update to that spec. However, I don't think it
> directly affects 4409bis.
> 

Agreed.  I think it would be a serious mistake to clutter
4409bis up with some of the details that have been suggested.
Even at full standard, authentication details are best
incorporated by reference to other documents than can be updated
when or as necessary.

    john

---

From: Chris Newman <[email-masked]>
Date: Thu, 06 May 2010 08:45:41 -0700

>> Any other people's thoughts?

I respectfully disagree that this is an AUTH-only problem.  It is a problem 
that impacts implementations of SMTP+Submission+STARTTLS (without AUTH), 
smtps (without AUTH), SMTP+Submission+AUTH+STARTTLS and smtps+AUTH. 
Unfortunately the only protocol specification we have that applies to all 
four problem scenarios is RFC 4409 (its application to smtps is by virtue 
of the fact smtps is only used for submission because it can't be an MX 
target).

> AUTH EXTERNAL might do happens automatically, whether the client asks for

AUTH EXTERNAL solves two problems.

First, for protocols with an explicit state model (e.g. POP3 and IMAP4), it 
causes a state change from the not-authenticated state where the protocol 
starts to the authenticated state.  The state model is actually very 
helpful when writing security-sensitive software as the threat-surface of 
the protocol can be significantly reduced when in the not-authenticated 
state.  Having as few protocol commands as possible actually trigger a 
state change makes it simpler to verify correct implementation.  SMTP 
doesn't get this benefit as long as we support legacy submission, but 
Submission port 587 can have this benefit.

Second, AUTH EXTERNAL (or more specifically the SASL authorization ID 
feature present in mulitple SASL mechanisms) allows a feature I like to 
call "administrative proxy authentication" where an administrator or more 
likely an automated tool will use administrative credentials to 
authenticate as a specific user.  For example, in a unified messaging 
system the voice mail system may wish to clear the "seen" flag for the 
voice mail in a user's INBOX after it has played that information to the 
user over a telephone interface.  Arguably, the AUTH= parameter in the SMTP 
AUTH spec provides an alternative and perhaps better mechanism to do this 
for SMTP, although I've seen implementations that (incorrectly) claimed to 
implement SMTP AUTH and failed to implement the AUTH= parameter.

So I will accept an argument that AUTH EXTERNAL provides little value 
specifically to SMTP (beyond and explicit protocol indication that the 
client certificate was acceptable for SMTP-level authentication to the 
server, something that helps the client to produce better end-user 
diagnostics), which is why I proposed option 2.  But I do consider AUTH 
EXTERNAL an important mechanism in general.

		- Chris

---

From: "John R. Levine" <[email-masked]>
Date: 6 May 2010 12:36:22 -0400

> Having as few protocol commands as possible actually trigger a state change 
> makes it simpler to verify correct implementation.

Well, gee, in your model one command triggers a state change.  In my 
model, zero commands trigger a state change -- if you're externally 
authorized, you're authorized and you go ahead and do what you're going to 
do.

> SMTP doesn't get this benefit as long as we support legacy submission, 
> but Submission port 587 can have this benefit.

Huh?  Only if you want to break every ISP's SUBMIT server which implicitly 
auth's anyone connecting from an IP inside the ISP's network, or that uses 
POP-before-SUBMIT.

To look at it another way, maybe I'm missing something, but it is my 
impression that if AUTH EXTERNAL fails in whatever protocol, you're going 
to report back to the user that he's not authorized and give up.  How is 
that better than the de facto alternative, where you get the same response 
and the same action when you try to do something?  You're not saving any 
code, the commands to do something are going to have to check that they're 
authorized either way.

R's,
John

---

From: SM <[email-masked]>
Date: Thu, 06 May 2010 12:34:10 -0700

Hi Chris,
At 04:28 PM 4/30/2010, Chris Newman wrote:
>I've been dealing with client certificate authentication lately and 
>realized we have an interoperability problem with our 
>standards.  For SMTP submission, when an end-user wishes to 
>authenticate using a client certificate, the protocol sequence to do 
>so is unclear.
>
>This sequence works assuming the server enables SASL EXTERNAL and 
>the client implements it (not true of all deployed software):
>
>====
>C: EHLO ...
>S: ...
>C: STARTTLS
>S: 220 2.5.0 Go ahead with TLS negotiation.
><negotiate TLS with client certificate>
>C: EHLO ...
>S: ...
>S: 250-AUTH EXTERNAL ...
>S: ...
>C: AUTH EXTERNAL =
>S: 235 2.7.0 EXTERNAL authentication successful.
>C: MAIL FROM:<[email-masked]>
>C: RCPT TO:<[email-masked]>
><relay is allowed>
>====
>
>This sequence of commands may or may not work but RFC 4409 section 
>4.3 can be interpreted to permit it assuming a client certificate 
>via TLS counts as "independently established authentication":
>
>====
>C: EHLO ...
>S: ...
>C: STARTTLS
>S: 220 2.5.0 Go ahead with TLS negotiation.
><negotiate TLS with client certificate>
>C: EHLO ...
>S: ...
>C: MAIL FROM:<[email-masked]>
>C: RCPT TO:<[email-masked]>
><relay may or may not be allowed>
>====
>
>An SMTP submission client that implements only the latter will not 
>interoperate with an SMTP submission server that implements only the 
>former (and possibly vice versa if the client is unwilling to "just 
>try it" when AUTH EXTERNAL isn't there).  One of these two cases 
>must be interpreted or declared not valid by the standards to 
>resolve the interoperability problem.

Section 3.3 of RFC 4409 discusses about authorized submissions.  The 
user may be authenticated using several methods.

>"independently established authentication" because there is no 
>indication in the TLS layer whether that authentication was deemed 
>acceptable for SMTP submission authentication.  Perhaps note as an 
>informational reference that "AUTH EXTERNAL" can be used to 
>determine the validity for that purpose.

SMTP AUTH is a MUST according to Section 4.3 but other types of 
authentication are allowed.  Section 4.1 of RFC 3207 mentions 
authentication.  In the above example there has been some level of 
authentication.  Option 1 gets into what constitutes authentication 
and it's not a good idea to get into a discussion about it in RFC 4409.

>situation, the server MUST behave as if an implicit "AUTH EXTERNAL 
>=" was issued by the client.
>
>Pros: Doesn't break any clients.  Cons: Some previously complaint 
>SMTP STARTTLS server implementations are declared incompliant 
>(including the shipping product from my company ;-).

The better place to add that is in a RFC about AUTH EXTERNAL.  Don't 
break existing implementations unless you really have to. :-)

>Option 3: Incompatible change to RFC 3207 (SMTP STARTTLS) to declare 
>that client authentication provided via STARTTLS is not considered 
>valid authentication at the SMTP layer until the client issues an 
>"AUTH EXTERNAL" command.
>
>Pros: Consistent with IMAP+STARTTLS and POP+STARTTLS.  Cons: Some 
>previously compliant SMTP STARTTLS client and server implementations 
>are declared incompliant.

Section 4.3 of RFC 3207 mentions that:

   "In fact, since the submission port is by definition not a publicly
    referenced SMTP server, the STARTTLS extension can be particularly
    useful by providing security and authentication for this service."

To summarize, the question is where to solve the problem.  I don't 
think that should be done in RFC 4409.

Regards,
-sm

---

From: Ned Freed <[email-masked]>
Date: Thu, 06 May 2010 13:20:36 -0700 (PDT)

> of the fact smtps is only used for submission because it can't be an MX

While it would be a lot more convenient if this weren't the case, I have
to agree with Chris here.

> Submission port 587 can have this benefit.

Agreed that this is a real and tangible benefit.

> implement SMTP AUTH and failed to implement the AUTH= parameter.

The main problem with AUTH= seems to be that people don't have a clue how or
when to use it. I'd like to be able to blame this on the text in RFC 4954, but
when I read it I find very little to criticize. Perhaps an example in the
document showing how to use AUTH= for proxy authentication would help, but
regardless of the cause, deployment of this approach has been problematic,
whereas authorization IDs have been more successful.

I will say that any update to RFC 4954 needs to specifically address the
distinction between SMTP submission and relay.

> EXTERNAL an important mechanism in general.

Agreed. I personally prefer to have a consistent approach across protocols,
which would argue for option 1 or 3, not 2, but I don't think the "it can all
be dealt with in an AUTH EXTERNAL update" idea passes technical muster.

				Ned

---

From: Tony Hansen <[email-masked]>
Date: Thu, 06 May 2010 17:46:24 -0400

Ned, do you think an update to 4954 (SMTP Service Extension for 
Authentication) plus an update to 4422 (SASL, where AUTH EXTERNAL is 
defined) would suffice?

While the issue does affect 4409 (Submit), I'm still not convinced that 
the issue needs to be solved in an 4409 update.

     Tony

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 10 May 2010 10:58:51 +0200

On 05/06/2010 06:36 PM, John R. Levine wrote:
> model, zero commands trigger a state change

IMAP assumes that one command does so. Are there any other protocols 
with the same feature? I don't know. IMO one example isn't enough to 
prove an idea bad, but two or three may well be.

> -- if you're externally
> authorized, you're authorized and you go ahead and do what you're going
> to do.

Except that you can't run any commands that have to be run before 
authentication, such as IMAP STARTTLS.

Arnt