Subject: Re: Comments on: draft-melnikov-smtp-priority-02
From: "Robert A. Rosenberg" <[email-masked]>
To: SMTP Discussion <[email-masked]>
Date: Thu, 28 Jul 2011 11:37:05 -0400
List-Id: <ietf-smtp.imc.org>

At 15:19 -0700 on 07/11/2011, Bill McQuillan wrote about Comments on: 
draft-melnikov-smtp-priority-02:

>Section 5.1:
>
>Since the vast majority of messages will either be marked as
>"PRIORITY 0" or presumed to be priority 0, is there any mechanism
>to prevent starvation for those messages marked at a negative
>priority? Note that for a high volume MTA there would likely
>always be SOME priority 0 messages in the queue which would
>preclude sending ANY messages at a lower priority.

Many queuing systems that I have seen reward items that have a low 
priority and thus have sat on the queue and never processed [due to 
there always being higher priority items on the queue that prevent 
their being selected] by periodically bumping their priority. In this 
type of system, you would, for example, designate that a low priority 
item gets its priority bumped one priority notch after it has been 
unprocessed for x minutes (until it gets to priority 0/normal). Thus 
if the bump interval is 15 minutes, a priority -3 will go normal 
after no longer than 45 minutes (and thus be sent as it gets to the 
top of the priority 0 queue [which I assume will be FIFO within 
priority] either due to FIFO on queue or submission timestamp [which 
would make the delayed and bumped item the highest priority 0 item]).

Date: Sat, 23 Jul 2011 22:17:06 -0700
From: Bill McQuillan <[email-masked]>
To: SMTP Discussion <[email-masked]>
CC: Alexey Melnikov <[email-masked]>, ken carlberg <[email-masked]>
Subject: Re: Comments on: draft-melnikov-smtp-priority-02
Sender: [email-masked]

On Fri, 2011-07-22, Alexey Melnikov wrote:

> ken carlberg wrote:
>> On Jul 13, 2011, at 3:03 AM, Bill McQuillan wrote:
>>   
>>>> A value of 0 indicates an email from a client/network not
>>>> supporting priorities or intended to be sent to such a server/
>>>> network; this is the same as not specifying the PRIORITY
>>>> parameter.
>>>>       
>>> The possibility that I might want to send a priority 0 (normal)
>>> message FROM a conforming system TO a conforming system seems not
>>> to be considered.
>>>     
>> ok.  It will be easy to add text and stating up front that a value of zero, by default, indicates "normal".  And then add the existing caveats.  And by "normal", I assume that we are in agreement that this describes the best effort service model that exists today as if the proposed extension never existed.
>>   
> I've tried to clarify this by removing "or intended to be sent", as this
> is not relevant here. My current text reads:

>       A value of 0 indicates an email from a client/network not
>       supporting priorities, which is the same as not specifying the
> PRIORITY parameter,
>       i.e. this value will cause the standard SMTP behaviour in absence
> of this extension.

> Is this better?

How about:

A value of 0 indicates an email which is to be handled at the
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
normal priority level or is from a client/network not supporting
^^^^^^^^^^^^^^^^^^^^^^^^^^^
priorities, which is the same as not specifying the PRIORITY
parameter, i.e. this value will cause the standard SMTP behaviour
in absence of this extension.

-- 
Bill McQuillan <[email-masked]>

Date: Fri, 22 Jul 2011 20:53:42 +0100
From: Alexey Melnikov <[email-masked]>
To: ken carlberg <[email-masked]>, Bill McQuillan <[email-masked]>
CC: SMTP Discussion <[email-masked]>
Subject: Re: Comments on: draft-melnikov-smtp-priority-02
Sender: [email-masked]

ken carlberg wrote:
> On Jul 13, 2011, at 3:03 AM, Bill McQuillan wrote:
>   
>>> A value of 0 indicates an email from a client/network not
>>> supporting priorities or intended to be sent to such a server/
>>> network; this is the same as not specifying the PRIORITY
>>> parameter.
>>>       
>> The possibility that I might want to send a priority 0 (normal)
>> message FROM a conforming system TO a conforming system seems not
>> to be considered.
>>     
> ok.  It will be easy to add text and stating up front that a value of zero, by default, indicates "normal".  And then add the existing caveats.  And by "normal", I assume that we are in agreement that this describes the best effort service model that exists today as if the proposed extension never existed.
>   
I've tried to clarify this by removing "or intended to be sent", as this 
is not relevant here. My current text reads:

      A value of 0 indicates an email from a client/network not
      supporting priorities, which is the same as not specifying the 
PRIORITY parameter,
      i.e. this value will cause the standard SMTP behaviour in absence 
of this extension.

Is this better?

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Discussion <[email-masked]>
Date: Wed, 13 Jul 2011 20:45:23 -0700
Subject: RE: More comments on: draft-melnikov-smtp-priority-02
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Alexey Melnikov
> Sent: Tuesday, July 12, 2011 10:34 AM
> To: Alessandro Vesely
> Cc: [email-masked]; SMTP Discussion
> Subject: Re: More comments on: draft-melnikov-smtp-priority-02
> 
> > Second, protecting MT-Priority by DKIM-signing it results in broken
> > signatures in case the priority is altered by a conforming server
> > before relaying to a non-conforming one.
> 
> Right. This is indeed a problem. But I am not yet sure what would be
> more important - preserving the priority value (in case some downstream
> MTA support it), or preserving the DKIM Signature. I need to think a
> bit more about that.

If the MTA is DKIM-aware, it could detect whether MT-Priority is signed and then decide not to change it.  But that seems pretty complicated.

I think in this case it should simply re-sign the message after alteration.

You could also discourage signing of it, citing this example.

Subject: Re: More comments on: draft-melnikov-smtp-priority-02
From: ken carlberg <[email-masked]>
Date: Wed, 13 Jul 2011 08:17:11 -0400
Cc: Alexey Melnikov <[email-masked]>, SMTP Discussion <[email-masked]>
To: Alessandro Vesely <[email-masked]>
Sender: [email-masked]

Alessandro,

>>> Would it make sense, instead, to have a process that collects a
>>> message with, say, priority -3 and 1000 recipients and re-queues
>>> it as 10 conveniently staggered messages with priority -2 and 100
>>> recipients each?
>> 
>> I don't think this is prohibited, but can you elaborate on why this
>> would be desirable? For example, while not keep the priority -3 when
>> generating multiple messages?

Given your description above, my initial reaction would be to not allow (or just to heavily discourage) the example that you give above.  I still need to think about this some more, but I would have a preference to maintain the consistency of the value of the signaled priority.  On the other hand, if there was a need for a local decision of how to service the set of prioritized messages, then perhaps weights could be set/changed in terms of how to service queues.

> Thank you for considering this problem.  I've seen Bill's observation
> on possible starvation at negative priorities, and wandered whether
> PRIORITY could solve delayed relaying in general, possibly as a
> secondary target.
> 
> Of course, there are myriads of full blown solutions for large scale
> mailing.  Still, some users prefer regular multi-recipient messages,
> even at the costs of breaking their recipient list in chunks of
> acceptable size, and managing bounces by themselves.  As a side
> effect, their behavior may delay normal messages that get queued
> behind the bunch.  For a local solution, a server may handle such
> messages using a specific algorithm for scheduling retries, and do a
> few recipients only on each retry.

As a side note, Alexey and I had a lengthy discussion of resource utilization, and tried to articulate some of this thought in sections 5.1.1 (probability) and 5.1.2 (preemption).  There are systems out there today that have a requirement for preemptive service -- to the point of indefinitely starving out other messages (and clearly, an extreme case).  And other systems have a requirement of focusing on probability so that all messages get serviced/forwarded, but that some will be treated preferentially over others.

-ken

Subject: Re: Comments on: draft-melnikov-smtp-priority-02
From: ken carlberg <[email-masked]>
Date: Wed, 13 Jul 2011 07:48:30 -0400
Cc: SMTP Discussion <[email-masked]>, Alexey Melnikov <[email-masked]>
To: Bill McQuillan <[email-masked]>
Sender: [email-masked]

On Jul 13, 2011, at 3:03 AM, Bill McQuillan wrote:

>> A value of 0 indicates an email from a client/network not
>> supporting priorities or intended to be sent to such a server/
>> network; this is the same as not specifying the PRIORITY
>> parameter.
> 
> The possibility that I might want to send a priority 0 (normal)
> message FROM a conforming system TO a conforming system seems not
> to be considered.

ok.  It will be easy to add text and stating up front that a value of zero, by default, indicates "normal".  And then add the existing caveats.  And by "normal", I assume that we are in agreement that this describes the best effort service model that exists today as if the proposed extension never existed.

-ken

Date: Wed, 13 Jul 2011 12:08:00 +0200
From: Alessandro Vesely <[email-masked]>
To: Alexey Melnikov <[email-masked]>
CC: [email-masked], SMTP Discussion <[email-masked]>
Subject: Re: More comments on: draft-melnikov-smtp-priority-02
Sender: [email-masked]

Hi Alexey,

On 12/Jul/11 19:34, Alexey Melnikov wrote:
> On 12/07/2011 18:06, Alessandro Vesely wrote:
>> Two comments for section 10, Security Considerations:
>>
>> First, the phrase
>>
>>                                       Message Submission Agents can
>>     implement a policy that only allows authenticated users (or only
>>     certain authenticated users) to specify message priorities
>>
>> seems redundant.  Since the message priority can only be specified
>> during transactions and authentication is implied at such stage, the
>> parenthesized "only certain" is the working case.
>
> There is no requirement to use SMTP AUTH before using this extension

(I meant before using MSA)

> (althought it would have been a good idea). But in general, I think
> emphasizing that priority for unauthenticated messages shouldn't be
> trusted is important in the Security Considerations section.

Yes, that looks right.  Then something like

   Message Submission Agents /and SMTP servers/ can implement a...

>> Second, protecting MT-Priority by DKIM-signing it results in broken
>> signatures in case the priority is altered by a conforming server
>> before relaying to a non-conforming one.
>
> Right. This is indeed a problem. But I am not yet sure what would be
> more important - preserving the priority value (in case some
> downstream MTA support it), or preserving the DKIM Signature. I need
> to think a bit more about that.

An MTA can also sign a message anew, after changing MT-Priority, if it
knows what it's doing --that is, if downstream agents trust its
signature better than the original one.  OTOH, if using PRIORITY
breaks DKIM, people may want to avoid using both at the same time.

>> Would it make sense, instead, to have a process that collects a
>> message with, say, priority -3 and 1000 recipients and re-queues
>> it as 10 conveniently staggered messages with priority -2 and 100
>> recipients each?
>
> I don't think this is prohibited, but can you elaborate on why this
> would be desirable? For example, while not keep the priority -3 when
> generating multiple messages?

Thank you for considering this problem.  I've seen Bill's observation
on possible starvation at negative priorities, and wandered whether
PRIORITY could solve delayed relaying in general, possibly as a
secondary target.

Of course, there are myriads of full blown solutions for large scale
mailing.  Still, some users prefer regular multi-recipient messages,
even at the costs of breaking their recipient list in chunks of
acceptable size, and managing bounces by themselves.  As a side
effect, their behavior may delay normal messages that get queued
behind the bunch.  For a local solution, a server may handle such
messages using a specific algorithm for scheduling retries, and do a
few recipients only on each retry.

Now, what if the server uses a smarthost?  If its connection gets
idle, it is a good moment for transferring anything, even negative
priority messages.  But how does it express the difference between a
whole-transfer and a specifically-scheduled partial relay?  The latter
has paid its waiting time already, so it would seem fair to reset any
negative priority that might have been imposed on it on submission.

Date: Wed, 13 Jul 2011 00:03:21 -0700
From: Bill McQuillan <[email-masked]>
To: SMTP Discussion <[email-masked]>
CC: Alexey Melnikov <[email-masked]>, [email-masked]
Subject: Re: Comments on: draft-melnikov-smtp-priority-02
Sender: [email-masked]

On Tue, 2011-07-12, Alexey Melnikov wrote:

>> Section 3, item 5:

>> This seems to reject that PRIORITY 0 should mean "Normal"
> Can you elaborate on why you think so?

In this sentence:

> A value of 0 indicates an email from a client/network not
> supporting priorities or intended to be sent to such a server/
> network; this is the same as not specifying the PRIORITY
> parameter.

The possibility that I might want to send a priority 0 (normal)
message FROM a conforming system TO a conforming system seems not
to be considered.

-- 
Bill McQuillan <[email-masked]>

Date: Tue, 12 Jul 2011 18:55:10 +0100
From: Alexey Melnikov <[email-masked]>
To: Bill McQuillan <[email-masked]>
CC: SMTP Discussion <[email-masked]>, [email-masked]
Subject: Re: Comments on: draft-melnikov-smtp-priority-02
Sender: [email-masked]

Hi Bill,
Thank you for the comments.

On 11/07/2011 23:19, Bill McQuillan wrote:
> Comments on: draft-melnikov-smtp-priority-02
> SMTP Extension for Message Priorities
>
>
> Section 2, last paragraph:
>
> Seems to be self-contradictory. Either there are enough resources to
> cause an email to be "ready to be sent" or there aren't!
I will check.
> Section 3, item 5:
>
> This seems to reject that PRIORITY 0 should mean "Normal"
Can you elaborate on why you think so?

> Section 4.2, next to last paragraph:
>
> I believe that the recommendation should be stronger, at least a
> "SHOULD". There must be some cost to using (non-negative)
> PRIORITY. This will guard against abuse. Otherwise, how do you
> deter spammers from always using PRIORITY 99?
This sounds sensible.
> Another thing to worry about is that this becomes a new Denial-
> of-Service mechanism if not controlled in some way.
>
> Either authentication, SPF (RFC 4408) or similar should be used
> before PRIORITY is allowed.
I will try to make the current text stronger on that.
> Section 4.2, last paragraph:
>
>     "priority (both to lower or to upper it)."
>
>   Perhaps:
>
>     "priority (both to lower or to raise it)."
Ok.
> Section 4.3, Item 1:
>
> I think that this means that a message received with with NO
> priority indication (thus "normal") MUST be relayed with a
> PRIORITY 0 parameter. This seems overly stringent.
This is necessary if there is a PRIORITY parameter and an MT-Priority 
header field value and they don't agree for whatever reasons.
> Also, does this paragrah mean that if an MTA only implements the
> suggested 6 priority levels, a message received with, e.g.,
> PRIORITY 93 MUST be relayed with a PRIORITY 93 parameter? This
> seems reasonable, but the text is not quite clear.
Right, that was the intent. I will clarify.
> Section 4.6, item 2:
>
> What is needed is some sort of meaning specified for the 199
> possible values that does not require reference to other mutually
> inconsistent priority schemes.
>
> E.g., MMHS: -20 = routine, 0 = priority. Really?
Right, I've messed up with the mapping. The meaning of priority 0 as 
"normal" was added later... I will fix.
> So, does "PRIORITY 50" mean:
>
>     "You should probably read this first"
> or:
>     "CAP'N SHE'S GONNA BLOW!"?
>
>
> Section 5.1:
>
> Since the vast majority of messages will either be marked as
> "PRIORITY 0" or presumed to be priority 0, is there any mechanism
> to prevent starvation for those messages marked at a negative
> priority? Note that for a high volume MTA there would likely
> always be SOME priority 0 messages in the queue which would
> preclude sending ANY messages at a lower priority.
There is no such mechanism for 5.1. 5.2 will work better in this respect.
> Section 5.1, paragraph 3:
>
> The first sentence is unclear. It seems to have *emails* being
> forbidden to initiate tranactions. I would expect MTAs to be the
> object of any restrictions.
Good point.

Date: Tue, 12 Jul 2011 18:34:16 +0100
From: Alexey Melnikov <[email-masked]>
To: Alessandro Vesely <[email-masked]>
CC: [email-masked], SMTP Discussion <[email-masked]>
Subject: Re: More comments on: draft-melnikov-smtp-priority-02
Sender: [email-masked]

Hi Alessandro,
Thank you for your comments.

On 12/07/2011 18:06, Alessandro Vesely wrote:
> Two comments for section 10, Security Considerations:
>
> First, the phrase
>
>                                       Message Submission Agents can
>     implement a policy that only allows authenticated users (or only
>     certain authenticated users) to specify message priorities
>
> seems redundant.  Since the message priority can only be specified
> during transactions and authentication is implied at such stage, the
> parenthesized "only certain" is the working case.
There is no requirement to use SMTP AUTH before using this extension 
(althought it would have been a good idea). But in general, I think 
emphasizing that priority for unauthenticated messages shouldn't be 
trusted is important in the Security Considerations section.
> Second, protecting MT-Priority by DKIM-signing it results in broken
> signatures in case the priority is altered by a conforming server
> before relaying to a non-conforming one.
Right. This is indeed a problem. But I am not yet sure what would be 
more important - preserving the priority value (in case some downstream 
MTA support it), or preserving the DKIM Signature. I need to think a bit 
more about that.
> If it has to be signed, I'd
> suggest to revise Section 4.4 so as to never formally alter the field
> after it has been signed (presumably by the MSA).  Further MTAs may
> treat the message as if it had a lower priority even without altering
> the field.
>
> I'd also put a question about section 4.5, Mailing lists and Aliases.
> Requiring that the existing priority is retained across expansions
> apparently discourages the use of low/negative priority for running
> large lists.
Yes, but it is a SHOULD level requirement (as opposed to a MUST).
> Would it make sense, instead, to have a process that
> collects a message with, say, priority -3 and 1000 recipients and
> re-queues it as 10 conveniently staggered messages with priority -2
> and 100 recipients each?
I don't think this is prohibited, but can you elaborate on why this 
would be desirable? For example, while not keep the priority -3 when 
generating multiple messages?

Best Regards,
Alexey

Date: Tue, 12 Jul 2011 19:06:24 +0200
From: Alessandro Vesely <[email-masked]>
To: [email-masked], [email-masked]
CC: SMTP Discussion <[email-masked]>
Subject: More comments on: draft-melnikov-smtp-priority-02
Sender: [email-masked]

Two comments for section 10, Security Considerations:

First, the phrase

                                     Message Submission Agents can
   implement a policy that only allows authenticated users (or only
   certain authenticated users) to specify message priorities

seems redundant.  Since the message priority can only be specified
during transactions and authentication is implied at such stage, the
parenthesized "only certain" is the working case.

Second, protecting MT-Priority by DKIM-signing it results in broken
signatures in case the priority is altered by a conforming server
before relaying to a non-conforming one.  If it has to be signed, I'd
suggest to revise Section 4.4 so as to never formally alter the field
after it has been signed (presumably by the MSA).  Further MTAs may
treat the message as if it had a lower priority even without altering
the field.

I'd also put a question about section 4.5, Mailing lists and Aliases.
Requiring that the existing priority is retained across expansions
apparently discourages the use of low/negative priority for running
large lists.  Would it make sense, instead, to have a process that
collects a message with, say, priority -3 and 1000 recipients and
re-queues it as 10 conveniently staggered messages with priority -2
and 100 recipients each?

Date: Mon, 11 Jul 2011 15:19:02 -0700
From: Bill McQuillan <[email-masked]>
To: SMTP Discussion <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Comments on: draft-melnikov-smtp-priority-02
Sender: [email-masked]

Comments on: draft-melnikov-smtp-priority-02
SMTP Extension for Message Priorities

Section 2, last paragraph:

Seems to be self-contradictory. Either there are enough resources to 
cause an email to be "ready to be sent" or there aren't!

Section 3, item 5:

This seems to reject that PRIORITY 0 should mean "Normal"

Section 4.2, next to last paragraph:

I believe that the recommendation should be stronger, at least a 
"SHOULD". There must be some cost to using (non-negative) 
PRIORITY. This will guard against abuse. Otherwise, how do you 
deter spammers from always using PRIORITY 99?

Another thing to worry about is that this becomes a new Denial-
of-Service mechanism if not controlled in some way.

Either authentication, SPF (RFC 4408) or similar should be used 
before PRIORITY is allowed.

Section 4.2, last paragraph:

   "priority (both to lower or to upper it)."
   
 Perhaps:
 
   "priority (both to lower or to raise it)."

Section 4.3, Item 1:

I think that this means that a message received with with NO 
priority indication (thus "normal") MUST be relayed with a 
PRIORITY 0 parameter. This seems overly stringent.

Also, does this paragrah mean that if an MTA only implements the 
suggested 6 priority levels, a message received with, e.g., 
PRIORITY 93 MUST be relayed with a PRIORITY 93 parameter? This 
seems reasonable, but the text is not quite clear.

Section 4.6, item 2:

What is needed is some sort of meaning specified for the 199 
possible values that does not require reference to other mutually 
inconsistent priority schemes. 

E.g., MMHS: -20 = routine, 0 = priority. Really?

So, does "PRIORITY 50" mean:

   "You should probably read this first"
or:
   "CAP'N SHE'S GONNA BLOW!"?

Section 5.1:

Since the vast majority of messages will either be marked as
"PRIORITY 0" or presumed to be priority 0, is there any mechanism
to prevent starvation for those messages marked at a negative
priority? Note that for a high volume MTA there would likely
always be SOME priority 0 messages in the queue which would
preclude sending ANY messages at a lower priority.

Section 5.1, paragraph 3:

The first sentence is unclear. It seems to have *emails* being 
forbidden to initiate tranactions. I would expect MTAs to be the 
object of any restrictions.

-- 
Bill McQuillan <[email-masked]>