Subject: [lemonade] update to RFC 1845 checkpoint/restart

From: Tony Finch <[email-masked]>
Date: Wed, 15 Mar 2006 04:46:06 +0000

A few weeks ago I started a discussion on the lemonade list about
efficient message transmission using a combination of PIPELINING and
CHUNKING. I've since written a draft which addresses the increased risk of
message duplication which can occur with fully-pipelined message
transmission. The meat of it is a more pipelinable version of RFC 1845
checkpoint/restart. The gravy is a few sections of rationale...

http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-replay.html

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NITON: A DEPRESSION MOVING EAST INTO TRAFALGAR WILL GENERATE STRONG EASTERLY
WINDS IN ALL AREAS DURING FRIDAY, PERHAPS GALE FORCE IN NORTH FITZROY, SOLE
AND SHANNON.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Wed, 15 Mar 2006 15:11:11 +0000

> http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-replay.html

Comments, in no particular order:

1) Why have a distinct response 255 instead of 355 <length of 
complete message>? A client's behaviour is likely to be the same, and 
semantically, there is no difference.

2) This specification is needlessly not compatible with RFC1845 on a 
number of points. I can't see why this needs to be so.

3) The DATA offset restriction (beginning of a line) is problematic 
because it forces a server to store the message one entire line in 
arrears, because of the multi-line termination of the data 
transmission with DATA. This is probably required, in order to gain 
compatibility with RFC1845, but it needs to be pointed out.

4) Section 9, first para, is an example of (2).

5) The syntax of REPLAY means that the syntax of txn-id is not 
accurate as given in section 6. A txn-id MUST NOT be "CLEAR", in 
particular. The syntax of REPLAY is, in my opinion, quite ugly as a 
result.

6) There's no mention here of BURL.

a) An octet-offset provided by the server MUST NOT point into a BURL 
chunk. BURL chunks MUST be appended atomically.

b) For BURL and BDAT commands, an additional parameter could be 
defined, which could then be send in lieu of, or in addition to, the 
octet-count by the server, thus providing for the (rare) case where a 
client cannot know the octet count of a BURL chunk without 
downloading it.

7) Your description of how to send no data with DATA is incorrect, 
it's actually DATA<CRLF><wait for 354><CRLF>.<CRLF>

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Tony Finch <[email-masked]>
Date: Wed, 15 Mar 2006 15:37:25 +0000

On Wed, 15 Mar 2006, Dave Cridland wrote:

Thanks for your comments.

> 1) Why have a distinct response 255 instead of 355 <length of complete
> message>? A client's behaviour is likely to be the same, and

I vacillated a bit here - at one point the spec also had a 555 response
instead of 355 0. You are probably right that 255 is also redundant.

> 4) Section 9, first para, is an example of (2).

Hmm, yes, I suppose it might be possible (and desirable) to be backwards
compatible.

The REPLAY spec is currently weak in descibing the line between a retry
and a replay. For example, the 555 response to MAIL would be better as a
455 that would cause the client to retry with a new transaction (i.e. a
fresh <txn-id>). The corresponding situation with an improved CHUNKING
would be an unexpected 355 response. The problem here is what happens if
the client has pipelined some data after the erroneous command?

> 3) The DATA offset restriction (beginning of a line) is problematic because it
> forces a server to store the message one entire line in arrears, because of
> the multi-line termination of the data transmission with DATA. This is
> probably required, in order to gain compatibility with RFC1845, but it needs
> to be pointed out.

I'm not sure it's necessary to be a line in arrears. You can store each
line as you receive it, until you get a line containing just a dot. You
only need a one-line window.

> 5) The syntax of REPLAY means that the syntax of txn-id is not accurate as
> given in section 6. A txn-id MUST NOT be "CLEAR", in particular. The syntax of
> REPLAY is, in my opinion, quite ugly as a result.

Perhaps it should be a separate command. I also considered making the
transaction log clearing implicit, when the server sees a QUIT from the
client. But what commend to use when the client wants to leave the
connection idle waiting for new messages? EHLO?

> a) An octet-offset provided by the server MUST NOT point into a BURL chunk.
> BURL chunks MUST be appended atomically.

Yes, that's a sound suggestion.

> b) For BURL and BDAT commands, an additional parameter could be defined, which
> could then be send in lieu of, or in addition to, the octet-count by the
> server, thus providing for the (rare) case where a client cannot know the
> octet count of a BURL chunk without downloading it.

Perhaps a chunk count as well as an octet count? There would have to be an
octet offset within the chunk, so that the chunk count and chunk offset
together mean the same thing as the octet offset. I'll see if I can think
of a good way of specifying it. I'm wary because its in danger of getting
a bit complicated.

> 7) Your description of how to send no data with DATA is incorrect, it's
> actually DATA<CRLF><wait for 354><CRLF>.<CRLF>

That's two octets of data (a blank line), not zero :-) but I should
mention the wait for 354. cf. the text about a "single dot" in RFC2920.

Tony.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 15 Mar 2006 16:46:10 +0000

> Thanks for your comments.
Pleasure. RFC1845 has a lot to offer, I think.

> the client has pipelined some data after the erroneous command?
I don't follow all of this.

> accurate as
No, because EHLO is already mandated as being a noisier RSET. I think 
a new command, or else changing the syntax of REPLAY. I have no 
preference as to which.

> Yes, that's a sound suggestion.
It's by far the simplest.

> defined, which
Yes, that would be complicated.

FWIW, I can only think of one case where the length of a resource is 
likely to be unknown, and that's the case where the client only has 
the URLAUTH URL. That said, the only case I can think of for that 
involves Alice granting Bob read access to a message via URLAUTH, and 
Bob using an ESMTP send with BURL in order to copy the message to his 
IMAP server in order to read it efficiently. That is a somewhat 
contrived use-case.

I'd be happy to see the entire problem noted and subsequently ignored.

> mention the wait for 354. cf. the text about a "single dot" in 

(This is "below") Ah, I sit corrected. I'd previously somehow misread 
RFC2821 section 4.1.1.4 para 2 as saying the exact opposite of what 
it does say. I have no idea how I misread that. This makes things 
much easier, and of course eliminates my point 3.

Dave.

---

From: Tony Finch <[email-masked]>
Date: Wed, 15 Mar 2006 19:52:14 +0000

> I don't follow all of this.

I'm worried about transaction ID collisions, which I suppose are most
likely to be caused by a client going insane (unless someone manages to
successfully attack a client's txn-id generation algorithm). At the moment
the replay spec makes this problem detectable by the 555 response to MAIL
(though I plan to change this to 455). Checkpoint assumes this situation
is OK since it looks to the server like a normal transaction restart.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOLE LUNDY FASTNET SOUTH IRISH SEA: EAST 4 OR 5, OCCASIONALLY 6 LATER.
SHOWERS, BECOMING WINTRY FOR A TIME. MODERATE OR GOOD, OCCASIONALLY POOR AT
FIRST.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 15 Mar 2006 23:33:36 +0000

> is OK since it looks to the server like a normal transaction 
Right, gotcha.

You're saying that in the case where a transaction id collides, or, 
for some reason, the client thinks it's sending the transaction anew 
instead of resuming, then there's a problem here that 1845 missed 
because of the lack of pipelining.

A solution is to state that use of TRANSID= denotes either resumption 
or new, and the client must wait for the response unless it knows 
what it will be in advance (by having previously issued a REPLAY 
command and examined the response). Then add an additional parameter 
which indicates to the server that the TRANSID is expected to be 
fresh, and if not, to reject this transaction. (Perhaps a  
TRANSID-NEW assertion parameter)

This means that RFC1845bis (ie, this spec) clients use:

MAIL FROM:<[email-masked]> TRANSID=xyz TRANSID-NEW

for starting a new transaction, and do not need to synchronize, 
whereas RFC1845 clients don't use the additional parameter, but do 
synchronize.

That handles the collision case, but the case where two clients 
simultaneously attempt to resume a TRANSID, is much more difficult to 
deal with.

You could define REPLAY as an assertion that subsequent usage of this 
TRANSID in a MAIL command in this session is a resumption, which 
prevents accidental duplication, and then you merely have to deal 
with failing DATA, BDAT, and BURL commands if a TRANSID is resumed by 
another session. (And, one assumes noting this in logs or to the 
sender as a possible bug or security issue.)

Dave.