Subject: Re: [Gen-art] Gen-ART LC review of draft-dusseault-http-patch-15.txt

From: Julian Reschke <[email-masked]>
Date: Fri, 13 Nov 2009 08:19:52 +0100

Brian E Carpenter wrote:
> As far as I can tell, the proposal places the burden for ensuring
> atomicity entirely on the server. However, PATCH is explicitly not 
> idempotent. If a client issues a PATCH, and the server executes the PATCH,
> but the client then fails to receive an indication of success due to
> an extraneous network glitch (and TCP reset?), then what prevents the client
> issuing the same PATCH again? In other words, absent a two-phase commit,
> there appears to be no transactional integrity.

How is this different from PUT or POST? If you need repeatability of the 
request, just make the request conditional using "if-match"...

BR, Julian

---

From: Eliot Lear <[email-masked]>
Date: Fri, 13 Nov 2009 10:00:23 +0100

Brian,

> there appears to be no transactional integrity.

I think Lisa should respond to this as well.  One of the issues here is 
that I believe there is an assumption that there will be some external 
means to know that in fact the PATCH succeeded, like for instance 
retrieving the impacted web pages and comparing results.  In thinking 
about this again, however, perhaps that may not be sufficient, if for 
instance, PATCH is used for purposes of a side effect that is not easily 
verified.  At that point you probably want something a kin to a 
transaction ID where you can check against that ID  to deterine if it 
had succeeded.

An alternative view, however, would be that for a specific object type 
where that is important, one could imbed such a transaction ID 
somewhere.  That's not so easy for common objects we retrieve today (you 
would have to add some sort of meta tag into an HTML doc, and who knows 
what you would do with images), but since those objects could easily be 
retrieved and compared, they're not the problem.

Put this another way: I don't think PATCH is a replacement for all the 
network database protocols and mechanisms that exist today, but simply 
closer to what it is named: a means to patch one or more objects.

Eliot

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 13 Nov 2009 22:00:58 +1300

> request, just make the request conditional using "if-match"...

PATCH seems more dangerous than those simply because it is partial
update of a resource, and I don't feel it's sufficient to say that
there might be a way of detecting that it has failed to complete,
if you're executing a series of patches that build on one another.

Talking about transactional integrity in the IETF has always been
hard, for some reason. But something described as "patch" is exactly
where you need it, IMHO.

    Brian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 13 Nov 2009 11:35:58 +0100

> if you're executing a series of patches that build on one another.

POST can be a partial update as well, for the simple reason that POST 
can be *anything*. As a matter of fact, people are using POST right now, 
as PATCH was removed in RFC 2616.

> hard, for some reason. But something described as "patch" is exactly

PATCH does not need to be special, and shouldn't be special. That being 
said, it wouldn't hurt to clarify that PATCH can be made repeatable 
(idempotent) by making it conditional.

Best regards, Julian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 14 Nov 2009 11:26:44 +1300

Julian,

> as PATCH was removed in RFC 2616.

I wasn't involved in reviewing RFC2616 (or in the original design of POST,
even though it happened about 20 metres from my office at that time). But
yes, POST also lacks transactional integrity. Incidentally, that recently
caused me to donate twice as much as I intended to the relief fund for
the tsunami in Samoa, although the direct cause was a network glitch.

> (idempotent) by making it conditional.

I would make that a SHOULD and, as I said originally, be more precise
about how to use strong ETags to achieve it.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 14 Nov 2009 13:16:15 +1300

Roy,

Just a couple more comments below, and then I will have
said all I can usefully say.

On 2009-11-14 12:57, Roy T. Fielding wrote:
> On Nov 13, 2009, at 2:26 PM, Brian E Carpenter wrote:
> That doesn't have anything to do with POST lacking transactions.
> Any server is fully capable of detecting repeated transactions
> just by looking at the data sent.

Not in this case. The service was instantiated once, received
a legitimate series of POSTs, including the final one triggered
by a SUBMIT button, and then the final response code to the client
(a standard browser) vanished, for whatever reason. [Digression
about why this failure was probably caused by a NAT deleted.]
The human client simply sees a browser timeout and has no possible
way of knowing whether the SUBMIT executed. As it turns out, my credit
card bill later showed that it did.

> No.  The patch format may be designed to be repeatable (append).
> The patch format may include its own conditions (context diffs,
> before/after metadata, etc.).  HTTP is neither a filesystem nor
> a database, so don't expect the protocol to interfere with
> legitimate communication just because some resources might
> require transactions for tightly-coupled behavior.  That is not
> the protocol's job.

Given that HTTP took the stateless approach that it did,
I don't expect the protocol to embed transactional integrity.
But I do expect the protocol spec to describe in normative terms
how to detect a relatively likely type of failure. I agree
that whether that is needed depends on the use case for PATCH.

     Brian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 13 Nov 2009 15:57:04 -0800

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 13 Nov 2009 19:47:26 -0800

> card bill later showed that it did.

Yes, and that is exactly what would happen even if HTTP supported
full two-phase commit, with all of its painful consequences, if the
response to commit was lost.  This is an application problem, not
a protocol requirement.  It can be "fixed" (to the degree that any
fix is possible in a distributed system) by providing the client
with a status resource within the same page as the SUBMIT, such
that the client can see the change in server state when it takes
place even if the specific 200 response is lost.  In other words,
provide the client with the information necessary to check the bill
before the final transaction is submitted.

> ...

> that whether that is needed depends on the use case for PATCH.

The use case for PATCH is requesting relative state changes to
a known resource.  The way to detect the resulting state of the
resource in case of communication error is to perform a GET
(perhaps with Range) on the same resource.  That is one of the
few benefits of defining PATCH instead of just reusing POST.

....Roy

---

From: John C Klensin <[email-masked]>
Date: Sun, 15 Nov 2009 15:03:02 -0500

Hi.

fwiw, I find myself strongly agreeing with what I believe is
Brian's main point, although what I infer from it may be a
little different from what he does (or may not... I'm not sure):

We should not be adopting a "patch" protocol that lacks both the
tools for, or a serious discussion of, transactional integrity.
The fact that existing mechanisms lack such provisions is not an
excuse when new mechanisms are specified (or even when older
ones are clarified).   As a look forward, the comments in this
note are intended as suggested requirements about revisions to
HTTP as well.

It seems to me that this general goal can be accomplished in any
of the following ways:

	(1) Add an explicit confirmation-handshake mechanism to
	PATCH.
	
	(2) Add an explicit discussion of how ETAG, or some
	other mechanism (such as refetching the page and
	checking whether the patch was applied), can be used to
	accomplish the purpose of a confirmation handshake.
	
	(3) Add an explicit Security Considerations discussion
	about the risks associated with either assuming a patch
	has properly occurred without verifying that fact in
	some way or of re-issuing a PATCH command without
	verifying that it had not been applied.  This discussion
	must including a discussion of proxies and caches to be
	sufficient.
	
Note that the third is not exclusive of the other two; it would
probably still be desirable and might be necessary.

Whatever is done, it must be in the document itself and must be
explicit: comments on the mailing list about mechanisms that
might be usable for the purpose are not, IMO, sufficient for a
standards-track document.

regards,
    john

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sun, 15 Nov 2009 14:54:48 -0800

> tools for, or a serious discussion of, transactional integrity.

John, HTTP is not SQL.  The transactional integrity is inside
the resource implementation.  The entire transaction consists
of a single request message and a single response message.
HTTP is responsible for the communication interface, not the
implementation of specific resources, and cannot proscribe a
specific transaction semantic because it is NOT WANTED by
many resources.

For example, a collaborative whiteboard in which many people
are patching at once, the patches consisting of context diffs
that are internally capable of distinguishing conflicts, would
be impossible to implement via standard etag behavior.
A non-HTTP example of that would be a Google wave resource
and its ability to compose a tree of wavelets.

Standard etag conflict resolution is not required because
it is not desirable for many applications of PATCH.  For
other applications of PATCH, it is already defined by HTTP
and does not need to be reiterated here.

> 	(1) Add an explicit confirmation-handshake mechanism to

I don't know what you mean by that.  If you mean that the server
needs to provide the client with a way to confirm that the PATCH
has been applied (or not), then the mechanism already exists.
Perhaps the draft just needs to make it obvious that GET on the
same URI (with Cache-Control: max-age=0) will do that.

If you mean that HTTP should implement 2PC, then forget it.
The confirmation of commit is just as likely to be lost as
the previous message saying it is okay to commit, so all we
would accomplish is doubling the chance of failure.

> 	accomplish the purpose of a confirmation handshake.

I think you mean "add a discussion of error-handling when the
response message is lost due to server or connection failure"
to section 2.2.

> 	must including a discussion of proxies and caches to be

How is that a security consideration?

> standards-track document.

Perhaps we should add these for discussion in HTTPbis, part 4.

Regarding future versions of HTTP, there is no general
solution to the lost-confirmation-message problem in
distributed networks.  The one I described (providing
a status URI before the action is attempted) can be done
using a standard Link relation in HTTP, HTML, or by using
dynamic javascript within the HTML.  All of those can be
done right now, without changing HTTP.  Note that those
are more applicable for POST, however, since PATCH is
confirmed via GET on the same URI, by definition.

If the IESG wants to add such a link relation to Mark's
draft currently in last call (I think) or within the link
relation registry it defines, I'm all for it.

....Roy

---

From: Eliot Lear <[email-masked]>
Date: Mon, 16 Nov 2009 07:49:18 +0100

On 11/15/09 11:54 PM, Roy T. Fielding wrote:
> John, HTTP is not SQL.

+1.  Right tool for the right job, please.

Eliot

---

From: John C Klensin <[email-masked]>
Date: Mon, 16 Nov 2009 10:24:58 -0500

--On Sunday, November 15, 2009 14:54 -0800 "Roy T. Fielding"
<[email-masked]> wrote:

>> We should not be adopting a "patch" protocol that lacks both
>> the tools for, or a serious discussion of, transactional
>> integrity.
> 

Roy, I certainly agree with the first part of this.  HTTP is not
SQL.  If it were, I (at least) would be loudly insisting on an
integrity mechanism.  However, normal IETF practice includes
being explicit about issues and traps with specifications.  So,
while it is plausible to take the position you express above,
that just about requires that the specification explicitly
indicate that verifying transactional integrity is the
responsibility of the calling application.  I'm also suggesting
that it should at least point to ways to do that.   The
paragraph in Security Considerations (Section 5) that reads:

	"A document that is patched might be more likely to be
	corrupted than a document that is overridden in
	entirety, but that concern can be addressed through the
	use of mechanisms such as conditional requests using
	ETags and the If-Match request header."

is not, IMO, adequate in that regard, at least absent a
normative reference.  It is also the key to the issue as I see
it: while POST can be used to simulate PATCH, the normal and
obvious use of POST is to supply some information to the server
or to replace ("override") an entire document, PATCH would seem
to have, well, patching as its primary purpose.  

> be impossible to implement via standard etag behavior.

Sure.  So say that in the document.   My problem is _not_ with
the mechanism, it is with what you apparently assume people will
figure out for themselves or learn through oral tradition.

> and does not need to be reiterated here.

We disagree.  I believe it does "need to be reiterated here",
either in-line or by an explicit, normative, pointer to the
relevant portion of the HTTP spec.

    john

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 16 Nov 2009 13:55:52 -0500

I agree with John on this one.

We do not need to provide the mechanism, we do need to provide the warning.

On Mon, Nov 16, 2009 at 10:24 AM, John C Klensin <[email-masked]> wrote:
> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

-- 
-- 
New Website: http://hallambaker.com/
View Quantum of Stupid podcasts, Tuesday and Thursday each week,
http://quantumofstupid.com/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Nov 2009 15:27:05 +0100

> relevant portion of the HTTP spec.

<http://tools.ietf.org/html/draft-dusseault-http-patch-15#section-2> 
already says:

    Collisions from multiple PATCH requests are more dangerous than PUT
    collisions, because a patch document that is not operating from a
    known base point may corrupt the resource.  Clients wishing to apply
    a patch document to a known entity can first acquire the strong ETag
    [RFC2616] of the resource to be modified, and use that Etag in the
    If-Match header on the PATCH request to verify that the resource is
    still unchanged.  If a strong ETag is not available for a given
    resource, the client can use If-Unmodified-Since as a less-reliable
    safeguard.

So would replacing the 2nd part by:

    Clients wishing to apply a patch to a known entity can first
    acquire an ETag ([RFC2616], Section 3.11) of the entity to be
    modified, and use that ETag to make the PATCH request conditional,
    such as by including the "If-Match" request header field ([RFC2616],
    Section 14.24).

address this concern?

Note that this makes some more changes:

1) the client can't control whether the etag will be strong, and weak 
etags may work just fine in certain instances, so just be silent about 
the type

2) speak of "conditional requests" in general, and just mention 
"If-Match" as one way to get there; there may be other ways such as 
using the WebDAV "If" header

3) Get rid of the text that suggests that a last modified date somehow 
is better than a weak etag.

Best regards, Julian

---

From: John C Klensin <[email-masked]>
Date: Tue, 17 Nov 2009 10:50:34 -0500

--On Tuesday, November 17, 2009 15:27 +0100 Julian Reschke
<[email-masked]> wrote:

> <http://tools.ietf.org/html/draft-dusseault-http-patch-15#sect
> ion-2> already says:
> 

Largely, yes, as long as the other changes you list below are
addressed (comments below).

> 1) the client can't control whether the etag will be strong,
> and weak etags may work just fine in certain instances, so
> just be silent about the type

Silent, no.  But saying "can't control... certain instances"
explicitly would be fine.  I'd be even happier with an
explanation of what such an instance might look like, but don't
see that as a requirement.

> 2) speak of "conditional requests" in general, and just
> mention "If-Match" as one way to get there; there may be other
> ways such as using the WebDAV "If" header

yes

> 3) Get rid of the text that suggests that a last modified date
> somehow is better than a weak etag.

Probably a good idea.

     john

---

From: Julian Reschke <[email-masked]>
Date: Tue, 17 Nov 2009 17:03:10 +0100

> see that as a requirement.

It's up to the server to decide whether it provides strong or weak 
etags. And it's also up to the server to decide whether you can use them 
in a conditional PATCH request (RFC 2616 disallowed this, but HTTPbis is 
lifting that restriction, and furthermore WebDAV never had it).

I think not stating this explicitly is the simplest approach (as this is 
true for any HTTP method), but I wouldn't object to have more text 
either (as long as that text wouldn't have to revised when HTTPbis is done).

BR, Julian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 18 Nov 2009 08:53:58 +1300

These are the sort of changes that would, I believe, give
sufficient indication to a would-be user of PATCH of how
to make it somewhat safe. Personally I'd prefer to see it
made more prominent by starting with something like:

Clients requiring to verify the consistent application of a
patch document to a known entity SHOULD first acquire an ETag...

Rationale: the use of a normative keyword will draw the
attention of implementors who might otherwise not think
about this issue.

   Brian

On 2009-11-18 05:03, Julian Reschke wrote:
> Gen-art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/gen-art
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 17 Nov 2009 18:18:59 -0800

> attention of implementors who might otherwise not think

It would also be wrong, because it is neither a requirement
for interoperation nor a potential for causing harm (RFC 2119).
Aside from which, it makes the original purpose of PATCH
non-compliant with its own specification.

The purpose of PATCH is to request that the server apply a
set of changes to the current state of the target resource.
The assumption that these changes will be dependent on a
specific prior representation of that resource is false.
The server is fully capable of detecting and reporting
conflicts when they occur with the current state, as only
truly known by the server.

In other words ...

 If the client wants to prevent the PATCH method from being
 applied to a resource for which the state has changed since
 the last state known by the client, then it SHOULD use one
 or more of the conditional request mechanisms of HTTP
 (If-Match and If-Unmodified-Since request headers [RFC2616])
 or WebDAV (If request header [RFC4918]) with the
 associated metadata from that prior resource state.
 However, if the patch media type contains its own mechanism
 for detecting conflicts, such as embedded context or metadata
 designed to allow non-overlapping changes to be safely applied,
 then the conditional request mechanisms SHOULD NOT
 be used with PATCH because they would interfere with
 collaborative applications, such as shared editors and
 whiteboards.

FTR, the prior sentence, that PATCH is somehow more likely
to result in corrupted state than a PUT, is simply false for
any patch format that contains context or post-application
integrity checks.  The only reason it was in the spec is
because earlier versions assumed a patch format that contains
nothing but byte-vector manipulations.  It should be removed,
or at least altered to be factual.

....Roy

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 19 Nov 2009 09:03:22 +1300

Roy,

At this point I think we're arguing in a circle, so I will
simply wait to see what the authors and Area Director want to
do next. A Gen-ART review has no more standing than any other
Last Call comment.

Regards
   Brian Carpenter

---

From: Lisa Dusseault <[email-masked]>
Date: Wed, 25 Nov 2009 10:44:08 -0800

I've added extra explanatory text in section 2 as well as the Security
Considerations section.  It's normative, but dependent on the use case
being appropriate, which gives emphasis to the implementor but still
allows the right thing to be done.

Lisa

---

From: Julian Reschke <[email-masked]>
Date: Thu, 26 Nov 2009 13:46:11 +0100

> or at least altered to be factual.

In the meantime, a new draft was published, see 
<http://tools.ietf.org/html/draft-dusseault-http-patch-16> and 
<http://tools.ietf.org/rfcdiff?url2=draft-dusseault-http-patch-16.txt> 
for the diffs.

The new text is:

    A PATCH request can be issued in such a way as to be idempotent,
    which also helps prevent bad outcomes from collisions between two
    PATCH requests on the same resource in a similar timeframe.
    Collisions from multiple PATCH requests may be more dangerous than
    PUT collisions, because some patch formats need to operate from a
    known base point or else corrupt the resource.  Clients using this
    kind of patch application SHOULD acquire a strong ETag [RFC2616] for
    the resource to be modified, and use that ETag in the If-Match header
    on the PATCH request to verify that the resource is still unchanged.
    If a strong ETag is not available for a given resource, the client
    can use If-Unmodified-Since as a less-reliable safeguard.

This text is still problematic, in that

1) It suggests that the client is in control of the etag ("...acquiere a 
strong etag..."); however, the client has no control over this at all; 
it's the server who decides, and also it's the server who's in charge in 
deciding what type of etags it accepts in which operations.

2) It doesn't mention that WebDAV defines another conditional header 
which doesn't have the limitations of If-Match (per RFC2616, not HTTPbis).

I found Roy's proposal both easy to understand and correct, and like to 
see it (or something more similar to it than the current text) being used.

Best regards, Julian

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 29 Nov 2009 09:27:33 +1300

I'm quite happy for the subject matter experts to decide between
these two approaches.

Thanks
   Brian