Subject: [alto] Incremental Update/SSE Key Design Issue 2: sending full resources at the beginning of stream

From: Xiao SHI <[email-masked]>
Date: Thu, 23 Oct 2014 06:37:12 -0400

Hi folks,

Another design question is whether we should include the full resource
(e.g. full network map/cost map) when the SSE stream is first initiated.

The advantage of doing so is that the server does not need to keep any old
version of the resources. The full map the server first sends to the client
will always be updated, and the updates come when they become available.

It also has the following disadvantage:
(1) Modularity: SSE streams are for incremental updates, it carries a
different functionality than full resource requests;
(2) Space efficiency: to reduce the payload for the SSE stream, it would be
nice not to do that. It is conceivable that the client fetches the full
resource somewhere else and then ask for incremental updates via HTTP SSE
stream.

If we do not send the full resource via SSE, the client needs to request
for incremental updates with resource-id and version-tag. there are two
possible extreme solutions:
(1) the laziest: The server accepts the request that have the most recent
version-tag; if the version-tag is not the most update version, the server
simply rejects this request.
(2) the most costly: The server remembers all history of all resources,
when the client requests with a resource-id and vtag, the server accepts
the connection, and immediately send out patches to "catch the client up."

*Example:*
Notate version tags as vi, e.g. v0, v1, v2, etc. And the patches that
updates vi to vj as patch(vi->vj) where vi is the old tag and vj is the new
tag.

Say the current network map is v3. Client 1 requests incr updates with
{"resource-id":"my-networkmap", "tag":"v0"} and Client 2 requests incr
updates with {"resource-id":"my-networkmap", "tag":"v3"}. The lazy server
would reject the request from client 1 and accepts that from client 2. The
costly server would accept both request and send patch(v0->v3)

Another possible solution is a hybrid of the two extremes. The server keeps
a patch that can update the resource from vi to vj, and accepts requests
that has vtag vk such that i<=k<=j.

Because of the JSON merge-patch replacement property, applying patch(*vi*->vj)
would update vk to vj. No matter what vtag vk the request comes in with,
the server accepts if i<=k<=j, and immediately sends patch(vi->vj) via the
stream. This will update the client's resource to the most updated version.
The subsequent updates will be pushed as they become available.The window
of [i, j] can be determined by the server, according to time or storage
constraints.

You comments would be much appreciated. Thank you!

Best,
Xiao

---

From: "Y. Richard Yang" <[email-masked]>
Date: Fri, 24 Oct 2014 00:46:34 -0400

Good design question.

On Thu, Oct 23, 2014 at 6:37 AM, Xiao SHI <[email-masked]> wrote:

> (2) Space efficiency: to reduce the payload for the SSE stream, it would
> be nice not to do that. It is conceivable that the client fetches the full

This also means that SSE serves to update only one resource?

> Another possible solution is a hybrid of the two extremes. The server
> keeps a patch that can update the resource from vi to vj, and accepts
> requests that has vtag vk such that i<=k<=j.
>
I prefer this last one :-)

Richard

-- 
-- 
 =====================================
| Y. Richard Yang <[email-masked]>   |
| Professor of Computer Science       |
| http://www.cs.yale.edu/~yry/        |
 =====================================

---

From: Xiao SHI <[email-masked]>
Date: Fri, 24 Oct 2014 00:50:30 -0400

On Fri, Oct 24, 2014 at 12:46 AM, Y. Richard Yang <[email-masked]> wrote:

>> The advantage of doing so is that the server does not need to keep any
>> old version of the resources. The full map the server first sends to the
>> client will always be updated, and the updates come when they become
>> available.
>>
[sx] not necessarily. say SSE updates cost map C which depends on network
map M, when a client requests update for cost map C, the SSE would initiate
the stream with full map M and C, then follow up with updates on C when
they become available.