Subject: [Jmap] new JMAP server for prototyping

From: Jamey Sharp <[email-masked]>
Date: Fri, 14 May 2021 11:32:42 -0700

Hey all: for the past few weeks I've been working on a new JMAP server with
the goal of making it easy to prototype new data models. I'd love it if
folks could take a look and maybe help implement or at least validate it.

https://github.com/jameysharp/jterritory

I went down this rabbit-hole after trying to figure out how to prototype a
feed-reader data model based on JMAP. I previously wrote up rationale for
that here:

https://jamey.thesharps.us/2020/08/06/feed-reader-cache-coherency/

Implementing JMAP correctly is pretty complicated, and while I do think
it's all necessary complexity, that makes it difficult to explore new
application domains. The core idea for this project is to not worry too
much about performance, but allow a developer to specify just a schema for
each data type and optionally some filter and sort extensions, and get the
standard methods for free.

I've been confused by a few things in RFC8620. I was only going to give a
couple examples but ended up with a complete list of everything I remember
having trouble with, so I hope it works out to just dump it here. I have
some more notes afterward on my current implementation and plans.

- Should creating multiple objects allow circular references, or must it be
a DAG of new objects? The former seems hard to get right if any of the
creations fail.

- Can result references appear in nested objects within a method's
arguments, or only on the arguments directly within the top-level
Invocation?

- Does the client need to ensure that it never uses a creation ID of "foo"
if there's some random string whose value happens to be "#foo", or does
/set need to know which properties have Id type?

- If the client uses the same creation ID in two method calls, despite the
"SHOULD" cautioning against it, and the second create fails, should the
first one's ID continue to be used? Is "the most recently created item with
that id" the most recent attempt, or the most recent success?

- I think I get it now, but I've been confused about how the responses from
/copy relate to its three phases. There's one /copy response covering
phases 1 and 2, then optionally a single /set response for phase 3, right?

- What value should the position attribute in a /query response have if the
requested position is past the end of the results?

- If a client has a sparse array of query results and gets a /queryChanges
response indicating that it should delete an object that it didn't have in
the array, then I don't see how it can tell whether to shift down any of
the objects it does have: the deleted object could have been anywhere. This
interface only seems safe so long as clients either always keep a complete
prefix of a query's results, or invalidate their cache if told to delete an
object they didn't know about.

- What does property immutability have to do with query result changes?
Isn't the only thing that matters whether the property actually changed,
rather than whether it could change? Is it just that some reasonable
implementations can't compute certain changes for mutable properties?

- The specification for upToId seems to imply to look up the position of
the given ID in the new results, though that isn't entirely clear to me.
But I think it makes more sense to find its position in the old results,
and send changes that update the client's cache to the same length. That
means the optimization still works even if the selected object is no longer
in the results, and puts an upper bound on how much data the client might
have to deal with: twice the number of items it has cached already.

But overall I've found the spec pretty clear and solid so far.

In the rest of this note I offer some more details on my current status and
approach.

I'm taking the simplifying assumption that there are no dynamically
computed properties; a call to /set has to update any related objects at
the same time. That lets me keep all objects in a single database table,
storing simple change-sequence numbers alongside raw JSON, instead of the
complexities recommended for email implementors. There are performance
costs to this simplicity, but that's a deliberate trade-off I'm making.

At this point I have the semantics for get, changes, set, and most of query
implemented, maybe even correctly. I've also implemented result references
and Core/echo, which go shockingly well together for testing. I haven't
hooked anything up to HTTP yet, just writing tests that pass JSON in to the
underlying functions.

Currently all my tests are property-based randomized testing. I'd like to
think my tests might be interesting for other server implementors. If there
are test suites I might be able to reuse, I'd love to hear about them.

My observation about implementing queryChanges is that you only need to be
able to report changes against queries which have previously actually been
run, so caching returned query results is enough to do the job generically,
without understanding the semantics of the data model or filters. Of
course, caching also helps with the fact that without that understanding, I
also can't provide indexes to make filters perform well. Refreshing the
cache can be as fast as calling /changes, as long as the cache includes all
the selected sort keys for each object. Also, I think I can enable
prototypes built on this framework to specify custom indexes, presumably
after the developer has gotten some experience with which filters are most
frequently used.

I've yet to tackle the session state, blobs, or push, but those don't seem
like they'll have hard questions about semantics, just some possibly tricky
engineering choices. We'll see how that goes.

Thanks for a nice spec and for getting to the end of this message. 😅
Jamey

---

From: "Neil Jenkins" <[email-masked]>
Date: Wed, 19 May 2021 13:05:48 +1000

Hi Jamey,

> Hey all: for the past few weeks I've been working on a new JMAP server with the goal of making it easy to prototype new data models.

That's awesome! Nice work.

> I've been confused by a few things in RFC8620. I was only going to give a couple examples but ended up with a complete list of everything I remember having trouble with, so I hope it works out to just dump it here. I have some more notes afterward on my current implementation and plans.
> 
> - Should creating multiple objects allow circular references, or must it be a DAG of new objects? The former seems hard to get right if any of the creations fail.

The relevant bit of the spec here is:

   The final state MUST be valid after the "Foo/set" is finished;
   however, the server may have to transition through invalid
   intermediate states (not exposed to the client) while processing the
   individual create/update/destroy requests.  For example, suppose
   there is a "name" property that must be unique.  A single method call
   could rename an object A => B and simultaneously rename another
   object B => A.  If the final state is valid, this is allowed.
   Otherwise, each creation, modification, or destruction of an object
   should be processed sequentially and accepted/rejected based on the
   current server state.

I don't think we actually considered a data type that supported circular references, but the way the current spec is written I would say that if the circular reference is valid for that data type and the final state is valid, then yes you would be allowed to do that in a single /set. But I agree that's a bit ambiguous, and I think you could probably forbid it in the /set description for the particular data type. Do you have an example in mind?

> - Can result references appear in nested objects within a method's arguments, or only on the arguments directly within the top-level Invocation?

Only on the top-level. The *arguments object* is the top-level object passed in the *Invocation*. So this:

   When processing a method call, the server MUST first check
   the arguments object for any names beginning with "#".

means just the top-level object.

> - Does the client need to ensure that it never uses a creation ID of "foo" if there's some random string whose value happens to be "#foo", or does /set need to know which properties have Id type?

When processing a /set you need to know which properties have Id type (well you need to know all the types really in order to be able to validate the data).

> - If the client uses the same creation ID in two method calls, despite the "SHOULD" cautioning against it, and the second create fails, should the first one's ID continue to be used? Is "the most recently created item with that id" the most recent attempt, or the most recent success?

Most recent success was the intention, but looking at it again I can see how you could interpret it both ways.

> - I think I get it now, but I've been confused about how the responses from /copy relate to its three phases. There's one /copy response covering phases 1 and 2, then optionally a single /set response for phase 3, right?

That's right.

> - What value should the position attribute in a /query response have if the requested position is past the end of the results?

The definition of "position" in the response is:

      The zero-based index of the first result in the "ids" array within
      the complete list of query results.

So I agree it's undefined if there are no results (but I also think it's unimportant, because there are no results to position!). I would just return the requested position I think. It would probably be worth making this well-defined in a future revision.

> - If a client has a sparse array of query results and gets a /queryChanges response indicating that it should delete an object that it didn't have in the array, then I don't see how it can tell whether to shift down any of the objects it does have: the deleted object could have been anywhere. This interface only seems safe so long as clients either always keep a complete prefix of a query's results, or invalidate their cache if told to delete an object they didn't know about.

Close. In this situation you have to invalidate any results in your sparse query after the first gap, but you can still keep the ones before. So if I have:

[A, B, -, -, D, -, -]

and queryChanges tells me that E has been removed, I would have to invalidate after the first gap, so my cached query would now be:

[A, B, -, -. -. -. -]

> - What does property immutability have to do with query result changes? Isn't the only thing that matters whether the property actually changed, rather than whether it could change? Is it just that some reasonable implementations can't compute certain changes for mutable properties?

Yes, it's that the spec was written with the idea that /queryChanges would be implemented based on knowing which records have changed from last time, not based on having the old query cached and comparing it with the new one. (This approach is hard to scale effectively.) If you only know that a record has changed, not which property on that record then you don't know what position it had in the client's current query cache (unless you have cached the old query, as you are doing), so have to tell the client to remove it an readd it to ensure it's correct.

If you can return precise changes with your implementation, that's great! This is really about guidance for what to do when you can calculate usable, if slightly less concise, changes efficiently. The text could definitely be better here, it's written a bit too much with an implementation approach in mind (in general we tried to not do that, as there are multiple good ways of implementing various bits of JMAP).

> - The specification for upToId seems to imply to look up the position of the given ID in the new results, though that isn't entirely clear to me. 

Yes, in the new results.

> But I think it makes more sense to find its position in the old results, and send changes that update the client's cache to the same length. That means the optimization still works even if the selected object is no longer in the results, and puts an upper bound on how much data the client might have to deal with: twice the number of items it has cached already.

I'm not quite sure what you mean by "to the same length", but yes if you have the old query cached you could optimise the case where the upToId is not in the new results a bit further. But again, the spec is written to allow implementers to calculate query changes *without* caching the old query state.

> But overall I've found the spec pretty clear and solid so far.

Great! This was really good feedback. I don't think any of it requires an errata (although if someone on the list disagrees, please pipe up!), but if we publish an updated RFC at some point in the future we know which points to clarify.

> Currently all my tests are property-based randomized testing. I'd like to think my tests might be interesting for other server implementors. If there are test suites I might be able to reuse, I'd love to hear about them.

Getting a good test suite for the JMAP spec is something we would love to have (and know from past experience is important for interoperability). I don't think anyone has published an open reusable one yet.

> I've yet to tackle the session state, blobs, or push, but those don't seem like they'll have hard questions about semantics, just some possibly tricky engineering choices. We'll see how that goes.

I think queryChanges is the trickiest bit to get your head around semantically so hopefully now that's done the rest is not too hard, but do keep us informed of your progress and any other spec issues you find!

Cheers,
Neil.

---

From: "Bron Gondwana" <[email-masked]>
Date: Wed, 19 May 2021 13:56:19 +1000

> I don't think we actually considered a data type that supported circular references, but the way the current spec is written I would say that if the circular reference is valid for that data type a…

I can think of one - anything double-linked - for example calendar events that have a rel=next and rel=prev relationship.

[ "Foo/set", {
  "a" : { next: "#b" }
  "b" : { prev: "#a" }
}, "x" ]

> Most recent success was the intention, but looking at it again I can see how you could interpret it both ways.

If you break it, you get to keep both pieces!

> Great! This was really good feedback. I don't think any of it requires an errata (although if someone on the list disagrees, please pipe up!), but if we publish an updated RFC at some point in the…

We should track these somewhere specific, because I for sure won't know which points to clarify in 5 years' time, and I'm unlikely to read every mail thread!  Maybe as open issues on the git repo, or even in a text file in the repo?

Cheers,

Bron.

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]

---

From: Jamey Sharp <[email-masked]>
Date: Fri, 21 May 2021 13:20:12 -0700

Thanks for answering my questions! I've snipped parts where your answer 
fully resolved my confusion, which was most of them really…

On Wed, May 19, 2021 at 01:05:48PM +1000, Neil Jenkins wrote:
>   > - Should creating multiple objects allow circular references, or 
>   > must it be a DAG of new objects? The former seems hard to get 
>   > right if any of the creations fail.
>
>    individual create/update/destroy requests. [...]
>
>   I don't think we actually considered a data type that supported 
>   circular references, but the way the current spec is written I would 
>   say that if the circular reference is valid for that data type and 
>   the final state is valid, then yes you would be allowed to do that 
>   in a single /set. But I agree that's a bit ambiguous, and I think 
>   you could probably forbid it in the /set description for the 
>   particular data type. Do you have an example in mind?

I don't have an example; I was actually hoping you'd say the intent was 
to forbid it. Since I'm writing an implementation which is agnostic to 
the particular data type, I need to figure out if this is a case I 
should allow. I haven't tested this case, but I think my current 
implementation fails all the creates in a cycle with invalidProperties.

Looking at a different bit of the text:

    In the case of records with references to the same type, the server 
    MUST order the creates and updates within a single method call so 
    that creates happen before their creation ids are referenced by 
    another create/update/destroy in the same call.

I think this wording is a little strange because I don't think destroys 
can reference creation IDs at all, right?

But more importantly: I think a strict reading would prohibit circular 
references in creates, since there is no ordering satisfying this MUST 
in that case.

Even under that reading, it only says that a create happens before 
"another" create which references it, but doesn't say anything about a 
create referencing itself, so that's ambiguous too I think.

>   > - Does the client need to ensure that it never uses a creation ID 
>   > of "foo" if there's some random string whose value happens to be
>   > "#foo", or does /set need to know which properties have Id type?
>
>   When processing a /set you need to know which properties have Id 
>   type (well you need to know all the types really in order to be able 
>   to validate the data).

Okay, sure, that's true. It's just that I wanted to substitute creation 
Ids first, without considering property types, and only validate the 
data afterward.

It seems easy enough for a client to use random creation Ids which are 
sufficiently large that they never collide with each other or with 
user-provided data. So if it's intended that clients may use strings 
which match their own creation Ids, I might just decide this 
implementation doesn't interoperate with clients that do that, which 
seems okay in a prototyping setting.

>   > - If the client uses the same creation ID in two method calls, 
>   > despite the "SHOULD" cautioning against it, and the second create 
>   > fails, should the first one's ID continue to be used? Is "the most 
>   > recently created item with that id" the most recent attempt, or 
>   > the most recent success?
>
>   Most recent success was the intention, but looking at it again I can 
>   see how you could interpret it both ways.

>   If you break it, you get to keep both pieces!

Hah, yeah. "Most recent success" means a client which violates this 
"SHOULD" can't predict which object it's going to be referencing, and 
the target could even be of a different type than expected. But after 
all, you did tell them they shouldn't do that…

I guess I feel like that SHOULD would be better as a MUST. Getting 
unexpected results, but only when an error occurs in a different 
operation, sounds like a surefire way to suffering and support calls.

On Wed, May 19, 2021 at 01:05:48PM +1000, Neil Jenkins wrote:
>   > - What value should the position attribute in a /query response 
>   > have if the requested position is past the end of the results?
>
>   So I agree it's undefined if there are no results (but I also think 
>   it's unimportant, because there are no results to position!). I 
>   would just return the requested position I think. It would probably 
>   be worth making this well-defined in a future revision.

I agree it shouldn't matter, but I think it might be helpful to define 
it any of these equivalent ways:

- the position immediately after the last item in the list
- the length of the list
- the minimum value of position for which you'd get the same response

A client can use calculateTotal to request that information, or negative 
position to implicitly reference it. But if they've gotten into a state 
where they thought there'd be more results than there really are, maybe 
it'd be nice to just tell them?

>   > - If a client has a sparse array of query results and gets a 
>   > /queryChanges response indicating that it should delete an object 
>   > that it didn't have in the array, then I don't see how it can tell 
>   > whether to shift down any of the objects it does have: the deleted 
>   > object could have been anywhere. [...]
>
>   Close. In this situation you have to invalidate any results in your 
>   sparse query after the first gap, but you can still keep the ones 
>   before.

Ohhhh. That makes sense, thanks. The spec is wrong here though, I think:

    The result of this is that if the client has a cached sparse array of
    Foo ids corresponding to the results in the old state, then:

    fooIds = [ "id1", "id2", null, null, "id3", "id4", null, null, null ]

    If it *splices out* all ids in the removed array that it has in its
    cached results, then:

       removed = [ "id2", "id31", ... ];
       fooIds => [ "id1", null, null, "id3", "id4", null, null, null ]

    and *splices in* (one by one in order, starting with the lowest
    index) all of the ids in the added array:

   added = [{ id: "id5", index: 0, ... }];
   fooIds => [ "id5", "id1", null, null, "id3", "id4", null, null, null ]

Here, "id31" is shown as removed, which wasn't in fooIds, but "id3" and 
"id4" come after a gap and weren't removed.

>   [...] the spec was written with the idea that /queryChanges would be 
>   implemented based on knowing which records have changed from last 
>   time, not based on having the old query cached and comparing it with 
>   the new one. (This approach is hard to scale effectively.)

Hah, definitely. Fortunately, I specifically don't care about 
scalability.

There are some neat tricks to play with storing deltas, and updating 
recent queries during /set, which I think might make it actually scale. 
But I'm not bothering to explore them, at least for now.

>   I'm not quite sure what you mean by "to the same length", but yes if 
>   you have the old query cached you could optimise the case where the 
>   upToId is not in the new results a bit further. But again, the spec 
>   is written to allow implementers to calculate query changes without 
>   caching the old query state.

I meant that if the client has cached the first 30 items, then upToId 
refers to the 30th item they knew about; a delta which gets them back to 
30 cached items seems like a reasonable thing to want.

I guess my feedback on this point is that it's another area that seems a 
little more focused on particular implementations than necessary. Which 
doesn't mean it needs to change, just that it's still on the list of 
things I'm confused about.

I can't quite figure out what client UX the upToId feature supports. I 
can imagine trying to keep the ten threads starting at #23 visible on 
the screen, or keep the thread list centered around the thread which is 
currently being displayed, but there's no reason to expect this will let 
me do either of those things, right?

Maybe if I understood the client side story better this would make more 
sense to me.

>   Great! This was really good feedback.

I'm glad to hear it!

>   > Currently all my tests are property-based randomized testing. I'd 
>   > like to think my tests might be interesting for other server 
>   > implementors. If there are test suites I might be able to reuse, 
>   > I'd love to hear about them.
>
>   Getting a good test suite for the JMAP spec is something we would 
>   love to have (and know from past experience is important for 
>   interoperability). I don't think anyone has published an open 
>   reusable one yet.

There are a few tests which only rely on core JMAP features that I think 
would make a good starting point. In particular, I really like that 
result references can be fully tested using two (or more) Core/echo 
calls in the same request.

>   > I've yet to tackle the session state, blobs, or push, but those 
>   > don't seem like they'll have hard questions about semantics, just 
>   > some possibly tricky engineering choices. We'll see how that goes.
>
>   I think queryChanges is the trickiest bit to get your head around 
>   semantically so hopefully now that's done the rest is not too hard, 
>   but do keep us informed of your progress and any other spec issues 
>   you find!

Will do. Thank you!
Jamey

---

From: "Neil Jenkins" <[email-masked]>
Date: Mon, 24 May 2021 14:59:02 +1000

> can reference creation IDs at all, right?

Technically it could destroy a creation in the same request (it's a weird edge case, but it was worded that way deliberately); I know we actually came across this scenario at some point, although I can't remember why just now.

> references in creates, since there is no ordering satisfying this MUST 

Yes, I would read that the same; I think this paragraph does implicitly rule out creating circular references directly.

> seems okay in a prototyping setting.

Hmm, I mean, sure you can do whatever you want for prototyping, but this is definitely not spec compliant. (My client, for example, just uses an incrementing integer for creation ids, which avoids reuse as per spec but is definitely not guaranteed to not appear by coincidence in other properties).

> "Most recent success" means a client which violates this 

It's a SHOULD because the server behaviour is (in theory) well defined, and in theory you could do something crazy (e.g. Try creating three different items then set a reference to the last one that succeeded…).

> >   The definition of "position" in the response is
> 

Sure, I think this value is a reasonable choice. It does mean there are a few optimisations you can't make though (because it's essentially the same as forcing the server to calculate the total), although they are unlikely to be important ones I think.

> "id4" come after a gap and weren't removed.

Hey damn, you're right! I think my brain glossed over this when proofreading because the line below starts **and truncates**, and in my code the flag that gets set when you have an unknown destroyed id is called `truncateAtFirstGap`. But the spec is actually just description the final step of adjusting the query results length, not removing the ids after the gap.

Would you like to submit the errata? Or I'm happy to do it.

> […]

It's purely an optimisation in both what's sent over the wire and (for certain implementations) how much work the server does. In most scenarios, a client will have just the start of a list cached. Extreme scenario, we have the first 3 ids cached from a list of 500 results:

[ id1, id2, id3, null4 … null500 ]

The goal of queryChanges is to ensure that the state of the client after applying the changes is an accurate representation of the new state on the server, while sending a minimal delta. Suppose the last 100 items are removed, and a further 50 items are inserted at the end instead. Simply truncating the client's sparse array to the new length (450) means it is now a valid representation of the new state: every id that's there is in the correct position. This is what upToId does: instead of sending 50 "added" and 100 "removed" ids you can just send the new total instead.

Cheers,
Neil.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 24 May 2021 17:54:42 +0200

On Monday 24 May 2021 06:59:02 CEST, Neil Jenkins wrote:
> Technically it could destroy a creation in the same request 
> (it's a weird edge case, but it was worded that way 
> deliberately); I know we actually came across this scenario at 
> some point, although I can't remember why just now.

It seems like a reasonable enough combination when a mobile device leaves 
airplane mode and a mobile client is playing back stored actions.

Arnt

---

From: Jamey Sharp <[email-masked]>
Date: Wed, 26 May 2021 15:28:36 -0700

On Mon, May 24, 2021 at 02:59:02PM +1000, Neil Jenkins wrote:
>>I think this wording is a little strange because I don't think 
>>destroys can reference creation IDs at all, right?
>
>Technically it could destroy a creation in the same request (it's a 
>weird edge case, but it was worded that way deliberately); I know we 
>actually came across this scenario at some point, although I can't 
>remember why just now.

On Mon, May 24, 2021 at 05:54:42PM +0200, Arnt Gulbrandsen wrote:
>It seems like a reasonable enough combination when a mobile device 
>leaves airplane mode and a mobile client is playing back stored 
>actions.

I suppose it's not unreasonable, but I can't find any hint in the 
specification that the keys of `update` or the entries of `destroy` can 
be creation id references. The most relevant text I can find is this:

    Some records may hold references to other records (foreign keys). 
    That reference may be set (via create or update) in the same request 
    as the referenced record is created.  To do this, the client refers 
    to the new record using its creation id prefixed with a "#".

My reading is that creation id references can only appear within a 
record, either inside the `Foo` in `create`, or inside the `PatchObject` 
in `update`. Am I missing something?

On Mon, May 24, 2021 at 02:59:02PM +1000, Neil Jenkins wrote:
>Hmm, I mean, sure you can do whatever you want for prototyping, but 
>this is definitely not spec compliant.

Yeah, fair enough; I'll think about how to make my validation step 
context-sensitive.

>>[...]
>Hey damn, you're right! I think my brain glossed over this when 
>proofreading because the line below starts and truncates, and in my 
>code the flag that gets set when you have an unknown destroyed id is  
>called truncateAtFirstGap. But the spec is actually just description 
>the final step of adjusting the query results length, not removing the 
>ids after the gap. Would you like to submit the errata? Or I'm happy to 
>do it.

As much as I enjoy the idea of having my name on a contribution to IETF 
work for the first time, I'm not sure how to word the replacement text, 
so I think probably you should do it.

>>I can't quite figure out what client UX the upToId feature supports.
>
>It's purely an optimisation in both what's sent over the wire and (for 
>certain implementations) how much work the server does. In most 
>scenarios, a client will have just the start of a list cached. Extreme 
>scenario, we have the first 3 ids cached from a list of 500 results:
>
>The goal of queryChanges is to ensure that the state of the client 
>after applying the changes is an accurate representation of the new 
>state on the server, while sending a minimal delta. Suppose the last 
>100 items are removed, and a further 50 items are inserted at the end 
>instead. Simply truncating the client's sparse array to the new length 
>(450) means it is now a valid representation of the new state: every id 
>that's there is in the correct position. This is what upToId does: 
>instead of sending 50 "added" and 100 "removed" ids you can just send 
>the new total instead.

That's a great example for me to keep in mind but it didn't make me less 
confused. I've figured out what's bothering me though.

Let's say the client has cached a prefix of the results, and then some 
of those items are destroyed.

old: ["A", "B", "C", "D", ...]
new: ["B", "D", ...]

Say upToId is "D"; you said I should look for it in the new result, 
where it is at index 1. Then 'any ids that were removed but have a 
higher index than "upToId" SHOULD be omitted.' Which means the removed 
list is permitted to contain only "A", right?

If so, now the client's cache says ["B", "C", "D"], which is wrong.

I imagine current implementations just report all destroyed records as 
removed whether they were in the old query or not, which would obscure 
this issue. It would also come up for sorts or filters on mutable 
properties, but as we've discussed, nobody does that right now.

What my intuition was prodding me about is that using the same index in 
the text of both "added" and "removed" can't be right; which led to me 
trying to guess client-side UX reasons why somebody might want that, 
instead of the minimum delta needed for correct operation.

Here's what I think are the minimum necessary rules for upToId; I'd 
appreciate any comments on whether I've got this right:

     If an "upToId" is supplied and existed in the old results, any ids 
     that were removed but had a higher index than "upToId" previously 
     did SHOULD be omitted. All others MUST NOT be omitted.

     If an "upToId" is supplied and existed in the old results, the last 
     id which was not removed, and which previously had an index no 
     higher than "upToId" did, is the client's last cached id. Any ids 
     which were added but now have a higher index than the client's last 
     cached id does SHOULD be omitted. All others MUST NOT be omitted.

This text might be more clear if written from the point of view of the 
client's requirements rather than restrictions on the server:

     If an "upToId" is supplied, any ids which the server is certain did 
     not appear at or before "upToId" in the old results SHOULD be 
     omitted.

     If an "upToId" is supplied, any ids which are not necessary to 
     return the client's cache to a valid prefix of the new results, 
     after removing the ids given in "removed", SHOULD be omitted.

I think any mention of sort/filter mutability here is an unnecessary 
constraint on server behavior, as is requiring upToId to be present in 
the new results.

But I believe these rules still allow every implementation the current 
spec allows, aside from prohibiting the situation I described above, 
since a server is always allowed to ignore upToId.

I hope that made sense this time...
Jamey

---

From: "Seph Gentle" <[email-masked]>
Date: Thu, 27 May 2021 00:19:02 +0000

On Fri, May 14, 2021, at 6:32 PM, Jamey Sharp wrote:
> Hey all: for the past few weeks I've been working on a new JMAP server with the goal of making it easy to prototype new data models. I'd love it if folks could take a look and maybe help implement or at least validate it.
> 

This is really cool. I wanted to make something like this for email a couple years ago, and got stuck parsing email in a jmap-kind of way. I ended up making this:

https://github.com/josephg/mime-to-jmap

This is the email envelope -> JMAP code path from cyrus, pulled out into a stand-alone library and compiled to wasm so it can be used in lots of contexts. It might take some work, but you could probably make a simple jmap email server if you want to by combining our efforts.

-Seph

---

From: "Neil Jenkins" <[email-masked]>
Date: Fri, 28 May 2021 14:47:14 +1000

> in `update`. Am I missing something?

Hmm, yes, it looks like we never made it explicit in the spec. It's a rare edge case, but possibly worth an errata too to be explicit.

> list is permitted to contain only "A", right?

That was not the intention, but I agree the spec is misleading here, to the point I think it is worth doing an errata; I'll do that too.

>      after removing the ids given in "removed", SHOULD be omitted.

That's absolutely correct, and was the intention here.

(The description in the spec is kind-of true, but only because it's grounded in the idea of a particular implementation where you have "tombstone" records for the deleted items that you can sort into the new query, so it's the index of *those* that are lower than the upToId. It's not that important, the spec is wrong and we need an errata, but I know why we thought it made sense when it was written!)

> constraint on server behavior, as is requiring upToId to be present in 

I agree. That's particular implementation details leaking into the spec again.

Thanks again for the great feedback! As an Area Director put it to me, if there's no erratas it doesn't mean a spec is perfect, it just means no one's reading it. 🙂

Neil.