Subject: Re: [netconf] restconf collections

From: Qin Wu <[email-masked]>
Date: Wed, 30 Sep 2020 01:27:43 +0000

发件人: netconf [mailto:[email-masked]] 代表 Kent Watsen
发送时间: 2020年9月30日 5:57
收件人: Martin Björklund <[email-masked]>
抄送: [email-masked]
主题: Re: [netconf] restconf collections

Hi Martin,

I don't think d) and e) should be restricted to obu lists, and I don't
think they should be restricted to "indexable" columns.

And I think "expression of some sort" should be XPath.

Also, since the main objective is efficient retrieval, "sort-by"
can perhaps be removed.  Consider large lists in operational state
that also change often.

Your last paragraph, if I read it correctly, says “maybe don’t do ‘d’”, which if applied to the first part of your response becomes "I don't think 'e' should be restricted to OBU-lists”, which is something I can agree with.  As for ‘d’ being a reasonable thing to support for OBU-lists, please explain the use-case.

Regarding “I don’t think they should be restricted to ‘indexable’ columns”, we need to keep in mind that, if the DB-backend can’t do it, then the server logic will need to it, likely by retrieving all records and brute-forcing its way through them, which would not only be painfully slow, but a lot of implementation complexity.  That said, I’m unsure what use-case you have in mind whereby a non-indexable column is being filtered/sorted.  Only a couple of the built-in types aren’t indexable, right?

Yes, XPath is the “right answer”, but we need to ensure it’s constrained enough to be mappable to common DB-backends.

Updating my previous response (per above, plus adding leaf-list, plus locking-in names, which I’m choosing only to NOT sound like SQL):

    For all lists and leaf-lists:

       a) count      (uint, default: 0)
       b) skip       (uint, default: 0)
       c) direction  (enum: forwards/backwards, default: forwards)

    For non “ordered-by user” lists only (N/A to leaf-lists), but
    only for indexable columns:

       d) sort-by    (string: a single column/leaf name)

    For all lists and leaf-lists, but only for indexable columns (note: for
    a leaf-list, it is its own indexable column):
       e) filter     (a constrained Xpath expression)

    Again: sequence of operation is e —> a.

We haven’t discussed “feature” statements yet, but it seems reasonable to me for both ‘d’ and ‘e’ to be optionally implemented.  Additionally, it may be that additional Xpath expressions can be unlocked by feature statements).

[Qin] Umm, this seems a good choice. I feel sort-by is nice to have feature, in many cases, very useful. But we need to when sort-by is required and when is not.

We could define a few more XPath functions for such comparisons.

Sure, but I wonder if, e.g., a netmask filter, is supportable by common DB-backends.  I’m hoping we have some DB-experts on the list!

K.

---

From: Qin Wu <[email-masked]>
Date: Wed, 30 Sep 2020 10:33:27 +0000

-----邮件原件-----
发件人: Robert Varga [mailto:[email-masked]] 
发送时间: 2020年9月30日 16:39
收件人: Qin Wu <[email-masked]>; Kent Watsen <[email-masked]>; Andy Bierman <[email-masked]>
抄送: [email-masked]
主题: Re: [netconf] restconf collections

On 30/09/2020 03:11, Qin Wu wrote:
> Hi, Robert:

Hellow Qin,

> -----邮件原件-----
> 发件人: netconf [mailto:[email-masked]] 代表 Robert Varga
> 发送时间: 2020年9月30日 7:05
> 收件人: Kent Watsen <[email-masked]>; Andy Bierman 
> <[email-masked]>
> 抄送: [email-masked]
> On 26/09/2020 20:57, Kent Watsen wrote:
>>> There are 3 completely different approaches:
>>>
>>>   1) stateless
>>>   2) stateful
>>>   3) stateful snapshot
>>>
>>> How to efficiently iterate through data that is changing constantly?
>>> This problem is harder than it looks.
>>
>> Agreed.  I recommend we NOT try to solve that problem.
> 
> Unfortunately you do need to pick an interaction model to tie together the first and second pagination requests. At the very least you need to specify consistency guarantees between the two -- which gives them worms juuust enough room to wiggle out of the can.
> 
> [Qin]:I think when the client Using page request to get access to the 
> element in the list, he only need to remember the position of last element in the first pagination request and set the number of element he want to request in the pagination requests, The server doesn't need to keep any additional states, the server only need to support list operation.

That is assuming the list in question has iteration order which is stable. I do not believe this is something RFC7950 mandates on anything except ordered-by=user.

> A fair answer is 'well, they are not tied together and if the 
> datastore moves, you are SOL' ... but then how do you make forward 
> progress with enough consistency to be useful in the real world? :)
> 
> 
> [Qin]: I think it doesn't matter whether the list will grow and add new elements, we could still use last element position and count of elements in the list that need to be queried to retrieve additional elements we want.

So let's say the list has 4 elements A, B, C, D.

1) The first request returns items A, B, C.
2) An item is inserted before C, so that the list is E, A, B, C, D.
3) What does the second request return?
Similarly:

1) The first request returns items A, B, C.
2) An item is deleted before C, so that the list is B, C, D.
3) What does the second request return?

Pagination is deceptively hard.
[Qin]: maybe using position of the last element is not accurate, the client only needs to remember the last element he request in the first pagination request, and use the last element as cursor reference point and set count of elements he want in the second pagination request, so in both cases, the second request will return D.
But if C is deleted in the extreme case, I think the server should return error to indicate the C element as last element doesn't exist.

Regards,
Robert

---

From: Qin Wu <[email-masked]>
Date: Thu, 1 Oct 2020 04:33:15 +0000

>> Yes, XPath is the “right answer”, but we need to ensure it’s 
>> constrained enough to be mappable to common DB-backends.
> 
> So the client talks to the server, and the server talks to the "db"
> backend (note that in many implementations  operational state is 
> typically not stored in a real database).

Some opstate must be persisted, e.g., long-lived counters, logs, etc., but it’s a good point about other opstate not being persisted.   Perhaps “node-tags” can be used here, to differentiate which is which…and servers can indicate if/how they support the ephemeral opstate leafs in queries?

[Qin]:That's a good case for node tag, in earlier discussion, we discussed operation type, which distinguishs cumulative statistics value from current value. The case discussed here is very close to operation type proposal discussed earlier.

> So we have:
> 
>  client ->  server ->  backend
> 
> If we don't have any filter capabilities, the client has to get 
> everything, and then filter.  If we support XPath and the backend 
> doesn't support it, the server will have to filter.  This is still 
> more efficient than filtering in the client.

Probably, especially when assuming the server has better resources and/or the client <--> server bandwidth is constrained.

> In ConfD/NSO we moved more and more filtering logic towards the 
> backend, to speed up performance.

Ack.

> I think e) is way more important than c).  I suggest focus on a + b + 
> c.

Noted, but I think you meant a + b + e.

Note sure how others feel about “direction: (c), but my primary use-case revolves around time-series data (e.g., logs), where the interest is commonly on the most-recent entries, so "reverse-->offset—>limit” works nicely.  

Perhaps an alternative would be to lift a concept from Python with negative indexes so, for instance, offset=-N and limit=-N gives the last N entries?
[Qin]: Yes, that's what I thought as well, with negative indexes, (b) and (c) seems to me, can be combined.

>> Sure, but I wonder if, e.g., a netmask filter, is supportable by 
>> common DB-backends.  I’m hoping we have some DB-experts on the list!
> 
> See above.  It can be quite efficient even if the backend doesn't 
> support it.

I don’t see that above, but I don’t doubt that it can be so, it’s just a whole lot of implementation complexity.  It seems that we should/must support servers doing it, we just need to find a way (node-tags?) to enable them to express that ability.
[Qin]: My feeling is this efficiency more depends on the amount of data we need to request. If amount of data we request is huge, maybe, client-> server-> backend may be the better choice.

K.

_______________________________________________
netconf mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netconf