Subject: Re: [netconf] restconf collections

From: Qin Wu <[email-masked]>
Date: Sat, 26 Sep 2020 04:30:01 +0000

发件人: netconf [mailto:[email-masked]] 代表 Andy Bierman
发送时间: 2020年9月26日 1:22
收件人: Kent Watsen <[email-masked]>
抄送: [email-masked]
主题: Re: [netconf] restconf collections

On Fri, Sep 25, 2020 at 9:56 AM Kent Watsen <[email-masked]<mailto:[email-masked]>> wrote:
Hi Andy,

This has been discussed a few times by the WG.

Perhaps a testament to it being a real problem?

The co-authors never agreed on the set of bells and whistles that should be supported.

True, but it's more like we lost interest / time to work on it...at least that's my story!  ;)

There was an objection from Phil Shafer at the time that pagination was not needed
at all and handling large responses from the server was just a minor implementation detail.

Clearly Phil is in the rough, as many APIs support pagination for a reason.   Even if constraining the solution to networking gear, interfaces and logs may be numerous, and on-box cli/web UIs can only show a screen at a time...

Our customers agree with you.  Some of them only use our <get-bulk>, and stopped using <get> and <get-config>.
Since it is implemented as an RPC, all YANG-based protocols can use it.
Obviously you need XPath filtering (XSLT-style) to select the list entries of interest.
Count and depth and other parameters from <get> are also useful.

(My concern is that the IETF version will be too complex to implement or use as the feature list grows.)
[Qin]:That’s the issue we try to address in our customer’s scenarios. If the solution only focuses on list, the complexity is controllable. If the scope is not limited to list, XAPTH filtering and query actually provides a good solution to navigate from root node to leaf node.

IMO, it's okay if we don't support for CT-lists (i.e., interfaces).  As you write above, and I wrote in before, it seems like a nice-to-have.   FYI, I'm asking Per in a fork of this thread to justify if supporting CT-lists is a "must-have"...hopefully he'll come back with something compelling.

There is no extra complexity to support config vs. operational data.
I don't see the value in restricting the operation.

CF-lists, on the other hand, can be enormous...so much so that trying to return all could exhaust server and/or client resources.   I think supporting CF-lists is a must-have.

The most significant objection was that this is a RESTCONF-only solution and that
was not acceptable to ignore NETCONF (since <get> and GET end being the same
in the end, so the problem also exists for NETCONF)

True.  If we can find a way to extend support to NETCONF, all the better.   Of course, the solutions would be wildly different, much like how NMDA had separate solutions for NC and RC (RFCs 8526 and 8527, respectively).  Perhaps Per or Martin can clue us into if they have a NC-equivalent solution?

It is easy. Use RPC operations which map to RESTCONF POST automatically.

[Qin]: Agree, NETCONF fragment provide an example on what RPC operations look like
https://tools.ietf.org/html/draft-zheng-netconf-fragmentation-02
which is similar to getbulk operation.

However, if NC-support cannot be achieved, then I don't think it means we don't do it for RC.   Much like NC supports somethings RC doesn't (e.g., confirmed commit), each protocol plays into its strengths.   So mote it be.

Andy

K.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Sat, 26 Sep 2020 17:21:01 +0000

Hi Qin,

> [Qin]:That’s the issue we try to address in our customer’s scenarios. If the solution only focuses on list, the complexity is controllable. If the scope is not limited to list, XAPTH filtering and…

The functionality is only for lists.  The functionality (e.g., paging) being discussed only make sense for lists...

> which is similar to getbulk operation.

The <get-block> RPC may be a good starting point.  It would be good to also look at Andy’s <get-bulk> RPC for inspiration.

K.

---

From: Andy Bierman <[email-masked]>
Date: Sat, 26 Sep 2020 10:31:13 -0700

On Sat, Sep 26, 2020 at 10:21 AM Kent Watsen <[email-masked]> wrote:

> [Qin]:That’s the issue we try to address in our customer’s scenarios. If
> the solution only focuses on list, the complexity is controllable. If the
> scope is not limited to list, XAPTH filtering and query actually provides a
> good solution to navigate from root node to leaf node.
>
>
> The functionality is only for lists.  The functionality (e.g., paging)
> being discussed only make sense for lists...
>
>
> The <get-block> RPC may be a good starting point.  It would be good to
> also look at Andy’s <get-bulk> RPC for inspiration.
>
>
>
The <get-block> operation seems very stateful and seems to be tied to the
existing get-config operation

The <get-bulk> operation is not stateful.
 http://www.netconfcentral.org/modulereport/yumaworks-getbulk

There are 3 completely different approaches:

  1) stateless
  2) stateful
  3) stateful snapshot

How to efficiently iterate through data that is changing constantly?
This problem is harder than it looks.

K.
Andy

---

From: Kent Watsen <[email-masked]>
Date: Sat, 26 Sep 2020 18:57:51 +0000

>  http://www.netconfcentral.org/modulereport/yumaworks-getbulk <http://www.netconfcentral.org/modulereport/yumaworks-getbulk>
This RPC is effectively 1-1 with RC query params approach being discussed.  I like it.

The “count” param is like “limit”, and “list-test” is like a “where”.

One difference is that <get-bulk> uses "last-keys” instead of an integral “offset”.   Any thoughts about the pros and cons?  I imagine performance depends on the backend database used.  For instance, SQL supports “offset” so, to support "last-keys” instead would entail a first query to determine the offset for a key.

Missing is a “sort”.  That said, of the four params mentioned (limit, offset, where, sort), it might be least important.  At least, a general sort is…and then there is the nasty issue of sorting a user-sorted list.  A likely must-have fallback would be “direction” so, e.g., queries return the most-recent (not the oldest) entires in a time-series list (e.g., a log).  A “direction” parameter would also work well on a user-sorted list.

> This problem is harder than it looks.

Agreed.  I recommend we NOT try to solve that problem.

K.

---

From: Andy Bierman <[email-masked]>
Date: Sun, 27 Sep 2020 08:12:06 -0700

On Sat, Sep 26, 2020 at 11:57 AM Kent Watsen <[email-masked]> wrote:

> One difference is that <get-bulk> uses "last-keys” instead of an integral
> “offset”.   Any thoughts about the pros and cons?  I imagine performance
> depends on the backend database used.  For instance, SQL supports “offset”
> so, to support "last-keys” instead would entail a first query to determine
> the offset for a key.
>
>

It would be unlikely that a distributed server (e.g. data can come from
linecards)
that it is efficient to find the Nth entry.

Using a count approach is obviously vulnerable to additions and deletions
to the dataset.
If the count keeps changing then the relative Nth entry also keeps changing.
This causes duplicates or missed entries in the paginated data.  Using key
values
instead of a count this source of inaccuracy is eliminated.  Both approaches
are vulnerable to "moved" entries (e.g. user-ordered list entry changes
position).
This can happen in theory but (1) user-ordered config=false data is very
rare,
(2) the "move" operation for user-ordered data is very rarely used and (3)
an entry would have to be moved just at the time the walk was in progress.

> Missing is a “sort”.  That said, of the four params mentioned (limit,
> offset, where, sort), it might be least important.  At least, a general
> sort is…and then there is the nasty issue of sorting a user-sorted list.  A
> likely must-have fallback would be “direction” so, e.g., queries return the
> most-recent (not the oldest) entires in a time-series list (e.g., a log).
> A “direction” parameter would also work well on a user-sorted list.
>

sort is much more expensive to implement correctly and makes more
sense when a stateful snapshot approach is used. Sorting each chunk
vs. the entire data-set is not that useful.

> Agreed.  I recommend we NOT try to solve that problem.

But we are solving that problem to some degree.
Operational state is changing constantly and iterating through a list
vs. a single GET usually makes the problem worse.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Mon, 28 Sep 2020 14:21:47 +0000

It seems that we need two I-Ds:

1) “restconf-lists"

	a) “update” RFC 8040 (and possibly RFC 8527)
	b) modify Section 3.5 (Data Resource) to state that “list” and “leaf-list” themselves (not just their entries) are “data resources”
	c) define impact on various HTTP methods GET, POST, PUT, DELETE, PATCH, etc.
	d) additionally define query parameters for the GET:
		i) limit: number to return.  Could name ‘count’ to match <get-bulk>
		ii) offset: where to begin.  Need to discuss if <get-bilk>’s “keys” is better
		iii) where: some xpath expression
		iv) direction: ascending | descending

2) “netconf-lists"

	a) “update” RFC 6241 (and possibly RFC 8526)
	b) introduce a <get-bulk> or <get-block> like RPC for paging through the entries of lists and leaf-lists.

Anyone interested in working on said I-Ds, please email me privately.

Thanks,
Kent. // as a contributor

> On Sep 27, 2020, at 11:12 AM, Andy Bierman <[email-masked]> wrote:
> 
> 
> 
> On Sat, Sep 26, 2020 at 11:57 AM Kent Watsen <[email-masked] <mailto:[email-masked]>> wrote:
> 
> It would be unlikely that a distributed server (e.g. data can come from linecards)
> Using a count approach is obviously vulnerable to additions and deletions to the dataset.
> This causes duplicates or missed entries in the paginated data.  Using key values
> are vulnerable to "moved" entries (e.g. user-ordered list entry changes position).
> This can happen in theory but (1) user-ordered config=false data is very rare,

---

From: Kent Watsen <[email-masked]>
Date: Tue, 29 Sep 2020 17:44:06 +0000

Firstly, thank you to those that responded showing interest in working on these I-Ds.  If anyone else is interested, please send me a PM. 

Regarding solution, I want to take one more stab at a sketch here, as a last-chance for comments before the authors dive into producing the I-Ds.

Already it is assumed that there will be an RPC for NC, and query params for RC…that not what this email is about.

Here’s my proposal:

	For all lists;

		a) “limit” or “count”
		b) “offset" or “skip”
		c) “direction”  (i.e., forward/backward)

	For non “ordered-by user” lists, but only for indexable columns:

		d) “sort-by” or “ordered-by” (single column max)
		e) “where” or “filter” (compound expression of some sort)

	Note: the lowercase “or” is just for the label selected (no semantic impact).

The order of operation is “e —> a”.  That is, first the filter is applied (if any), then the sort is applied (if any), then the direction is applied (if any), then the offset/skip is applied (if any), and then the limit/count is applied (if any).

Now the tricky stuff.

	- “key” and “unique” leafs are indexed by default, assuming
	  the leaf’s base type is indexable
	- a special extension is used to index other nodes
	- other indexable nodes must be a “leaf"
	- the leaf must be “mandatory true”
	- the leaf must not be under a “list” or “choice” node
	- the leaf must have an indexable base type
		- all built-in types supported except:
			- binary
			- instance-identifier
		- unions are indexable, but assumes all values stored
		   as a “string” in the DB.
		- “string” types may be coerced into more DB-friendly types
			- “yang:date-and-time” mapped to a DB “DateTime” type

Cons:
	- smart (e.g., netmask-based) filter/sorts is unlikely supportable
		- at least not without custom (not generic) code

Comments?

K.

---

From: Martin Björklund <[email-masked]>
Date: Tue, 29 Sep 2020 20:57:10 +0200 (CEST)

Hi,

Kent Watsen <[email-masked]> wrote:
> 
> Firstly, thank you to those that responded showing interest in working
> on these I-Ds.  If anyone else is interested, please send me a PM.
> 
> Regarding solution, I want to take one more stab at a sketch here, as
> a last-chance for comments before the authors dive into producing the
> I-Ds.
> 
> Already it is assumed that there will be an RPC for NC, and query
> params for RC…that not what this email is about.
> 
> Here’s my proposal:
> 
> 	For all lists;
> 
> 		a) “limit” or “count”
> 		b) “offset" or “skip”

I don't think d) and e) should be restricted to obu lists, and I don't
think they should be restricted to "indexable" columns.

And I think "expression of some sort" should be XPath.

Also, since the main objective is efficient retrieval, "sort-by"
can perhaps be removed.  Consider large lists in operational state
that also change often.

> The order of operation is “e —> a”.  That is, first the filter is
> applied (if any), then the sort is applied (if any), then the
> direction is applied (if any), then the offset/skip is applied (if
> any), and then the limit/count is applied (if any).
> 
> Now the tricky stuff.
> 

We could define a few more XPath functions for such comparisons.

/martin

---

From: Kent Watsen <[email-masked]>
Date: Tue, 29 Sep 2020 21:56:39 +0000

Hi Martin,

> can perhaps be removed.  Consider large lists in operational state

Your last paragraph, if I read it correctly, says “maybe don’t do ‘d’”, which if applied to the first part of your response becomes "I don't think 'e' should be restricted to OBU-lists”, which is something I can agree with.  As for ‘d’ being a reasonable thing to support for OBU-lists, please explain the use-case.

Regarding “I don’t think they should be restricted to ‘indexable’ columns”, we need to keep in mind that, if the DB-backend can’t do it, then the server logic will need to it, likely by retrieving all records and brute-forcing its way through them, which would not only be painfully slow, but a lot of implementation complexity.  That said, I’m unsure what use-case you have in mind whereby a non-indexable column is being filtered/sorted.  Only a couple of the built-in types aren’t indexable, right?

Yes, XPath is the “right answer”, but we need to ensure it’s constrained enough to be mappable to common DB-backends.

Updating my previous response (per above, plus adding leaf-list, plus locking-in names, which I’m choosing only to NOT sound like SQL):

	For all lists and leaf-lists:

		a) count		(uint, default: 0)
		b) skip		(uint, default: 0)
		c) direction	(enum: forwards/backwards, default: forwards)

	For non “ordered-by user” lists only (N/A to leaf-lists), but
	only for indexable columns:

		d) sort-by		(string: a single column/leaf name)

	For all lists and leaf-lists, but only for indexable columns (note: for
	a leaf-list, it is its own indexable column):
		e) filter		(a constrained Xpath expression)

	Again: sequence of operation is e —> a.

We haven’t discussed “feature” statements yet, but it seems reasonable to me for both ‘d’ and ‘e’ to be optionally implemented.  Additionally, it may be that additional Xpath expressions can be unlocked by feature statements).

> We could define a few more XPath functions for such comparisons.

Sure, but I wonder if, e.g., a netmask filter, is supportable by common DB-backends.  I’m hoping we have some DB-experts on the list! 

K.

---

From: Robert Varga <[email-masked]>
Date: Wed, 30 Sep 2020 01:05:29 +0200

> Agreed.  I recommend we NOT try to solve that problem.

Unfortunately you do need to pick an interaction model to tie together
the first and second pagination requests. At the very least you need to
specify consistency guarantees between the two -- which gives them worms
juuust enough room to wiggle out of the can.

A fair answer is 'well, they are not tied together and if the datastore
moves, you are SOL' ... but then how do you make forward progress with
enough consistency to be useful in the real world? :)

At the end of the day, you will end up creating some state on the
server, which the server will need to be free to revoke at any time (due
to resource constraints, or whatever implementation-specific reasons).

Regards,
Robert

---

From: Robert Varga <[email-masked]>
Date: Wed, 30 Sep 2020 01:23:49 +0200

> can perhaps be removed.  Consider large lists in operational state

+1.

Getting hold of enough state to guarantee stable iteration order can be
done quite efficiently.

Sorting requires touching the data indices at the very least (to
understand that no sorting is needed), potentially a huge
CPU/MEM/network operation at the worst (a.k.a. "establish global order
of entities in a distributed system").

If the criteria is user-specified, the possibility of a DoS becomes very
real -- and hence the question becomes 'can the query be sufficiently
analyzed to reject it early and prevent overloading the system?' :)

Regards,
Robert

---

From: Martin Björklund <[email-masked]>
Date: Wed, 30 Sep 2020 08:43:12 +0200 (CEST)

Hi,

Kent Watsen <[email-masked]> wrote:
> Your last paragraph, if I read it correctly, says “maybe don’t do
> ‘d’”, which if applied to the first part of your response becomes "I
> don't think 'e' should be restricted to OBU-lists”, which is something
> I can agree with.  As for ‘d’ being a reasonable thing to support for
> OBU-lists, please explain the use-case.

If we agree to not do d) then we don't need any use cases for it ;-)

Anyway, if you want to sort on different fields you probably want to
do this regardless of how the list is orderd on the server.

> Regarding “I don’t think they should be restricted to ‘indexable’
> columns”, we need to keep in mind that, if the DB-backend can’t do it,
> then the server logic will need to it, likely by retrieving all
> records and brute-forcing its way through them, which would not only
> be painfully slow, but a lot of implementation complexity.  That said,
> I’m unsure what use-case you have in mind whereby a non-indexable
> column is being filtered/sorted.  Only a couple of the built-in types
> aren’t indexable, right?

My comment was for both d) and e) (sort and filter).  As for filter,
an example is to get interfaces and with oper-status == 'down'.

> Yes, XPath is the “right answer”, but we need to ensure it’s
> constrained enough to be mappable to common DB-backends.

So the client talks to the server, and the server talks to the "db"
backend (note that in many implementations  operational state is
typically not stored in a real database).  So we have:

  client ->  server ->  backend

If we don't have any filter capabilities, the client has to get
everything, and then filter.  If we support XPath and the backend
doesn't support it, the server will have to filter.  This is still
more efficient than filtering in the client.

In ConfD/NSO we moved more and more filtering logic towards the
backend, to speed up performance.

> Updating my previous response (per above, plus adding leaf-list, plus
> locking-in names, which I’m choosing only to NOT sound like SQL):
> 
> 
> We haven’t discussed “feature” statements yet, but it seems reasonable
> to me for both ‘d’ and ‘e’ to be optionally implemented.
> Additionally, it may be that additional Xpath expressions can be
> unlocked by feature statements).

I think e) is way more important than c).  I suggest focus on a + b +
c.

> Sure, but I wonder if, e.g., a netmask filter, is supportable by
> common DB-backends.  I’m hoping we have some DB-experts on the list!

See above.  It can be quite efficient even if the backend doesn't
support it.

/martin

---

From: Kent Watsen <[email-masked]>
Date: Wed, 30 Sep 2020 17:55:23 +0000

> If we agree to not do d) then we don't need any use cases for it ;-)

We don’t agree that sorting isn’t ever need, but I think that it’s fair to say that, amongst the various other parameters, it’s the least important, and so enabling it via a feature seems prudent.

BTW, sort (d) comes after filter (e), in our e-->a processing model.  Hopefully the (fast) filter whittles the result-set down to something manageable so the (slow) sort isn’t painfully too egregious.

> do this regardless of how the list is orderd on the server.

Possibly, and certainly it would be easy to implement, I’m just trying to understand when that might be the case.  For instance, using a firewall rulebase (an OBU-list) as an example, when could getting the rules in any other order be meaningful to a client?

>> columns”, <snip/>

“oper-status” appears to be an enum, an hence indexable…or perhaps this is a forward-reference to your next comment about some opstate not being in the DB...

> typically not stored in a real database).  

Some opstate must be persisted, e.g., long-lived counters, logs, etc., but it’s a good point about other opstate not being persisted.   Perhaps “node-tags” can be used here, to differentiate which is which…and servers can indicate if/how they support the ephemeral opstate leafs in queries?

> So we have:

Probably, especially when assuming the server has better resources and/or the client <--> server bandwidth is constrained.

> backend, to speed up performance.

Ack.

> I think e) is way more important than c).  I suggest focus on a + b +

Noted, but I think you meant a + b + e.

Note sure how others feel about “direction: (c), but my primary use-case revolves around time-series data (e.g., logs), where the interest is commonly on the most-recent entries, so "reverse-->offset—>limit” works nicely.  

Perhaps an alternative would be to lift a concept from Python with negative indexes so, for instance, offset=-N and limit=-N gives the last N entries?

> See above.  It can be quite efficient even if the backend doesn't

I don’t see that above, but I don’t doubt that it can be so, it’s just a whole lot of implementation complexity.  It seems that we should/must support servers doing it, we just need to find a way (node-tags?) to enable them to express that ability.

K.

---

From: Olof Hagsand <[email-masked]>
Date: Wed, 30 Sep 2020 20:59:53 +0200

> BTW, sort (d) comes after filter (e), in our e-->a processing model.  Hopefully the (fast) filter whittles the result-set down to something manageable so the (slow) sort isn’t painfully too egregio…

I agree sorting is less important (than a,b,e).
Is it not also somewhat beyond the present yang concepts where there is
already ordered-by and key indices?
Supplying a differing sorting could in effect require a different set of
index(es) which could be quite costly in runtime for large data sets.
If you want to sort differently, should it not be present in the
underlying model, ie as declaring a secondary set of keys, for example,
as an extension of the original yang model (thus being out of scope here)?

>>> constrained enough to be mappable to common DB-backends.

I would think a (strongly) constrained XPath would be a very nice solution.

> Perhaps an alternative would be to lift a concept from Python with negative indexes so, for instance, offset=-N and limit=-N gives the last N entries?

To me reverse direction seems a subset of ordering/sorting? If sorting
is less important, so would an alternate direction?
Unless there are some important use-cases, reverse order having the
advantage of the same sorting mechanism/indexes, just in reverse (not
introducing other keys, for example)

> _______________________________________________
> netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>

---

From: Kent Watsen <[email-masked]>
Date: Wed, 30 Sep 2020 20:00:24 +0000

Hi Olaf,

> I agree sorting is less important (than a,b,e).

Noted.

> as an extension of the original yang model (thus being out of scope here)?

A couple messages I sent yesterday alluded to defining an extension statement that could be added to nodes in the YANG modules.

Now I’m thinking that one-size doesn’t necessarily fit all, and something like a server-specific “node-tag” would be more flexible.  (https://tools.ietf.org/html/draft-tao-netmod-yang-node-tags <https://tools.ietf.org/html/draft-tao-netmod-yang-node-tags>)

Maybe the answer is somewhere in-between, i.e., an extension that acts like a “hint” and a node-tag indicating if the server delivered it.  Bad news w/ an extension is that legacy modules would need to be revved or augmented.

One thing, though, filtering on non-indexed nodes is guaranteed to have slow performance, thus it seems that we need to enable indexing for more than just a list’s keys.

My conjecture is that, once we have the indexes for filtering (e), they can be used for sorting (d).  At least, that’s the way it works with some DB-backends I’m familiar with...

> I would think a (strongly) constrained XPath would be a very nice solution.

+1

> To me reverse direction seems a subset of ordering/sorting?

Indeed.  In some DB-backends, “direction" is a sub-statement to the “sort” statement.

> If sorting is less important, so would an alternate direction?

Maybe, but an independent “direction" works with OBU-lists, which is why I teased it out.

And, besides, that assumes sorting is less important.  I do hear a consensus beginning to form in that direction, so we’ll see...

All I know is that there are tons of UIs that enable column sorting (e.g., an email client).  Do we feel data models defined by YANG are somehow immune to this?   

I hear folks expressing concern around sorting big data (e.g., logs), which I grok, but what about for non-OBU CT lists (e.g., admins)?  Wouldn’t we like to sort by name, last-time-connected, etc.?  Can not supporting sorting on big data set be expressed via some combination of extension + node-tag, or implicitly disabled if the pre-filter fails to whittle the result set down to something that can be sorted reasonably?

> introducing other keys, for example)

Sorry, I didn’t parse that.  Can you say another way?

K.

---

From: Martin Björklund <[email-masked]>
Date: Thu, 01 Oct 2020 08:24:18 +0200 (CEST)

> index(es) which could be quite costly in runtime for large data sets.

Agreed.

> as an extension of the original yang model (thus being out of scope here)?

The problem is that the designer of the data model is in most cases
not the client who asks for a particular sort order, and it is not
easy to anticipate all future use cases when a model is designed.

> I would think a (strongly) constrained XPath would be a very nice solution.

I think constraing would be a mistake.  Note that we're talking about
XPath 1.0 here.  (I hope).

/martin

---

From: Martin Björklund <[email-masked]>
Date: Thu, 01 Oct 2020 08:33:49 +0200 (CEST)

Hi,

Kent Watsen <[email-masked]> wrote:
> A couple messages I sent yesterday alluded to defining an extension
> statement that could be added to nodes in the YANG modules.
> 
> Now I’m thinking that one-size doesn’t necessarily fit all, and
> something like a server-specific “node-tag” would be more flexible.
> (https://tools.ietf.org/html/draft-tao-netmod-yang-node-tags
> <https://tools.ietf.org/html/draft-tao-netmod-yang-node-tags>)
> 
> Maybe the answer is somewhere in-between, i.e., an extension that acts
> like a “hint” and a node-tag indicating if the server delivered it.
> Bad news w/ an extension is that legacy modules would need to be
> revved or augmented.
> 
> One thing, though, filtering on non-indexed nodes is guaranteed to
> have slow performance, thus it seems that we need to enable indexing
> for more than just a list’s keys.

I disagree with both these statements.  This is an implementation
issue.

> My conjecture is that, once we have the indexes for filtering (e),
> they can be used for sorting (d).  At least, that’s the way it works
> with some DB-backends I’m familiar with...

This is one implementation strategy.  We should not design the
standard around one implementation.

> And, besides, that assumes sorting is less important.  I do hear a
> consensus beginning to form in that direction, so we’ll see...
> 
> All I know is that there are tons of UIs that enable column sorting
> (e.g., an email client).  Do we feel data models defined by YANG are
> somehow immune to this?
> 
> I hear folks expressing concern around sorting big data (e.g., logs),
> which I grok, but what about for non-OBU CT lists (e.g., admins)?
> Wouldn’t we like to sort by name, last-time-connected, etc.?  Can not
> supporting sorting on big data set be expressed via some combination
> of extension + node-tag, or implicitly disabled if the pre-filter
> fails to whittle the result set down to something that can be sorted
> reasonably?

I agree that this can be useful.  But as has been pointed out earlier
in this thread, sorting implies a stateful solution which will consume
more resources and probably comes with some additional complexity.

At Cisco/tail-f we implemented such a solution; a more generic "query"
mechanism, for both NETCONF and RESTCONF.  Perhaps Per A can share
some details.

/martin