Subject: [netconf] restconf collections

From: Kent Watsen <[email-masked]>
Date: Fri, 25 Sep 2020 11:03:31 +0000

The restconf-collections draft [1] expired long ago, and yet the problem remains.

What is the problem?  In my opinion, the problem is more narrow in scope than before.  Specifically, I feel that it is limited to just "config false" lists, which tend to contain an enormous number entries.  Classic examples include "audit logs" and "device logs".

I do not feel that "config true" lists are a problem because 1) if it's possible for the client to configure a list, then it should be no problem for the client to handle getting it back and 2) clearly this is what everyone is doing today, unless using proprietary solutions, as RFC 8040 offers no option.

It's understood that supporting "config true" lists could improve performance some, but this seems more like a "nice to have" compared to supporting "config false" lists, where returning all list entries in a single response risks exhausting internal memory buffers resulting in failures, and thus solving this problem seems more of a "must have".  Not to mention, handling "ordered-by user" lists would be very difficult.

[As an aside, how are people handling large "config false" lists today?  Is each list-entry streamed out as a "notification"?  Isn't there still always a need to have at least the most-recent list entries, which could still be a lot, on the server?]

The net-net is that I'm thinking we should brushoff the restconf-collections draft, and I'm wondering to what extent the WG agrees.

Anyone?

Regarding solution, I suggest a two-fold approach:

  1) only support query parameters (e.g., offset, limit, sort, where) if a "config false" list is queried directly.  E.g., GET {+restconf}/ds/ietf-datastores:operational/example-module:widgets/widget?offset=20?limit=50.  Note that RESTCONF does not allow the list itself to be a resource target is exactly why this makes sense.

  2) any "config false" list that is a descendent of the resource target returns a placeholder response, rather than its contents.  Specifically, a top-level query on <operational> would NOT return any "config false" list entries.  As for the placeholder response, perhaps it could return metadata, something like: { "example-module:widgets": { "widget": [ { "@": { "ietf-collections:redacted": [None] } } ] } }.  To be backwards-compatible, it could be this list "entry" is only returned if the client sends a special query parameter (e.g., ?collections-supported) or, alternately, the server implicitly always only returns at most one list-entry (e.g., as if ?limit=1 was passed).

Comments?

K.

[1] https://datatracker.ietf.org/doc/draft-ietf-netconf-restconf-collection <https://datatracker.ietf.org/doc/draft-ietf-netconf-restconf-collection>

---

From: "Per Andersson (perander)" <[email-masked]>
Date: Fri, 25 Sep 2020 12:31:07 +0000

> The net-net is that I'm thinking we should brushoff the restconf-collections draft, and I'm wondering to what extent the WG agrees.

I agree it should be brushed off.

I have been looking at restconf-next on github where this has been mentioned.
Will it be part of restconf-next or a separate effort?

Today we implement limit and offset as proprietary extensions to RESTCONF.
It would be good to have them (or some other pagination mechanism)
standardized.

If I recall correctly, we have use cases for config true list sizes in the thousands
or even tens of thousands.

>  1) only support query parameters (e.g., offset, limit, sort, where) if a "config false" list is queried directly.  E.g., GET {+restconf}/ds/ietf-datastores:operational/example-module:widgets/widge…

I would suggest supporting (at least) pagination for any list.

What is the point of selective support? As a user I would find it strange.

>  2) any "config false" list that is a descendent of the resource target returns a placeholder response, rather than its contents.  Specifically, a top-level query on <operational> would NOT return…

It feels like the implementation could be simpler with the implicit ?limit=1
behaviour. Even with pagination the list could be large and perhaps
streamed out which would make collection of metadata more challenging.

Another possibility is to let the client handle such queries, instead of
replacing list contents with a placeholder let the client set ?depth=1
and effectively omit list contents..

--
Per

---

From: Kent Watsen <[email-masked]>
Date: Fri, 25 Sep 2020 15:49:09 +0000

Hi Per,

> I agree it should be brushed off.

Interested in being a co-author?

> I have been looking at restconf-next on github where this has been mentioned.

Ref: https://github.com/netconf-wg/restconf-next/issues/1 <https://github.com/netconf-wg/restconf-next/issues/1>

> Will it be part of restconf-next or a separate effort?

If an NBC change is needed, it would have to be a -bis, otherwise, if all backwards-compatible, it could be a -bis or a new RFC that "updates" RFC 8040.

> It would be good to have them (or some other pagination mechanism)

By "we", do you mean the Tail-F group within Cisco?  What I'm really asking here is if the proprietary extensions are essentially captured by Martin in the restconf-collections draft.

> or even tens of thousands.

List of "interfaces" is the quintessential example I've heard in the past.   Does your implementation support the extensions of CT lists?    If so, does it support them on "ordered-by user" lists too?    Is the use-case (to support on CT lists) a "nice-to-have" or a "must-have"?   If a "must-have", how is it justified?

> I would suggest supporting (at least) pagination for any list.

You mean to let the "limit" and "offset" params be available for both CT and CF lists, but limit "sort" and "where" to CF lists?  That might work, as it sidesteps the odd "sorting a user-sorted list" case...

> What is the point of selective support? As a user I would find it strange.

If by "selective support" you mean the "where" parameter, I think that it's only useful for CF lists (e.g., logs) whereby the client only wishes to fetch entries matching some criteria (e.g., where user==foo or source-addr=bar).  Makes sense?

> streamed out which would make collection of metadata more challenging.

An implicit ?limit=1 would be simple to implement, but assuming we're trying to introduce the behavior in a backwards-compatible manner, how to differentiate a truncated list from a list that actually contains just one value?   Assuming a legacy client (that does not know to look for or grok a ietf-restconf-collections module in yang-library)...or are you saying that the current behavior is so broken that such an NBC change would actually be welcomed by legacy clients?

> and effectively omit list contents..

Assuming the "depth" query parameter is supported, but the server could still *implicitly* support ?depth=1 w/o supporting the capability.  That said, the response would still be the size of the list, right?   I've also thought to let server's implicitly return as if "?fields=<key>*" (i.e., just return the keys) has been passed but, again, the response would be the size of the list.  Such options might be okay for CT-lists, but not so much for the potentially enormous CF-lists I have in mind...

K.

---

From: Andy Bierman <[email-masked]>
Date: Fri, 25 Sep 2020 08:59:52 -0700

Hi,

This has been discussed a few times by the WG.
The co-authors never agreed on the set of bells and whistles that should be
supported.
There was an objection from Phil Shafer at the time that pagination was not
needed
at all and handling large responses from the server was just a minor
implementation detail.
The most significant objection was that this is a RESTCONF-only solution
and that
was not acceptable to ignore NETCONF (since <get> and GET end being the same
in the end, so the problem also exists for NETCONF)

Andy

On Fri, Sep 25, 2020 at 4:03 AM Kent Watsen <[email-masked]> wrote:

> What is the problem?  In my opinion, the problem is more narrow in scope
> than before.  Specifically, I feel that it is limited to just "config
> false" lists, which tend to contain an enormous number entries.  Classic
> examples include "audit logs" and "device logs".
>
> I do not feel that "config true" lists are a problem because 1) if it's
> possible for the client to configure a list, then it should be no problem
> for the client to handle getting it back and 2) clearly this is what
> everyone is doing today, unless using proprietary solutions, as RFC 8040
> offers no option.
>
> It's understood that supporting "config true" lists could improve
> performance some, but this seems more like a "nice to have" compared to
> supporting "config false" lists, where returning all list entries in a
> single response risks exhausting internal memory buffers resulting in
> failures, and thus solving this problem seems more of a "must have".  Not
> to mention, handling "ordered-by user" lists would be very difficult.
>
> [As an aside, how are people handling large "config false" lists today?
> Is each list-entry streamed out as a "notification"?  Isn't there still
> always a need to have at least the most-recent list entries, which could
> still be a lot, on the server?]
>
> The net-net is that I'm thinking we should brushoff the
> restconf-collections draft, and I'm wondering to what extent the WG agrees.
>
> Anyone?
>
>
>   1) only support query parameters (e.g., offset, limit, sort, where) if a
> "config false" list is queried directly.  E.g., GET
> {+restconf}/ds/ietf-datastores:operational/example-module:widgets/widget?offset=20?limit=50.
> Note that RESTCONF does not allow the list itself to be a resource target
> is exactly why this makes sense.
>
>   2) any "config false" list that is a descendent of the resource target
> returns a placeholder response, rather than its contents.  Specifically, a
> top-level query on <operational> would NOT return any "config false" list
> entries.  As for the placeholder response, perhaps it could return
> metadata, something like: { "example-module:widgets": { "widget": [ { "@":
> { "ietf-collections:redacted": [None] } } ] } }.  To be
> backwards-compatible, it could be this list "entry" is only returned if the
> client sends a special query parameter (e.g., ?collections-supported) or,
> alternately, the server implicitly always only returns at most one
> list-entry (e.g., as if ?limit=1 was passed).
>
>
> Comments?
>
> K.
>
>
> [1]
> https://datatracker.ietf.org/doc/draft-ietf-netconf-restconf-collection
> _______________________________________________
> netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>

---

From: Kent Watsen <[email-masked]>
Date: Fri, 25 Sep 2020 16:56:31 +0000

Hi Andy,

> This has been discussed a few times by the WG.

Perhaps a testament to it being a real problem?

> The co-authors never agreed on the set of bells and whistles that should be supported.

True, but it's more like we lost interest / time to work on it...at least that's my story!  ;)

> There was an objection from Phil Shafer at the time that pagination was not needed
> at all and handling large responses from the server was just a minor implementation detail.

Clearly Phil is in the rough, as many APIs support pagination for a reason.   Even if constraining the solution to networking gear, interfaces and logs may be numerous, and on-box cli/web UIs can only show a screen at a time...

IMO, it's okay if we don't support for CT-lists (i.e., interfaces).  As you write above, and I wrote in before, it seems like a nice-to-have.   FYI, I'm asking Per in a fork of this thread to justify if supporting CT-lists is a "must-have"...hopefully he'll come back with something compelling.

CF-lists, on the other hand, can be enormous...so much so that trying to return all could exhaust server and/or client resources.   I think supporting CF-lists is a must-have.

> The most significant objection was that this is a RESTCONF-only solution and that

True.  If we can find a way to extend support to NETCONF, all the better.   Of course, the solutions would be wildly different, much like how NMDA had separate solutions for NC and RC (RFCs 8526 and 8527, respectively).  Perhaps Per or Martin can clue us into if they have a NC-equivalent solution?

However, if NC-support cannot be achieved, then I don't think it means we don't do it for RC.   Much like NC supports somethings RC doesn't (e.g., confirmed commit), each protocol plays into its strengths.   So mote it be.

K.

---

From: Andy Bierman <[email-masked]>
Date: Fri, 25 Sep 2020 10:22:10 -0700

On Fri, Sep 25, 2020 at 9:56 AM Kent Watsen <[email-masked]> wrote:

> Clearly Phil is in the rough, as many APIs support pagination for a
> reason.   Even if constraining the solution to networking gear, interfaces
> and logs may be numerous, and on-box cli/web UIs can only show a screen
> at a time...
>
>

Our customers agree with you.  Some of them only use our <get-bulk>, and
stopped using <get> and <get-config>.
Since it is implemented as an RPC, all YANG-based protocols can use it.
Obviously you need XPath filtering (XSLT-style) to select the list entries
of interest.
Count and depth and other parameters from <get> are also useful.

(My concern is that the IETF version will be too complex to implement or
use as the feature list grows.)

IMO, it's okay if we don't support for CT-lists (i.e., interfaces).  As you
> write above, and I wrote in before, it seems like a nice-to-have.   FYI,
> I'm asking Per in a fork of this thread to justify if supporting CT-lists
> is a "must-have"...hopefully he'll come back with something compelling.
>
>
There is no extra complexity to support config vs. operational data.
I don't see the value in restricting the operation.

> CF-lists, on the other hand, can be enormous...so much so that trying to
> return all could exhaust server and/or client resources.   I think
> supporting CF-lists is a must-have.
>
>
> True.  If we can find a way to extend support to NETCONF, all the better.
>   Of course, the solutions would be wildly different, much like how NMDA
> had separate solutions for NC and RC (RFCs 8526 and 8527, respectively).
> Perhaps Per or Martin can clue us into if they have a NC-equivalent
> solution?
>
>
It is easy. Use RPC operations which map to RESTCONF POST automatically.

> However, if NC-support cannot be achieved, then I don't think it means we
> don't do it for RC.   Much like NC supports somethings RC doesn't (e.g.,
> confirmed commit), each protocol plays into its strengths.   So mote it be.
>
>
> Andy
>
>
> K.
>
>
>
Andy

---

From: Kent Watsen <[email-masked]>
Date: Sat, 26 Sep 2020 17:07:07 +0000

Hi Andy,

> Our customers agree with you.  Some of them only use our <get-bulk>, and stopped using <get> and <get-config>.

> Obviously you need XPath filtering (XSLT-style) to select the list entries of interest.
> (My concern is that the IETF version will be too complex to implement or use as the feature list grows.)

Ack.

> I don't see the value in restricting the operation.

Okay.  This appears to be the prevailing interest.

> It is easy. Use RPC operations which map to RESTCONF POST automatically.

Gotcha.  

These RPCs could also be used for RESTCONF, but much better would be to use query parameters in the GET calls.

K.