Subject: [netconf] Current activities on RESTCONF/NETCONF to support paging

From: "Wisotzky, Sven (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Thu, 14 Mar 2019 09:23:53 +0000

All,

There was some discussion in earlier days (back in 2012/2014) to support paging for extra-long lists. The main use-case is clearly to improve support for interactive user-frontends, as loading entire lists with potentially more than 100.000’s objects easily can become a nightmare: loading time, memory consumption, crashes, …

For the sake of implementation, one would simply provide an attribute called “max-count”, “max-entries”, “max-elements” or “limit” – which basically defines the upper limit of list entries to be returned. Asking for limit=10 – even if the list contains a 100k the result would simply show 10. From an user-interface point of view, this might be enough – as users would see how the list entries look like to narrow down the search by adding additional search/filter criteria. So to protect the client system and improvements on loading time/memory consumptions – this might be just enough.

More advanced implementations might allow the NETCONF client to specify the “offset” which is basically the index of the first list entry to be returned. In such case, the client could continue to load further pages on customer demand (like dynamically while scrolling down the list or pressing on the next button). Even if people are not curious that much about memory consumption, even it should be possible to continue loading the remaining list in the background – while showing the first X items faster on the user-interfaces.

To make such paging solution even more complete, following use-cases are to be reviewed from a RESTCONF/NETCONF API point of view:
/1/ How to figure out the number of list entries matching the given criteria for a given list, without the need to poll the entire list? This could be helpful to operators straight away, but also allows easier implementation for things like progress bars.
/2/ Define a sorting criteria for search results
/3/ Improve filtering capabilities for subtree filters (e.g. contains operator)

I did not found any active DRAFT or RFC for this. That said, we can see various vendors implementing proprietary solutions for this in their RESTCONF stacks.
Would really like to see an IETF defined approach for this, to avoid multi-vendor clients need to deal various vendors implementations.

/wiso

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 14 Mar 2019 10:35:52 +0100

One of the issues to consider is that depending on the datastore
accessed, the data can change more or less rapidly and it is unclear
whether you want servers to maintain potentially big amounts of state
between requests. If you retrieve data in chunks, the question is
whether you can simply put the data back together or the client side
or whether this may lead to data that never really existed in that
form on the server. With RESTCONF, etags may help to check whether the
resource queried remains consistent (but then the server still has to
be able to decide etag changes).

But backing up for this discussion of details, I agree that this is a
problem worth solving by standardizing a common solution.

/js

> _______________________________________________
> netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 14 Mar 2019 10:37:50 +0100

On Mar 14, 2019, at 10:23, Wisotzky, Sven (Nokia - DE/Stuttgart) <[email-masked]> wrote:
> 
> paging for extra-long lists

We happen to have page/count in the CoRE resource directory for this purpose.

https://tools.ietf.org/html/draft-ietf-core-resource-directory-20#section-6.2

This is not YANG-based but maybe an example for how this can be done.

And, no, we haven’t solved the concurrency problem either.

Grüße, Carsten

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 14 Mar 2019 10:41:30 +0100 (CET)

> problem worth solving by standardizing a common solution.

We have run into several use-cases where we'd like to be able to handle 
tens of millions of entries and still keep it declarative configuration 
and stay away from RPCs (which is what often is suggested).

Examples of these use-cases are AFTR binding tables (think huge NAT/tunnel 
box), DHCP servers (with millions of leases potentially also configured 
static leases) and DNS servers (tens of millions of DNS zones).

So yes, I fully support working on solving this problem-space.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Wisotzky, Sven (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Thu, 14 Mar 2019 10:07:33 +0000

Hi Jürgen,

I am not sure if this is so much different from retrieving huge lists using get or get-config today. Capturing information, XML rendering and transmission will take time - so nobody would be surprised, that receiving a list with +100k entries easily takes several minutes. In such situation, it really depends on the server implementation - if concurrent changes impact the result of get/get-config operation on-the-fly, making the result inconsistent.

Clearly a NETCONF/RESTCONF server could avoid such problems by taking a short-term log while capturing and caching all the information, while the XML rendering and transmission would use that snapshot which would be fully consistent. But to implement a NC/RC server that way, comes at some expense of memory footprint which is typically quite limited on network equipment and is likely one of the reasons why NETCONF XPATH support is rarely supported in the industry.

That being said, the only way for a multi-vendor NETCONF client to ensure data consistency is to lock the running datastore while executing a get-config operation.

Based on this, the same could easily be applied to paging. So if paging is simply used to make the user-interface showing tables faster while loading the remaining table in the background, the netconf server could lock the datastore for the entire operation to enforce a better level of consistency.

However, as per my initial post below, just defining a limit (aka max-count) is a very efficient initial step to improve loading time and avoid crashes. And the user-interface could be implemented in a way to first just ask for 100 entries to have something to show, while after this asking the entire list in the background. Just not a big fan of this, because if would cause some overhead.

About the number of list entries, clearly one could put this number for any list into the state model. But again, I believe it is much better to have a generic approach for this.

/wiso

On 14.03.19, 10:35, "Juergen Schoenwaelder" <[email-masked]> wrote:

    
    /js

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Thu, 14 Mar 2019 10:24:09 +0000

I agree that this would be a useful problem to solve.

I also agree with Juergen that there will likely be consistency issues depending on implementation, particularly if the data is coming from <operational>.  But I suspect that this issue already exists for many non-trivial implementations (e.g. where the operational data is distributed to daemons, linecards, or remote slave devices).

Requiring that a consistent view of the conventional configuration datastores can be provided (without locking) might be a reasonable compromise.

Thanks,
Rob

-----Original Message-----
From: netconf <[email-masked]> On Behalf Of Juergen Schoenwaelder
Sent: 14 March 2019 09:36
To: Wisotzky, Sven (Nokia - DE/Stuttgart) <[email-masked]>
Cc: [email-masked]
Subject: Re: [netconf] Current activities on RESTCONF/NETCONF to support paging

One of the issues to consider is that depending on the datastore accessed, the data can change more or less rapidly and it is unclear whether you want servers to maintain potentially big amounts of state between requests. If you retrieve data in chunks, the question is whether you can simply put the data back together or the client side or whether this may lead to data that never really existed in that form on the server. With RESTCONF, etags may help to check whether the resource queried remains consistent (but then the server still has to be able to decide etag changes).

But backing up for this discussion of details, I agree that this is a problem worth solving by standardizing a common solution.

/js

On Thu, Mar 14, 2019 at 09:23:53AM +0000, Wisotzky, Sven (Nokia - DE/Stuttgart) wrote:
> All,
> 
> There was some discussion in earlier days (back in 2012/2014) to 
> support paging for extra-long lists. The main use-case is clearly to 
> improve support for interactive user-frontends, as loading entire 
> lists with potentially more than 100.000’s objects easily can become a 
> nightmare: loading time, memory consumption, crashes, …
> 
> /2/ Define a sorting criteria for search results /3/ Improve filtering 
> capabilities for subtree filters (e.g. contains operator)
>

---

From: "Wisotzky, Sven (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Thu, 14 Mar 2019 10:41:51 +0000

Hi Rob,

In that case we would somehow require to cache the result and to have some sort of query-identifier, to reference back to the result vector. Sounds a bit like overkill to me. Really pushes the server-side requirements (e.g. memory footprint) - but also keeps the question open, on how long you would like the keep the result in the query cache.

/wiso

On 14.03.19, 11:24, "Rob Wilton (rwilton)" <[email-masked]> wrote:

    
    
    Thanks,
    Rob

    -----Original Message-----
    From: netconf <[email-masked]> On Behalf Of Juergen Schoenwaelder
    Sent: 14 March 2019 09:36
    To: Wisotzky, Sven (Nokia - DE/Stuttgart) <[email-masked]>
    Cc: [email-masked]
    Subject: Re: [netconf] Current activities on RESTCONF/NETCONF to support paging

    
    /js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 14 Mar 2019 11:49:48 +0100

On Thu, Mar 14, 2019 at 10:07:33AM +0000, Wisotzky, Sven (Nokia - DE/Stuttgart) wrote:
> 
> However, as per my initial post below, just defining a limit (aka max-count) is a very efficient initial step to improve loading time and avoid crashes.

Sorry, but if code crashes, then someone has to fix the code. Adding
query parameters is not the right answer for this.

/js

---

From: "Wisotzky, Sven (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Thu, 14 Mar 2019 12:14:15 +0000

Depends on whatever we call here a crash. The challenging part of NETCONF/RESTCONF is, that the client would not know how big the get/get-config reply would become. Having a JS-based WebUI client running a RESTCONF query, you easily end up in situations, where you basically need to terminate the javascript because it runs out of memory. And I've even seen situations, where the browser itself did not respond anymore (swapping to disk, ...).

By introducing a protection mechanisms like max-count, the client can control to a certain extent how much data will be received. And by using paging for post loading in the background, the client could still decide if it is possible to load another page or not. So when running out-of-memory or when start swapping, the client could avoid loading any further pages. Generally I am thinking this is a very reasonable aka controlled  approach.

With NETCONF today, for sure one could terminate the NETCONF sessions if too much data is received and might try using a XML stream reader to process what has been received so far. But would claim that this is not a good approach at all and should be avoided.

/wiso

On 14.03.19, 11:49, "Juergen Schoenwaelder" <[email-masked]> wrote:

    
    /js

---

From: tom petch <[email-masked]>
Date: Thu, 14 Mar 2019 17:06:45 +0000

----- Original Message -----
From: "Wisotzky, Sven (Nokia - DE/Stuttgart)" <[email-masked]>
Sent: Thursday, March 14, 2019 12:14 PM

> Depends on whatever we call here a crash. The challenging part of
NETCONF/RESTCONF is, that the client would not know how big the
get/get-config reply would become. Having a JS-based WebUI client
running a RESTCONF query, you easily end up in situations, where you
basically need to terminate the javascript because it runs out of
memory. And I've even seen situations, where the browser itself did not
respond anymore (swapping to disk, ...).

One afternoon, I set an SNMP get-next going on RMON and it was still
going the next day; I then tried over a weekend and it was still going
the next week.  Data was being added to the table faster than my high
speed LAN could retrieve it.  Neither system crashed although response
times for other applications elongated.

I am reminded of e-mail where a POP client may access a very large list
of e-mails on a server.  The first response is a count of how many there
are currently; the client can then request the summary data for each
before deciding which to download in full.  This was designed before the
cloud was thought of and it is a system that still works well.

Tom Petch

>
> By introducing a protection mechanisms like max-count, the client can
control to a certain extent how much data will be received. And by using
paging for post loading in the background, the client could still decide
if it is possible to load another page or not. So when running
out-of-memory or when start swapping, the client could avoid loading any
further pages. Generally I am thinking this is a very reasonable aka
controlled  approach.
>
> With NETCONF today, for sure one could terminate the NETCONF sessions
if too much data is received and might try using a XML stream reader to
process what has been received so far. But would claim that this is not
a good approach at all and should be avoided.
>
> /wiso
>
>
>
>
> On 14.03.19, 11:49, "Juergen Schoenwaelder"
<[email-masked]> wrote:
>     > However, as per my initial post below, just defining a limit
(aka max-count) is a very efficient initial step to improve loading time
and avoid crashes.

---

From: Douglas Hubler <[email-masked]>
Date: Thu, 14 Mar 2019 16:08:40 -0400

On Thu, Mar 14, 2019 at 5:24 AM Wisotzky, Sven (Nokia - DE/Stuttgart) <
[email-masked]> wrote:

> There was some discussion in earlier days (back in 2012/2014) to support
> paging for extra-long lists. The main use-case is clearly to improve
> support for interactive user-frontends, as loading entire lists with
> potentially more than 100.000’s objects easily can become a nightmare:
> loading time, memory consumption, crashes, …
>

speaking from experience, RESTCONF is very usable straight from interactive
user-frontends and this is one of the key things missing.  I did add a
custom parameter to FreeCONF's RESTCONF implementation to support this.
Maybe I took this took far, but I decided that because data is in a
hierarchy you might have multiple lists you need to page thru so I came up
this this parameter

   range=b/c!N-M

returns rows N thru M inclusive in list under path b/c.  And that you can
specify multiple ranges for multiple paths.  Front ends usually have no
problem remembering what number they are on to track N and M.

As brought-up, for lists that are highly dynamic, this strategy is bit
naive and maybe etags can help the clients that need to know when list has
changed and they might need to start over.  For my backend, I could start
iteration at row N without have to manage a cursor.

I did not found any active DRAFT or RFC for this. That said, we can see
> various vendors implementing proprietary solutions for this in their
> RESTCONF stacks.
>
> Would really like to see an IETF defined approach for this, to avoid
> multi-vendor clients need to deal various vendors implementations.
>

+1

---

From: Andy Bierman <[email-masked]>
Date: Sun, 17 Mar 2019 13:39:26 -0700

On Thu, Mar 14, 2019 at 3:24 AM Rob Wilton (rwilton) <[email-masked]>
wrote:

> I also agree with Juergen that there will likely be consistency issues
> depending on implementation, particularly if the data is coming from
> <operational>.  But I suspect that this issue already exists for many
> non-trivial implementations (e.g. where the operational data is distributed
> to daemons, linecards, or remote slave devices).
>
> Requiring that a consistent view of the conventional configuration
> datastores can be provided (without locking) might be a reasonable
> compromise.
>
>
This thread has already mentioned some of the reasons the RESTCONF
Collections draft got shelved.

 1) this is not a RESTCONF-specific problem, so a RESTCONF-specific
solution will not solve the problem
 2) there is no agreement on how much state a server should maintain during
the "get-bulk walk".
 3) some features like sorting require all data to be retrieved within the
server and sorted, instead of streaming the data
 4) position-based "get-next" is error-prone in dynamic operational state;
it requires expensive locking in configuration

I agree a standard solution would be beneficial to client developers, if
all 4 issues could be solved

Thanks,
> Rob
Andy

> One of the issues to consider is that depending on the datastore accessed,
> the data can change more or less rapidly and it is unclear whether you want
> servers to maintain potentially big amounts of state between requests. If
> you retrieve data in chunks, the question is whether you can simply put the
> data back together or the client side or whether this may lead to data that
> never really existed in that form on the server. With RESTCONF, etags may
> help to check whether the resource queried remains consistent (but then the
> server still has to be able to decide etag changes).
>
> > For the sake of implementation, one would simply provide an attribute
> called “max-count”, “max-entries”, “max-elements” or “limit” – which
> basically defines the upper limit of list entries to be returned. Asking
> for limit=10 – even if the list contains a 100k the result would simply
> show 10. From an user-interface point of view, this might be enough – as
> users would see how the list entries look like to narrow down the search by
> adding additional search/filter criteria. So to protect the client system
> and improvements on loading time/memory consumptions – this might be just
> enough.
> >
> > More advanced implementations might allow the NETCONF client to specify
> the “offset” which is basically the index of the first list entry to be
> returned. In such case, the client could continue to load further pages on
> customer demand (like dynamically while scrolling down the list or pressing
> on the next button). Even if people are not curious that much about memory
> consumption, even it should be possible to continue loading the remaining
> list in the background – while showing the first X items faster on the
> user-interfaces.
> >
> > To make such paging solution even more complete, following use-cases are
> to be reviewed from a RESTCONF/NETCONF API point of view:
> > /1/ How to figure out the number of list entries matching the given
> criteria for a given list, without the need to poll the entire list? This
> could be helpful to operators straight away, but also allows easier
> implementation for things like progress bars.