Subject: Re: [netconf] Current activities on RESTCONF/NETCONF to support paging

From: wangzitao <[email-masked]>
Date: Fri, 15 Mar 2019 01:54:36 +0000

发件人: netconf [mailto:[email-masked]] 代表 Douglas Hubler
发送时间: 2019年3月15日 4:09
收件人: Wisotzky, Sven (Nokia - DE/Stuttgart) <[email-masked]>
抄送: [email-masked]
主题: Re: [netconf] Current activities on RESTCONF/NETCONF to support paging

On Thu, Mar 14, 2019 at 5:24 AM Wisotzky, Sven (Nokia - DE/Stuttgart) <[email-masked]<mailto:[email-masked]>> wrote:
There was some discussion in earlier days (back in 2012/2014) to support paging for extra-long lists. The main use-case is clearly to improve support for interactive user-frontends, as loading entire lists with potentially more than 100.000’s objects easily can become a nightmare: loading time, memory consumption, crashes, …

speaking from experience, RESTCONF is very usable straight from interactive user-frontends and this is one of the key things missing.  I did add a custom parameter to FreeCONF's RESTCONF implementation to support this.  Maybe I took this took far, but I decided that because data is in a hierarchy you might have multiple lists you need to page thru so I came up this this parameter

   range=b/c!N-M

returns rows N thru M inclusive in list under path b/c.  And that you can specify multiple ranges for multiple paths.  Front ends usually have no problem remembering what number they are on to track N and M.

As brought-up, for lists that are highly dynamic, this strategy is bit naive and maybe etags can help the clients that need to know when list has changed and they might need to start over.  For my backend, I could start iteration at row N without have to manage a cursor.

I did not found any active DRAFT or RFC for this. That said, we can see various vendors implementing proprietary solutions for this in their RESTCONF stacks..
Would really like to see an IETF defined approach for this, to avoid multi-vendor clients need to deal various vendors implementations.

+1
Agree with you. Actually, there are some discussions in this WG. And there is an expired<https://datatracker.ietf.org/drafts/draft-ietf-netconf-restconf-collection> draft that try to address these issues (https://tools.ietf.org/html/draft-ietf-netconf-restconf-collection-00).
IMO, these problems also exist in NETCONF, therefore, a common solution is required.

Best Regards!
-Michael

---

From: "Wisotzky, Sven (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Fri, 15 Mar 2019 08:48:01 +0000

> Actually, there are some discussions in this WG. And there is an https://datatracker.ietf.org/drafts/draft-ietf-netconf-restconf-collection draft that try to address these issues (https://tools.ietf.org/html/draft-ietf-netconf-restconf-collection-00). IMO, these problems also exist in NETCONF, therefore, a common solution is required. 
 
It should have been clear from my initial ask. Yes, I am actually interested in a solution which serves both NETCONF and RESTCONF. The problem might be seen more important for RESTCONF - as this is considered protocol of choice for network controllers, talking about a larger scale of objects and traditionally used for communication between WebUI and back-end.

The bigger challenge regarding NETCONF is, that networking devices are often more restricted regarding management plane footprint (CPU/Memory). So whatever solution we come up with, it must be easy to implement without requiring large server-side footprint.

Sven

---

From: Robert Varga <[email-masked]>
Date: Fri, 15 Mar 2019 15:12:54 +0100

> It should have been clear from my initial ask. Yes, I am actually interested in a solution which serves both NETCONF and RESTCONF. The problem might be seen more important for RESTCONF - as this is…

That is a valid point, but I think this would end up creating a protocol
on top of RESTCONF, where start of iteration/pagination would create
some session state on the server.

I do not want to explode the scope of work, but wouldn't WebUI users be
better served with a general XPath-based query facility (for example
allowing filter on GETs)?

Alternatively, could the problem perhaps be reformulated in terms of
draft-ietf-netconf-yang-push?

Regards,
Robert

---

From: "Wisotzky, Sven (Nokia - DE/Stuttgart)" <[email-masked]>
Date: Fri, 15 Mar 2019 14:51:13 +0000

Robert,

I would be okay with the limit/offset attributes suggested in draft-ietf-netconf-restconf-collection-00. So this would basically avoid any session state on the server. I do agree, that such approach comes at the caveat that tables might dynamically change while one is requesting follow-up pages. But with all fairness, this is not even guaranteed with traditional NETCONF get / get-config today - given existing vendor's implementations.

Is your suggestion around XPATH, that the WebUI would automatically render smart selection filters such as:
//foo/bar[100 <= position() and position() < 200]
subsequence( /foo/bar, 100, 100 )

I do agree, that this certainly would serve its purpose, depending on the support of XPATH including the position/subsequence functions. But would be concerned would be this dependency. Not really sure, how many vendors do really have full filter/xpath support today in their NETCONF/RESTCONF stacks - at least I do not remind any. Also I could not think of an efficient implementation of this. Likely one would render the full XML document and after this applying the XPATH query. As we are speaking here about extra-large tables - the raw XML document could easily have 100.000's of entries aka 100's of Mbytes. Not really sure about the performance impact and footprint of such solution - but doubt this is the right way of doing it.

I guess your other proposal about YANG PUSH would be almost in the lines of gNMI ONCE subscription. Not sure about about all the implications of this.

/wiso

On 15.03.19, 15:13, "Robert Varga" <[email-masked]> wrote:

    

    Regards,
    Robert