Subject: Re: [netconf] restconf collections

From: Kent Watsen <[email-masked]>
Date: Thu, 1 Oct 2020 13:36:39 +0000

Hi Martin,

>> A couple messages I sent yesterday alluded to defining an extension
>> statement that could be added to nodes in the YANG modules.
>> 
>> Now I’m thinking that one-size doesn’t necessarily fit all, and
>> something like a server-specific “node-tag” would be more flexible.
>> (https://tools.ietf.org/html/draft-tao-netmod-yang-node-tags
>> <https://tools.ietf.org/html/draft-tao-netmod-yang-node-tags>)
>> 
>> Maybe the answer is somewhere in-between, i.e., an extension that acts
>> like a “hint” and a node-tag indicating if the server delivered it.
>> Bad news w/ an extension is that legacy modules would need to be
>> revved or augmented.
>> 
>> One thing, though, filtering on non-indexed nodes is guaranteed to
>> have slow performance, thus it seems that we need to enable indexing
>> for more than just a list’s keys.
> 
> I disagree with both these statements.  This is an implementation
> issue.

Which two statements?   And how do you come to that conclusion, unless you’re suggested that all queries from particular server are equally slow?  ;)

>> My conjecture is that, once we have the indexes for filtering (e),
>> they can be used for sorting (d).  At least, that’s the way it works
>> with some DB-backends I’m familiar with...
> 
> This is one implementation strategy.  We should not design the
> standard around one implementation.

Please elucidate how this isn’t the case.

While a true statement, it’s an unfair implication, as pretty much *everything* I’ve written on this thread has been about trying to enable servers with various DB-backends to express which parts of the solution they can support.   Apparently some backends don’t support “sorts” while others do; some backends can support full Xpath while other can’t, etc.   All this means to me is that we need to define the necessary features and/or node-tags or whatever to enable each to advertise what it can do best, or at all.

If the WG wants the lowest-common denominator, we will likely end up with just ‘a', ‘b', and a heavily redacted Xpath for ‘e’.  I’m hoping that we can do better than that, but I’m also okay with leaving it to each solution can then create proprietary extensions/node-tags, if that is the WG consensus.

> I agree that this can be useful.  But as has been pointed out earlier
> in this thread, sorting implies a stateful solution which will consume
> more resources and probably comes with some additional complexity.

True, except I think the augment was in *favor* of doing that, at least in some cases.

Also note that the “more resources” augment needs to be amortized over the number of queries.  There must be a break-even point whereby preparing a (resource-intensive) snapshot actually uses less resources over time, after it's been queried a certain number of times, than a server doing a fresh (lower-resource) query for each request.

> At Cisco/tail-f we implemented such a solution; a more generic "query"
> mechanism, for both NETCONF and RESTCONF.  Perhaps Per A can share
> some details.

That would be helpful.  The analysis can only take into account that which is known.  I know the Tail-f products used a custom database, but few know anything about it.  

Creating a custom database is not easy.  I know, as I personally wrote a very-fast logging system, complete with indices, for a logging database taking in logs at the rate of 50,000 logs per second, with the total number of logs retained measured in the billions.  

I personally do not wish for a *basic* server to have to go through that pain, hence the interest in supporting common DB-backends (NoSQL, SQL, etc.), while also enabling systems with special abilities to present that which can.

K.

---

From: Martin Björklund <[email-masked]>
Date: Thu, 01 Oct 2020 16:55:48 +0200 (CEST)

Hi,

Kent Watsen <[email-masked]> wrote:
> Which two statements?

1. filtering on non-indexed nodes is guaranteed to have slow
   performance

2. it seems that we need to enable indexing  for more than just a
   list’s keys

> And how do you come to that conclusion, unless
> you’re suggested that all queries from particular server are equally
> slow?  ;)

I'm suggesting that this is highly implementation dependent.

> While a true statement, it’s an unfair implication, as pretty much
> *everything* I’ve written on this thread has been about trying to
> enable servers with various DB-backends to express which parts of the
> solution they can support.  Apparently some backends don’t support
> “sorts” while others do; some backends can support full Xpath while
> other can’t, etc.  All this means to me is that we need to define the
> necessary features and/or node-tags or whatever to enable each to
> advertise what it can do best, or at all.
> 
> If the WG wants the lowest-common denominator, we will likely end up
> with just ‘a', ‘b', and a heavily redacted Xpath for ‘e’.  I’m hoping
> that we can do better than that, but I’m also okay with leaving it to
> each solution can then create proprietary extensions/node-tags, if
> that is the WG consensus.

I would prefer an unconstrained filter, with the XPath 1.0 that is
already used in YANG.  It would be ok if support for such a filter was
optional (a capability).  My point is that even if the backend doesn't
support filtering on a particular node, filtering in the server will
be much more efficient than filtering in the client.  Hence filtering
shouldn't be constrained to certain nodes.  It also makes the solution
much easier to use for clients.

> Also note that the “more resources” augment needs to be amortized over
> the number of queries.  There must be a break-even point whereby
> preparing a (resource-intensive) snapshot actually uses less resources
> over time, after it's been queried a certain number of times, than a
> server doing a fresh (lower-resource) query for each request.

To be clear, I think such a solution would be useful.

> That would be helpful.  The analysis can only take into account that
> which is known.  I know the Tail-f products used a custom database,
> but few know anything about it.
> 
> Creating a custom database is not easy.

Right, but note that this filtering can and should be applied to all
nodes, also operational state nodes that isn't stored in any database
at all, imo.

/martin

> I know, as I personally wrote
> a very-fast logging system, complete with indices, for a logging
> database taking in logs at the rate of 50,000 logs per second, with
> the total number of logs retained measured in the billions.
> 
> I personally do not wish for a *basic* server to have to go through
> that pain, hence the interest in supporting common DB-backends (NoSQL,
> SQL, etc.), while also enabling systems with special abilities to
> present that which can.
> 
> K.
> 
> 
> 
>

---

From: Kent Watsen <[email-masked]>
Date: Thu, 1 Oct 2020 19:54:02 +0000

Hi Martin,

> much easier to use for clients.

1) Current NC/RC Xpath-filter found in <get> was not intended to filter large sets of logs.

2) The Xpath-filters used for streaming (s4.4.3 in rfc8040 and "stream-xpath-filter” in rfc8639) aren’t a fair comparison, as said stream-filters are already dispatching one event at a time.  But do note that rc8639 says the following, indciatiing that full Xpath cannot apply (e.g., preceding-sibling):

	The XPath expression is evaluated on the representation of
	individual, delineated event records as contained in
	the event stream.

3) Here we’re talking about querying a potentially large repository of logs that are at rest, and full table scans should avoided.  I suggest that we’re dealing with something new here.

IMO, for pretty much any CT data, full table scans isn’t a deal-breaker, as I don’t feel that configured data is ever that big (even for interfaces or a firewall rulebases).  It would be unfortunate for servers to have a reimplement all the filtering logic that is normally provided by backends (just because Xpath supports something that isn’t mappable), and it wouldn’t be as fast, but it is possible to implement and, to your point, it’s likely faster for the server to do it than the client.

But when it comes to CF data, doing a full table scan is impractical and index-backed queries can make the difference between the client waiting seconds vs minutes.  This is why I was initially trying to focus on just solving the CF problem, where the pain is acute.

That said, I propose we do “both”.  That is, the default is a customer-satisfying Xpath-subset known to be mappable to common DB-backends, and a feature statement is used to enable full Xpath support.  In both cases, the Xpath context node is the individual list-entry.

For some servers claiming full Xpath support, it could be that if a client’s query uses something beyond the mappable-subset, then the server silently switches to its “slow-path” logic (e.g., table scan until “count” is reached, if specified).  The server’s documentation can describe the behavior,  and even go so far as to suggests that it’s not that big of a deal for CT lists, but that the client may be waiting for a long time and/or the server may run out of resources, if applied to a big CF list.

> nodes, also operational state nodes that isn't stored in any database

This could be when a server silently switches to its slow-path logic.

K.

---

From: Martin Björklund <[email-masked]>
Date: Fri, 02 Oct 2020 08:23:31 +0200 (CEST)

Hi,

Kent Watsen <[email-masked]> wrote:
> 2) The Xpath-filters used for streaming (s4.4.3 in rfc8040 and
> "stream-xpath-filter” in rfc8639) aren’t a fair comparison, as said
> stream-filters are already dispatching one event at a time.  But do
> note that rc8639 says the following, indciatiing that full Xpath
> cannot apply (e.g., preceding-sibling):
> 
> 3) Here we’re talking about querying a potentially large repository of
> logs that are at rest, and full table scans should avoided.  I suggest
> that we’re dealing with something new here.

Perhaps we don't understand each other.  Here's an pseudo-code example
of what I envision:

  GET /interfaces/interface
  filter  'oper-status = "up" and statistics/in-errors > 0'

This is XPath 1.0, applied to one instance at the time.

> IMO, for pretty much any CT data, full table scans isn’t a
> deal-breaker, as I don’t feel that configured data is ever that big
> (even for interfaces or a firewall rulebases).  It would be
> unfortunate for servers to have a reimplement all the filtering logic
> that is normally provided by backends (just because Xpath supports
> something that isn’t mappable), and it wouldn’t be as fast, but it is
> possible to implement and, to your point, it’s likely faster for the
> server to do it than the client.
> 
> But when it comes to CF data, doing a full table scan is impractical
> and index-backed queries can make the difference between the client
> waiting seconds vs minutes.  This is why I was initially trying to
> focus on just solving the CF problem, where the pain is acute.

Well, it is less likely that an implementation uses a feature-rich (in
terms of query language) DB to store all its state, than its config.
So it would probably be easier to implement this using "an index" for
config than for config false...

> That said, I propose we do “both”.  That is, the default is a
> customer-satisfying Xpath-subset known to be mappable to common
> DB-backends, and a feature statement is used to enable full Xpath
> support.  In both cases, the Xpath context node is the individual
> list-entry.
> 
> For some servers claiming full Xpath support, it could be that if a
> client’s query uses something beyond the mappable-subset, then the
> server silently switches to its “slow-path” logic (e.g., table scan
> until “count” is reached, if specified).  The server’s documentation
> can describe the behavior, and even go so far as to suggests that it’s
> not that big of a deal for CT lists, but that the client may be
> waiting for a long time and/or the server may run out of resources, if
> applied to a big CF list.

I agree.

/martin

---

From: Kent Watsen <[email-masked]>
Date: Fri, 2 Oct 2020 13:07:36 +0000

> This is XPath 1.0, applied to one instance at the time.

Of course I understand Xpath filters, other relevant examples here:

	- https://tools.ietf.org/html/rfc5277#section-5.2 <https://tools.ietf.org/html/rfc5277#section-5.2>
	- https://tools.ietf.org/html/rfc8040#appendix-B.3.6 <https://tools.ietf.org/html/rfc8040#appendix-B.3.6>

The point was/is that supporting full-Xpath for stream-filters is easy, or rather doesn’t entail special consideration, as already each event is in memory for an Xpath analysis.  Thus, pointing to existing usage as evidence that unconstrained Xpath can be used everywhere, including for potentially very-large CF-lists, is not a given, as it would be performance-limiting to bring every list-entry into application-level memory in order to use an Xpath library.  In this case, it is very much desired to push all the querying logic to the DB-backend.

> config than for config false...

Really?  ...my experience is seems opposite.  

A number of server implementations I’m aware of persist the entire configuration as a flat-file (e.g., pretty-print XML or JSON).  They bring it into memory and act on it using a feature-rich library (e.g., Yangson).  In my view, this approach hits the low-bar in terms of what constitutes a “database”. 

On the flip side, handling large CF-lists (e.g., logs) is where servers pretty much have to use a real database.  An RDB (relational DB) is common, but a TSDB (time-series DB) <https://en.wikipedia.org/wiki/Time_series_database> is better suited.

Perhaps we’re saying the same thing (yes, config APIs tends to have better query expressions)…it might just be your last line regarding “index” that’s throwing me (as clearly the DBs storing large CF-lists are having indexes).

>> That said, I propose we do “both”.  <snip/>
> 
> I agree.

Okay, we’ll shoot for a two-pronged approach.

Separately, for the RC-list draft only, the plan is to declare “list” and “leaf-list” nodes (not their entries) as data resource targets.  We only need to do this for GET, but I assume that it's better to cover all the HTTP operations - thoughts?

K.

---

From: Hongwei Li <[email-masked]>
Date: Fri, 2 Oct 2020 09:31:22 -0400

Hi Kent,

On Fri, Oct 2, 2020 at 9:07 AM Kent Watsen <[email-masked]> wrote:

> - https://tools.ietf.org/html/rfc5277#section-5.2
> - https://tools.ietf.org/html/rfc8040#appendix-B.3.6
>
> The point was/is that supporting full-Xpath for stream-filters is easy, or
> rather doesn’t entail special consideration, as already each event is in
> memory for an Xpath analysis.  Thus, pointing to existing usage as evidence
> that unconstrained Xpath can be used everywhere, including for potentially
> very-large CF-lists, is not a given, as it would be performance-limiting to
> bring every list-entry into application-level memory in order to use an
> Xpath library.  In this case, it is very much desired to push all the
> querying logic to the DB-backend.
>
>
> A number of server implementations I’m aware of persist the entire
> configuration as a flat-file (e.g., pretty-print XML or JSON).  They bring
> it into memory and act on it using a feature-rich library (e.g.,
> Yangson).  In my view, this approach hits the low-bar in terms of what
> constitutes a “database”.
>
> On the flip side, handling large CF-lists (e.g., logs) is where servers
> pretty much have to use a real database.  An RDB (relational DB) is common,
> but a TSDB (time-series DB)
> <https://en.wikipedia.org/wiki/Time_series_database> is better suited.
> [HL] A user case for time series data (e.g., logs), customers want to get
> the data in between Time A and Time B. Do we use timestamp filter here?
>

> Perhaps we’re saying the same thing (yes, config APIs tends to have better
> query expressions)…it might just be your last line regarding “index” that’s
> throwing me (as clearly the DBs storing large CF-lists are having indexes).
>
>
> Separately, for the RC-list draft only, the plan is to declare “list” and
> “leaf-list” nodes (not their entries) as data resource targets.  We only
> need to do this for GET, but I assume that it's better to cover all the
> HTTP operations - thoughts?
> [HL] Data resources should not be bound to some specific operation.
> Correct me, if my understanding is wrong.
>
> K.
> _______________________________________________
> netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>

---

From: Kent Watsen <[email-masked]>
Date: Fri, 2 Oct 2020 14:46:27 +0000

Hi Hongwei,

> [HL] A user case for time series data (e.g., logs), customers want to get the data in between Time A and Time B. Do we use timestamp filter here?

Some events have only one timestamp.  The database likely has a primary key (e.g., "record-id”) and a separate “timestamp” field.  For all intents and purposes, the logs are persisted in time-order, so no additional sorting is needed.  The physical order is good enough.

However, other events may have distinct "time-generated" and "time-received” fields.  This is most notable for a “log-receiver” (or just “receiver", in rfc8639 parlance), as there may be a delay between when a log is generated by a publisher and when it is persisted by the receiver.   In this case, the user-expectation is undoubtedly to sort on “time-generated”. The “record-id” and “time-received” fields are physically in order (same as above), but “time-generated” could be all over the place, so a sort is needed.

FWIW, if the goal is to find logs generated in a window of time around a timestamp, a fast-filter can be used to whittle down the result-set to a (hopefully) manageable size before the slow-sort as follows:

	filter
		"time-generated >= timestamp-of-interest - some-window”
		and
		“time-received <= timestamp-of-interest + some-window"
	sort-by
		“time-generated"

K.

---

From: Hongwei Li <[email-masked]>
Date: Fri, 2 Oct 2020 11:08:56 -0400

Thanks Kent
This is clear enough. Just want to make sure we cover that.
The next question is: can we avoid the slow-sort? Like to split the request
automatically or something else? IMO, this is more on implementation side.

On Fri, Oct 2, 2020 at 10:46 AM Kent Watsen <[email-masked]> wrote:

> Some events have only one timestamp.  The database likely has a primary
> key (e.g., "record-id”) and a separate “timestamp” field.  For all intents
> and purposes, the logs are persisted in time-order, so no additional
> sorting is needed.  The physical order is good enough.
>
> However, other events may have distinct "time-generated" and
> "time-received” fields.  This is most notable for a “log-receiver” (or just
> “receiver", in rfc8639 parlance), as there may be a delay between when a
> log is generated by a publisher and when it is persisted by the receiver.   In
> this case, the user-expectation is undoubtedly to sort on “time-generated”. The
> “record-id” and “time-received” fields are physically in order (same as
> above), but “time-generated” could be all over the place, so a sort is
> needed.
>
> FWIW, if the goal is to find logs generated in a window of time around a
> timestamp, a fast-filter can be used to whittle down the result-set to a
> (hopefully) manageable size before the slow-sort as follows:
>
> filter

---

From: Kent Watsen <[email-masked]>
Date: Thu, 29 Oct 2020 00:32:57 +0000

[Update: a spinoff team has formed and is working on two I-D’s]

One issue has come up, regarding the encoding when GET returns a list or leaf-leaf.

Generally, in RFC 8040, GET returns an instance of the targeted resource.  This continues to be possible with JSON, as the list/leaf-list is a JSON array, but no array-equivalent exists with XML.  

This is likely why Section 4.3 states "More than one element MUST NOT be returned for XML encoding”, though this statement never needed to be said, as there is no case where more than one element would be returned anyway!

This intent of this email is to discuss changing that statement in 4.3, that is, to allow more than one element to be returned for the XML encoding.

Following are what I hope the WG can agree on being valid responses:

===== JSON =====

 REQUEST

    GET /restconf/data/example-jukebox:jukebox/library/artist?where=<expr>
    Host: example.com
    Accept: application/yang-data+json

 RESPONSE-1 (list exists, but no matching content)

    HTTP/1.1 204 No Content
    Date: Thu, 26 Jan 2017 20:56:30 GMT
    Server: example-server

 RESPONSE-2 (matching content)

    HTTP/1.1 200 OK
    Date: Thu, 26 Jan 2017 20:56:30 GMT
    Server: example-server
    Content-Type: application/yang-data+json

    {
      "example-jukebox:artist”: [
        {
         “name”: "Foo Fighters”,
          “album”: {
            “name”: "One by One”,
            “year”: 2012
          }

        },
        {
          “name”: "Nick Cave and the Bad Seeds”,
          “album”: {
            “name”: "Tender Pre”,
            “year”: 1988
          }
        }
      ]
    }

===== XML =====

  REQUEST

    GET /restconf/data/example-jukebox:jukebox/library/artist?where=<expr>
    Host: example.com
    Accept: application/yang-data+xml

  RESPONSE-1 (list exists, but no matching content)

    HTTP/1.1 204 No Content
    Date: Thu, 26 Jan 2017 20:56:30 GMT
    Server: example-server

  RESPONSE-2 (matching content)

    HTTP/1.1 200 OK
    Date: Thu, 26 Jan 2017 20:56:30 GMT
    Server: example-server
    Content-Type: application/yang-data+xml

    <artist xmlns="https://example.com/ns/example-jukebox">
      <name>Foo Fighters</name>
      <album>
        <name>One by One</name>
        <year>2012</year>
      </album>
    </artist>
    <artist xmlns="https://example.com/ns/example-jukebox">
      <name>Nick Cave and the Bad Seeds</name>
      <album>
        <name>Tender Prey</name>
        <year>1988</year>
      </album>
    </artist>

Thoughts?  If no objection, we will proceed as per above.

PS: I am aware, of course, of the pseudo-parent element (“collection”) defined in in draft-ietf-netconf-restconf-collection.  The hope is to avoid needing to introduce a pseudo element, as the above seems simpler, more consistent, and without issues, other than requiring that statement in Section 4.3 to be rescinded.

K.

---

From: Martin Björklund <[email-masked]>
Date: Thu, 29 Oct 2020 08:51:08 +0100 (CET)

Kent Watsen <[email-masked]> wrote:
> Generally, in RFC 8040, GET returns an instance of the targeted
> resource.  This continues to be possible with JSON, as the
> list/leaf-list is a JSON array, but no array-equivalent exists with
> XML.
> 
> This is likely why Section 4.3 states "More than one element MUST NOT
> be returned for XML encoding”, though this statement never needed to
> be said, as there is no case where more than one element would be
> returned anyway!
> 
> This intent of this email is to discuss changing that statement in
> 4.3, that is, to allow more than one element to be returned for the
> XML encoding.
> 
> PS: I am aware, of course, of the pseudo-parent element (“collection”)
> defined in in draft-ietf-netconf-restconf-collection.  The hope is to
> avoid needing to introduce a pseudo element, as the above seems
> simpler, more consistent, and without issues, other than requiring
> that statement in Section 4.3 to be rescinded.

This is not a good proposal, since the result is not a valid XML
document.  An XML document has exactly one root element.

Also, the media types used in the examples are misleading.  The
example reply doesn't match the media type
"application/yang-data+json" (same for xml).

I suggest you define new media types and a "collection" root
element; or keep the media types and define "collection" in a YANG
module as:

  anydata collection;

/martin

---

From: Kent Watsen <[email-masked]>
Date: Thu, 29 Oct 2020 16:05:13 +0000

Hi Martin,

> document.  

How is this important?  It’s not an HTTP-constraint, right?  and I don’t think it is needed for message-framing either.   

I guess, as you mention below, the media-type is inappropriate (though 11.3.1 isn’t as clear as one might hope), so maybe the response is okay if only changing the media type?

> An XML document has exactly one root element.
> 

How isn’t the media-type okay for JSON, for the response in the previous message?  It seems valid to me...

No dispute regarding the XML media-type.  

> element; or keep the media types and define "collection" in a YANG

Noted, but first would like to understand why simply changing the media-type alone (while keeping the responses in the previous message) isn’t sufficient, for both xml and json. 

Each media-type can define its own format so, e.g., “application/yang-data+xml-list" could be used to define a list of XML documents, right?

That said, I acknowledge that having a root-element may aid xml-parsers, and thus perhaps worthwhile for that reason...just want to ensure we understand the motivations accurately.

K.

---

From: Martin Björklund <[email-masked]>
Date: Thu, 29 Oct 2020 17:22:20 +0100 (CET)

Kent Watsen <[email-masked]> wrote:
> How is this important?

I will be difficult to parse, since you can't just parse it with a
standard XML parser.

> It’s not an HTTP-constraint, right?

No.

> and I
> don’t think it is needed for message-framing either.
> 
> I guess, as you mention below, the media-type is inappropriate (though
> 11.3.1 isn’t as clear as one might hope), so maybe the response is
> okay if only changing the media type?

Sure, from a pure media type pow; but I still think it would be a
mistake to send a concatenation of valid XML documents w/o any
separators (and adding custom separators may work, but note how the
original NETCONF XML encoding tried to do that, but failed...).

> How isn’t the media-type okay for JSON, for the response in the
> previous message?  It seems valid to me...

You're right, it is valid.  My bad.  I forgot how clever the JSON
encoding of a single list entry was ;-)

> Noted, but first would like to understand why simply changing the
> media-type alone (while keeping the responses in the previous message)
> isn’t sufficient, for both xml and json.
> 
> Each media-type can define its own format so, e.g.,
> “application/yang-data+xml-list" could be used to define a list of XML
> documents, right?
> 
> That said, I acknowledge that having a root-element may aid
> xml-parsers, and thus perhaps worthwhile for that reason...just want
> to ensure we understand the motivations accurately.

Yes this is my concern.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 29 Oct 2020 11:59:59 -0700

> I will be difficult to parse, since you can't just parse it with a
+1

It is perplexing why interoperability with standard XML tools would be
forfeited
just to save the encoding length of 1 container  (about 30 bytes)

Andy