Subject: [netconf] get-data origin filters

From: Andy Bierman <[email-masked]>
Date: Sat, 5 Oct 2019 18:11:59 -0700

Hi,

I am trying to figure out how to use the origin-filter and
negated-origin-filter
in the <get-data> operation in RFC 8526.

          leaf-list origin-filter {
             type or:origin-ref;
             description
               "Filter based on the 'origin' annotation.  A
                configuration node matches the filter if its 'origin'
                annotation is derived from or equal to any of the given
                filter values.";
           }

These filters seem kind of worthless if implemented according to the text.
Consider a simple example where there is 1 learned leaf within a list:

module: address
  +--rw addresses
     +--rw address* [last-name first-name]
        +--rw last-name     string
        +--rw first-name    string
        +--rw street?       string
        +--rw city?         string
        +--rw zipcode?      string
        +--rw phone* [phone-type]
           +--rw phone-type      enumeration
           +--rw phone-number    string

Let's say the "zipcode" field is learned in <operational>
(e.g. ZIP code lookup expands missing or 5 digit zipcode to full 9 digit
zipcode).
So /addresses and /addresses/address have origin "intended".
Only the /addresses/address/zipcode leaf has origin "learned".

So how does origin-filter=learned find all the learned leafs?
What filters are required to return only the learned entries + ancestors +
ancestor-or-self keys?  Seems like this filter mechanism has to be used
to retrieve the exact leaf that might be learned, and the client
needs to know in advance all the possible nodes that might be learned.

Want to be able to retrieve an ancestor that is intended and still find the
learned entries

   get-data xpath-filter=/addresses/address origin-filtter=learned

The draft shows an example where both "intended" and "system" are given
as filters.  This will work but will include all the "intended" leafs as
well.
What if a "learned" node is within a "system" node within an "intended"
node?
Seems like the client needs to know a lot about the server implementation
details
in order to use the origin filters.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 06 Oct 2019 17:32:56 +0200 (CEST)

Hi,

> So how does origin-filter=learned find all the learned leafs?

Perhaps I don't understand your question; IMO you give the answer to

>    get-data xpath-filter=/addresses/address origin-filtter=learned

... here.  So this request will return:

   <addresses or:origin="or:intended">
     <address>
       <last-name>...</last-name>
       <first-name>...</first-name>
       <zipcode or:origin="or:learned">...</zipcode>
     </address>
     ...
   </addresses>

> What if a "learned" node is within a "system" node within an "intended"

This works as well.  Note that the get-data description says:

          Any ancestor nodes (including list keys) of nodes selected by
          the filters are included in the response.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Sun, 6 Oct 2019 09:18:42 -0700

>        <zipcode or:origin="or:learned">...</zipcode>
I do not interpret the text the same way as you.

                     The content returned

          by get-data must satisfy all filters, i.e., the filter
          criteria are logically ANDed.

              Configuration nodes that do not have an
              'origin' annotation are treated as if they have the
              'origin' annotation 'or:unknown'.

>           the filters are included in the response.

The issue is how the /iaddresses and /addresses/address nodes match the
origin "learned".
The leafs in list "address" are a mixture of "intended" and "learned"
origin.
The text clearly says that a node has a single origin property, coupled to
the annotation.

Issue 1: mixed origin descendant nodes
So how does a search on /addresses/address match origin-filter=learned?
I cannot find any text that says what the origin of a list or P-container
is if it
contains nodes of mixed origin.

Issue 2: NP-containers

Also from RFC 8342:

   The origin applies to all configuration nodes except non-presence
   containers.

What if the top-level node is an NP-container in this case.
I thought the top-level node MUST have an origin attribute.

The text is not clear how NP-containers are handled.
Do they have an origin attribute? If not then RFC 8526 says they have
origin "unknown".
Is the intent that NP-containers always pass the origin-filter tests (test
skipped)?

/martin
Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 07 Oct 2019 09:43:27 +0200 (CEST)

> I do not interpret the text the same way as you.

Does this mean that you think that the reply is different from what I
show above?  If so, what would it be, and why?

> The issue is how the /iaddresses and /addresses/address nodes match the

They don't, but they are included b/c of the quoted text above (i.e.:
      Any ancestor nodes (including list keys) of nodes selected by
      the filters are included in the response.)

> contains nodes of mixed origin.

See above.

> Is the intent that NP-containers always pass the origin-filter tests (test

No, since they don't have an origin value they will not be selected by
the filter.  But an NP-container will be included in the reply if it
is the ancestor of a node that is selected by the filter.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 7 Oct 2019 07:11:37 -0700

On Mon, Oct 7, 2019 at 12:43 AM Martin Bjorklund <[email-masked]> wrote:

> > > > What filters are required to return only the learned entries +
> ancestors
> > > +
> > > > Want to be able to retrieve an ancestor that is intended and still
> find
> > > the
> > > > learned entries
> > > >

Explain how the list address node has origin "learned".

The filter is for /addresses/address and only origin=learned.
How does the list node have origin=learned?
It can only have 1 value.
It has child nodes with both intended and learned as origin.
I do no understand how the origin=learned matched this node.

>       the filters are included in the response.)

No.

If the filter was for /addresses/address/zipcode then maybe that text
applies.
It is still unclear that the XPath is fully processed and then the
origin-filter is processed.
The RFC just says they are ANDed together.

> > contains nodes of mixed origin.

No text above explains how the list origin is tagged if it has multiple
types of child nodes.

> is the ancestor of a node that is selected by the filter.
The RFC text does not really say that.
Since it is very difficult to know if a data node 5 layers deep is going to
match,
implementing these filters according to this vague interpretation is
unlikely.

Andy

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 7 Oct 2019 14:36:35 +0000

Hi Andy,

Don’t all the filters effectively work this way?

They select a subset of the nodes to include in the response, and must also include all ancestor nodes and required list keys to the selected nodes, regardless of whether those ancestor/key nodes were also selected by the query.

E.g. a “config false” filter will still return “config true” nodes if they are ancestors or list keys to a descendant config false node.  The same logic applies for xpath and origin filters as well.

Thanks,
Rob

From: netconf <[email-masked]> On Behalf Of Andy Bierman
Sent: 07 October 2019 15:12
To: Martin Bjorklund <[email-masked]>
Cc: Netconf <[email-masked]>
Subject: Re: [netconf] get-data origin filters

On Mon, Oct 7, 2019 at 12:43 AM Martin Bjorklund <[email-masked]<mailto:[email-masked]>> wrote:
Andy Bierman <[email-masked]<mailto:[email-masked]>> wrote:
> On Sun, Oct 6, 2019 at 8:32 AM Martin Bjorklund <[email-masked]<mailto:[email-masked]>> wrote:
>
> > Hi,
> >
> > Andy Bierman <[email-masked]<mailto:[email-masked]>> wrote:
> > > Hi,
> > >

No.

If the filter was for /addresses/address/zipcode then maybe that text applies.
It is still unclear that the XPath is fully processed and then the origin-filter is processed.
The RFC just says they are ANDed together.

See above.

No text above explains how the list origin is tagged if it has multiple types of child nodes.

The RFC text does not really say that.
Since it is very difficult to know if a data node 5 layers deep is going to match,
implementing these filters according to this vague interpretation is unlikely.

/martin

Andy

---

From: Andy Bierman <[email-masked]>
Date: Mon, 7 Oct 2019 07:51:47 -0700

> Don’t all the filters effectively work this way?

I do not see the text that explains origin-filter and negated-origin-filter
working the way Martin
describes it.  These filters do not say anywhere to select a node because
it has descendants
that match the origin filters.  It says very clearly that the filter test
is on the specified node.
It also says the origin is derived from the origin annotation for that node.
Since only 1 instance of the origin annotation is allowed per node, there
is no way to tag
a node with multiple origins.

If implementation is too complex then people will just leave it out (w/ a
deviation).
It is unlikely that the instrumentation knows at any given instant all the
origin values
of all the descendant dynamic data at the instant the <get-data> request is
processed.

>
> They select a subset of the nodes to include in the response, and must
> also include all ancestor nodes and required list keys to the selected
> nodes, regardless of whether those ancestor/key nodes were also selected by
> the query.
>
>
>

Yes. Understood.
Still does not explain how a filter for the list node selects descendant
nodes that match the origin filters.

> E.g. a “config false” filter will still return “config true” nodes if they
> are ancestors or list keys to a descendant config false node.  The same
> logic applies for xpath and origin filters as well.
>
>
>

No they won't.
Where is that text?

    get-data config=filter=false

This starts from top-level YANG nodes.
If the top-level YANG node is not config=false then the server will not
keep looking for descendants that match.

Andy

>
>
>
>
> *From:* netconf <[email-masked]> *On Behalf Of *Andy Bierman
> *Sent:* 07 October 2019 15:12
> *To:* Martin Bjorklund <[email-masked]>
> *Cc:* Netconf <[email-masked]>
> *Subject:* Re: [netconf] get-data origin filters
>
>
>
>
>
>
>

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Mon, 7 Oct 2019 16:34:56 +0000

My understanding of the intention of the way the filters are logically meant to work are:

  1.  Construct the full response to the request (as if no filters are to be returned):
  2.  Restrict the response, so that the selected elements match any subtree filter.
  3.  Restrict the response, so that the selected elements match any xpath filter.
  4.  Restrict the response, so that the selected elements match any config true/false filter.
  5.  Restrict the response, so that the selected elements match any origin filter.
  6.  Constrain to the requested depth
  7.  Add in required ancestors and list keys.
  8.  Return the result.

I don’t see text in the filters that states that if a node is filtered then all of its descendants are automatically filtered as well.  I think that you are assuming this behaviour.

A node always only has a single origin, although it could change.  E.g. if a system configured was explicitly configured, then it would make sense to change its origin to configured because it would exist regardless of whether it was originally added by the system.

Thanks,
Rob

From: Andy Bierman <[email-masked]>
Sent: 07 October 2019 15:52
To: Rob Wilton (rwilton) <[email-masked]>
Cc: Martin Bjorklund <[email-masked]>; Netconf <[email-masked]>
Subject: Re: [netconf] get-data origin filters

On Mon, Oct 7, 2019 at 7:36 AM Rob Wilton (rwilton) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Andy,

I do not see the text that explains origin-filter and negated-origin-filter working the way Martin
describes it.  These filters do not say anywhere to select a node because it has descendants
that match the origin filters.  It says very clearly that the filter test is on the specified node.
It also says the origin is derived from the origin annotation for that node.
Since only 1 instance of the origin annotation is allowed per node, there is no way to tag
a node with multiple origins.

If implementation is too complex then people will just leave it out (w/ a deviation).
It is unlikely that the instrumentation knows at any given instant all the origin values
of all the descendant dynamic data at the instant the <get-data> request is processed.

Yes. Understood.
Still does not explain how a filter for the list node selects descendant nodes that match the origin filters.

No they won't.
Where is that text?

This starts from top-level YANG nodes.
If the top-level YANG node is not config=false then the server will not keep looking for descendants that match.

Thanks,
Rob

Andy

From: netconf <[email-masked]<mailto:[email-masked]>> On Behalf Of Andy Bierman
Sent: 07 October 2019 15:12
To: Martin Bjorklund <[email-masked]<mailto:[email-masked]>>
Cc: Netconf <[email-masked]<mailto:[email-masked]>>
Subject: Re: [netconf] get-data origin filters

No.

See above.

/martin

Andy

---

From: Andy Bierman <[email-masked]>
Date: Mon, 7 Oct 2019 09:54:50 -0700

>    7. Add in required ancestors and list keys.
The RFC has no such procedure defined.
The text that is there says nothing about matching descendant nodes.

Andy

>
>
>
>
> I don’t see text in the filters that states that if a node is filtered
> then all of its descendants are automatically filtered as well.  I think
> that you are assuming this behaviour.
>
>
>
> A node always only has a single origin, although it could change.  E.g. if
> a system configured was explicitly configured, then it would make sense to
> change its origin to configured because it would exist regardless of
> whether it was originally added by the system.
>
>
>
> Thanks,
>
> Rob
>
>
>
>
>
> *From:* Andy Bierman <[email-masked]>
> *Sent:* 07 October 2019 15:52
> *To:* Rob Wilton (rwilton) <[email-masked]>
> *Cc:* Martin Bjorklund <[email-masked]>; Netconf <[email-masked]>
> I do not see the text that explains origin-filter and
> negated-origin-filter working the way Martin
>

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 07 Oct 2019 19:39:09 +0200 (CEST)

> is on the specified node.

Let's simplify the example somewhat.  Suppose the client sends:

  <get-data>
    <datastore>ds:operational</datastore>
    <origin-filter>or:learned</origin-filter>
    <with-origin/>
  </get-data>

Are you saying that b/c the top-level node doesn't have or:learned,
this filter will return nothing?

It seems you are assuming that the server first matches the top-level
nodes, and if they match, then match descendants and so one.  The text
doesn't say that, however.  It says:

                A configuration node matches the filter if its
                'origin' annotation is derived from or equal to any of
                the given filter values.

and:

So I think it is pretty clear that the origin filter matches the
zipcode nodes in the example, and then the ancestors are included in
the reply.

> a node with multiple origins.

Agreed.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 7 Oct 2019 10:50:17 -0700

> this filter will return nothing?

Yes -- the origin for addresses is "intended".
The text clearly says the origin-filter has to match the origin annotation
for the node.
The annotation for "address" can have only 1 value.

I don't think this is implementable so I will just use
deviation(not-supported) and ignore it

Andy

> > > > > > These filters seem kind of worthless if implemented according to
> the
> > > > > text.
> > > > > > (e.g. ZIP code lookup expands missing or 5 digit zipcode to full
> 9
> > > digit
> > > > > > zipcode).
> > > > > > ancestor-or-self keys?  Seems like this filter mechanism has to
> be
> > > used
> > > > The text clearly says that a node has a single origin property,
> coupled
> > > to
> > > > the annotation.
> > > >
> > > Since it is very difficult to know if a data node 5 layers deep is
> going
> > > to match,
> > >

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 07 Oct 2019 21:04:01 +0200 (CEST)

> deviation(not-supported) and ignore it

Fine; up to you!

But the question is if the spec is unclear or not.  From what I can
tell so far, it seems it is not incorrect, but can probably be
clarified, perhaps along the lines of Rob's explanation.

/martin