Subject: filtering problem

From: Andy Bierman <[email-masked]>
Date: Fri, 13 Jul 2007 08:19:58 -0700

Hi,

There is another problem with filtering and the examples in sec 5: (!!!)

The draft must say exactly where in the <notification> element
that the <filter> is applied.  The current text does not say anything.

All the examples in sec. 5 are broken because the 'notification type'
layer is missing.  The examples assume the filters can only be
applied to a conceptual datastore, just like the <rpc> filter.

However, the most commonly needed filter is going to be
on the notification type itself!

Example (w/o namespaces):

   <notification>
     <configChange>
        <configChangeTime>date-time string...</configChangeTime>
        <configChangedBy>[email-masked]</configChangedBy>
        <configTarget>/interfaces/interface[name='eth0']</configTarget>
         ...
     </configChange>
   </notification>

For simplicity, assume the manager just wants
this one notification type. The filter is going to be something like:

  <filter type="subtree">
    <configChange/>
  </filter>

A filter for configChange just on a specific configTarget might be:

  <filter type="subtree">
    <configChange>
      <configTarget>/interfaces/interface[name='eth0']</configTarget>
    </configChange>
  </filter>

The way the draft is now does not reflect how notifications are
actually structured.

Andy

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 13 Jul 2007 16:43:42 +0200 (CEST)

> that the <filter> is applied.  The current text does not say anything.

Agreed.

> layer is missing.

I think it is ok.  Specify that the filter is applied to the
'notificationContent' element as root (which is the abstract element).

> The examples assume the filters can only be

Yes, and IMO this is consistent with the examples.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 13 Jul 2007 09:18:05 -0700

> 'notificationContent' element as root (which is the abstract element).

Ok -- if it is explained like this, the <event> element is renamed
to <fooEvent>, and the <notification> wrapper is shown, then I agree.

Andy

---

From: "tom.petch" <[email-masked]>
Date: Mon, 16 Jul 2007 18:16:27 +0200

This may relate to something that has been bugging me.

XPath is - at times - specified in terms of a root, and an XML document has a
single root element.  What is on the wire is an XML document but the datastore
is not - as Andy has pointed out before.

So what is an XPath filter being applied to?  The event as it would appear on
the wire as an XML document?  A conceptual document created in the datastore for
the purposes of filtering?  And if the latter, should we - we should! - specify
a root, either for the purposes of the examples or else for everything.

RFC4741 is quite discursive about roots but the notification I-D is silent; I
think that this last needs to change.

Tom Petch

---

From: Balazs Lengyel <[email-masked]>
Date: Thu, 19 Jul 2007 17:14:07 +0200

Hello,
I think the datastore will often have multiple roots. I foresee/have heard of the following 
situations that as I understand are all valid in Netconf:
1) One root in one namespace, other namespaces might be mounted as subtrees
2) One root per namespace
3) One namespace but with many root elements
4) Many roots in many namespaces

Case 2 can be viewed as a the single rooted case 1) where the first branching is according to 
the namespace.

Case 3) and 4) I feel the filter should be applied to each root and the results of these 
combined under the <data> element in the get/get-config reply.

Comments?

Balazs

---

From: "David B Harrington" <[email-masked]>
Date: Fri, 20 Jul 2007 01:20:12 -0400

I think we need one root, so a get-config will be able to include
everything.

David Harrington
[email-masked]
[email-masked]

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Balazs Lengyel
> Sent: Thursday, July 19, 2007 11:14 AM
> To: tom.petch
> Cc: [email-masked]; [email-masked]
> Subject: Re: roots wasRe: filtering problem
> 
> Hello,

---

From: Andy Bierman <[email-masked]>
Date: Thu, 19 Jul 2007 22:37:48 -0700

> I think we need one root, so a get-config will be able to include

We already have this capability.
If <get> or <get-config> has no filter, then it will
return everything.  Since the XML document starts with <rpc-reply>
and all the returned DM nodes are rooted at level 3, under <data>,
there is no XML problem.

The answer XML already supports is (4) - many NS, many roots.
Anything else would be a Clever Little Restriction
that the WG would be inventing.

But standard data models should not follow the same ad-hoc design
as vendors.

A manager should expect any nodes within the <rpc-reply>
to use namespaces and process them correctly.   In addition,
the IETF might want to impose some structure on the standard conceptual
data model, and perhaps even plan ahead for more data model modules
in the future, developed by other WGs.

Andy

---

From: "Vincent Cridlig \(vcridlig\)" <[email-masked]>
Date: Fri, 20 Jul 2007 11:23:45 +0200

I also think one root including mounted subtrees for various namespaces
is better than concatenated multiple roots. These multiple roots can
become children of a main root in any case.
Otherwise filters will have to apply to many roots, rather than one. How
do you build the reply in case nodes from various roots are selected...?
This can happen when namespaces are reused in various data models to
include data following a generic structure.

In Xpath, one missing thing IMO is how is built the reply after an Xpath
get-request. Xpath result is always a node set. Is the reply a
concatenation of the result nodes, or is it a reconstructed document
containing the selected nodes? I don't think this is a data model
dependant question. This aspect is defined in the case of subtree
filtering and it should be defined also for Xpath.

Regards,
Vincent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of David B Harrington
Sent: vendredi 20 juillet 2007 7:20
To: 'Balazs Lengyel'; 'tom.petch'
Cc: [email-masked]; [email-masked]
Subject: RE: roots wasRe: filtering problem

David Harrington
[email-masked]
[email-masked]

> I think the datastore will often have multiple roots. I foresee/have 
> heard of the following situations that as I understand are all valid 
> in Netconf:
> Case 2 can be viewed as a the single rooted case 1) where the first 
> branching is according to the namespace.
> 
> Case 3) and 4) I feel the filter should be applied to each root and 
> the results of these combined under the <data> element in the 
> get/get-config reply.
> 
> Comments?
> 
> Balazs
> 
> tom.petch wrote:
> >>> The examples assume the filters can only be applied to a 
> >>> conceptual datastore, just like the <rpc> filter.
> >>>
> >>> However, the most commonly needed filter is going to be on the 
> >>> notification type itself!
> >>>
> >>> For simplicity, assume the manager just wants this one 
> >>> notification type. The filter is going to be
> something like:
> >>>
> >>>   <filter type="subtree">
> >>>     <configChange/>
> >>>   </filter>
> to unsubscribe send a message to [email-masked] with the

> word 'unsubscribe' in a single line as the message text body.

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

---

From: Andy Bierman <[email-masked]>
Date: Fri, 20 Jul 2007 08:02:52 -0700

> include data following a generic structure.

Think about what this means.
Let's say there is a top-level element everybody is forced
to use, called <top>, in the NETCONF base namespace.
Every vendor would need to change every agent and every
manager to support this change.  NETCONF is supposed
to be content-independent.  This doesn't seem so independent.

All vendor and even all IETF data model objects are
going to be in different namespaces, so the child nodes of <top>
are all the nodes that would have appeared in <config> or <filter>,
accept moved down one extra XML layer.  It doesn't change the filtering
mechanism at all.  (This is the XML equivalent of the Spinal Tap
moment - "But this amp goes to 11").

The real problem that exists (if any), is the lack of a consistent
XML container for <copy-config> output.  I use the <config> element
in the NETCONF base namespace for this purpose.  It doesn't matter
so much what the QName value is for this top-level container,
but there does need to be one.  It ensures the <copy-config> output
will be valid XML, even if multiple data models from vendors
and the IETF exist in the agent.

Andy

---

From: "Vincent Cridlig \(vcridlig\)" <[email-masked]>
Date: Fri, 20 Jul 2007 17:49:28 +0200

Well, ok but 
- if I send a get-config to your agent and I need to validate it on the
manager side, or
- if I build a config on the manager side and I want to validate it
before sending, 
I will also need to append a root node before validation, that in any
case need to be defined in the data model schema. Now that I think about
it, you can validate any individual root separately... 

This does not seem very natural to me. A root would remove the need to
append a "config" for copy-config as you pointed out.
I might be wrong, but it seems to me that, in most XML implementations,
creating a root node and appending multiple child subtrees from other
XML documents will be a pain and very costly, because you need to change
the document owner of every nodes to have a consistent document. Some
API may make it easier but, IMHO, it would be so much more simple to add
a root.

I understand your point with the consequences of such a change. Maybe,
we could just *recommend* that the data model defines a root node. This
is not a killing issue.

My 2 cents.

Vincent

-----Original Message-----
From: Andy Bierman [mailto:[email-masked]] 
Sent: vendredi 20 juillet 2007 17:03
To: Vincent Cridlig (vcridlig)
Cc: David B Harrington; Balazs Lengyel; tom.petch; [email-masked]
Subject: Re: roots wasRe: filtering problem

Vincent Cridlig (vcridlig) wrote:
> I also think one root including mounted subtrees for various 
> namespaces is better than concatenated multiple roots. These multiple 
> roots can become children of a main root in any case.
> Otherwise filters will have to apply to many roots, rather than one. 
> How do you build the reply in case nodes from various roots are
selected...?

Think about what this means.
Let's say there is a top-level element everybody is forced to use,
called <top>, in the NETCONF base namespace.
Every vendor would need to change every agent and every manager to
support this change.  NETCONF is supposed to be content-independent.
This doesn't seem so independent.

All vendor and even all IETF data model objects are going to be in
different namespaces, so the child nodes of <top> are all the nodes that
would have appeared in <config> or <filter>, accept moved down one extra
XML layer.  It doesn't change the filtering mechanism at all.  (This is
the XML equivalent of the Spinal Tap moment - "But this amp goes to
11").

The real problem that exists (if any), is the lack of a consistent XML
container for <copy-config> output.  I use the <config> element in the
NETCONF base namespace for this purpose.  It doesn't matter so much what
the QName value is for this top-level container, but there does need to
be one.  It ensures the <copy-config> output will be valid XML, even if
multiple data models from vendors and the IETF exist in the agent.

> In Xpath, one missing thing IMO is how is built the reply after an 
> Xpath get-request. Xpath result is always a node set. Is the reply a 

Andy

> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of David B Harrington
>> the

---

From: "tom.petch" <[email-masked]>
Date: Fri, 20 Jul 2007 18:47:20 +0200

----- Original Message -----
From: "Balazs Lengyel" <[email-masked]>
To: "tom.petch" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Thursday, July 19, 2007 5:14 PM
Subject: Re: roots wasRe: filtering problem

> combined under the <data> element in the get/get-config reply.

Mmmm - even more complex than I thought.  My initial concern was whether or not
the Notifications, the draft in Last Call, were regarded for filtering purposes
as part of the configuration datastore at all, or whether the filter was applied
to the document as it would appear on the wire were it to pass a filter.

Tom Petch

---

From: Andy Bierman <[email-masked]>
Date: Fri, 20 Jul 2007 11:13:52 -0700

> to the document as it would appear on the wire were it to pass a filter.

This is the $64 question.
When a <filter> element is provided in the <create-subscription>,
does it apply to the conceptual (internal) NETCONF database,
or rather an XML instance document representing the database?  (no)

The examples seem to contradict text that says it works like <get>.
They show the filter is for the <notification> contents,
starting at its one and only child node -- e.g., <event>.

The draft needs to specify the data root for the notification <filter>.

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 30 Jul 2007 18:32:30 +0200

Hello,
I think a filter that works on the content of the notification (as it would be sent) is needed. 
  People usually filter on things like severity or eventType. E.g. severity is not even part of 
the data model.

I think the filter should work on the notification content.

Balazs

Andy Bierman wrote:
>>> heard of
>> the following
>>> branching
>> is according to
>>>> XPath is - at times - specified in terms of a root, and an XML 
>>>> document has
>> a
>>>> So what is an XPath filter being applied to?  The event as it would 
>>>> appear
>> on
>>>> the wire as an XML document?  A conceptual document created in the 
>>>> datastore
>> for
>>>> RFC4741 is quite discursive about roots but the notification I-D is 
>>>> silent;
>> I

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
TSP System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320     email: [email-masked]

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 30 Jul 2007 18:32:30 +0200

Hello,

Balazs

---

From: "David B Harrington" <[email-masked]>
Date: Tue, 31 Jul 2007 05:43:28 -0400

Hi,

SNMPv3 provides two types of filtering of notifications:
1) access control (where the "filters" are defined by the admin), and
2) customized filtering (where the user filters out uninteresting
data)

Do I understand correctly that you are proposing adding the second
type of filtering to netconf notifications?

Where does one find severity in a netconf notification? Is that part
of the netconf data model, i.e. part of the netconf stream?

IIRC, in Montreal we talked about filters being specific to the stream
type, so a syslog stream might be filtered on severity and facility
(or priority), but it would be hard to filter an SNMP stream on those
properties. Are you proposing some type of netconf-specific classes
for filtering on these types of properties? How would you map to these
for the SNMP stream?

David Harrington
[email-masked]
[email-masked]

> Sent: Monday, July 30, 2007 12:33 PM
> To: Andy Bierman
> Cc: tom.petch; [email-masked]
> I think a filter that works on the content of the 
> notification (as it would be sent) is needed. 
> >>> 1) One root in one namespace, other namespaces might be 
> mounted as 
> >>> subtrees
> >>> Case 3) and 4) I feel the filter should be applied to 
> each root and the
> >> results of these
> >> the Notifications, the draft in Last Call, were regarded 
> for filtering 
> >> purposes
> >> as part of the configuration datastore at all, or whether 
> the filter 
> >> was applied
> >>>> single root element.  What is on the wire is an XML 
> document but the
> >> datastore
> >>>> the purposes of filtering?  And if the latter, should we 
> - we should! -
> >> specify
> >>>>>> There is another problem with filtering and the 
> examples in sec 5: 
> >>>>>> (!!!)
> >>>>>>

---

From: Andy Bierman <[email-masked]>
Date: Tue, 31 Jul 2007 09:55:00 -0700

> type of filtering to netconf notifications?

NETCONF has always have several custom filtering mechanisms,
and vendors are adding more all the time (e.g., with-defaults).
Whatever makes it easiest on the operator will most likely get done.

The filtering in Notifications is different for 2 reasons,
as Martin has described in hallway BoFs...

   1) the filter applies to the notification message, not the conceptual
      NETCONF configuration database, so the agent code may need significant
      modification to support notification filtering.

      If the agent supports the #xpath capability and the #notification
      capability, then it must support xpath filtering of notifications.
      (This is what the current draft implies.)

   2) Instead of pruning data, like for <get> responses, the notification
      filter returns true or false, and determines whether the entire
      filter is dropped or not on a particular session.

> of the netconf data model, i.e. part of the netconf stream?

It is part of a data model for the notification content,
written by some vendor or SDO.

The WG did not reach agreement on a generic (same say reinvented)
event classification system.

More standards work is needed on the structure of content in
the NETCONF stream.  (People will submit proposals I think ;-)

> for filtering on these types of properties? How would you map to these

This would be a new kind of filtering.
We are trying to reuse the existing mechanisms as much as possible.
Access control on a stream?  Filter on a stream?
IMO, this is not relevant to the NETCONF protocol.
NETCONF says that the filters work on XML, not conceptual streams.
That can only mean the individual notification elements which
contain an eventType from a particular namespace.  There is no
restriction that the same QName could not be used in a different
stream (especially combo-streams).  So it may seem like a nit,
but let's try to keep the XML stuff straight.

For our purposes, the agent must construct (at least conceptually)
an XML instance document to compare against the filter, and
interpret the output and process it according to what RFC 4741 or the
Notification draft expects to happen.

Andy