Subject: RE: Notification Message Processing Model

From: "Hector Trevino \(htrevino\)" <[email-masked]>
Date: Thu, 23 Mar 2006 10:14:13 -0800

From an operational standpoint it makes much more sense to have
dedicated connections or channels for commands/control & notifications.
When using SSH it is an operational/resource constraint decision whether
or not a full dedicated SSH session is used for commands & notifications
or a single session with multiple channels is utilized.
A notifications session/channel needs to allow the transfer of commands
(i.e. rpcs and notifications mixed) because for example (just one, there
are others) a management application dedicated to
monitoring/troubleshooting may need to retrieve additional information
in response to a received fault-notification. I think that in this
situations it is ok to simply interleave messages. 

Now, given the above, someone may decide to use a single session for
both rpcs and notifications. This is a lose-lose situation as you point
out below. One thing that could be done if need be is to allow
configuration of priorities (prioritize rpcs over notifications or
viceversa but that's about it) & things run to completion. 

See additional comments in-line
Hector

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Andy Bierman
> Sent: Wednesday, March 22, 2006 8:57 PM
> To: Netconf (E-mail)
> Subject: Notification Message Processing Model
> 
> Hi,
> 
> I am trying to understand how the protocol will have to be 
> modified to support notifications and RPCs on the same 
> channel-less connection.
> 
> The document will have to define all possible interactions 
> between RPC and notifications in the architecture, regardless 
> of whether 1 or 2 connections are used.  However, if a shared 
> connection is used, then all possible message processing 
> interactions within the same session also needs to be defined.
> 
> Some example issues to consider:
> 
> 1) agent message priority (<rpc-reply> vs. <notification>).
>    It is possible that messages of both type need to be queued
>    for output in the agent.  Who wins if both are waiting?
>    (Hint: operationally, both answers could be wrong.)

Sure, no right answer here. But as I said above the implementation could
allow prioritization of rpcs or notifications at the user's convinence. 

> 
> 2) The sword cuts both ways.
>    We could have really big notifications, not just a
>    really big <rpc-reply>.  This can seriously impair
>    the use of the session for time-sensitive RPC operations
>    (like fire-fighting a virus attack by making a configuration
>    change to a firewall, router interface ACL entry, etc.).

Yes. But IMO this is an operational matter. The protocol cannot solve
bad operational planning (at least not w/o much not needed complexity).
The operator(s) responsible for managing the network are likely aware of
this or could be made aware and should choose to have dedicated
sessions/channels. 

>   
> 3) lack of an <abort> command.
>    The manager cannot shut up the agent if it is spewing a reply
>    or a notification, except by closing the connection.
>    We took this command out of the protocol when we took out
>    channels.  We took out channels because of unwarranted inherent
>    complexity.

I think an abort command would've been useful regardless of whether or
not you have
mixed rpcs and notifications. 

> 
> 4) notification amplification
>    Multiple notification subscriptions per session means that
>    there are potentially N copies of every notification that
>    get generated by the agent per session, where N is the number
>    of subscriptions. This makes the impact of issues (2) and (3)
>    even worse.

Yes, sure but again this comes back to operations as I commented in 2.
The document could have some operational guidelines/recommendations on
this. 

> 
> 5) notification rate control
>    The content-filtering is not a rate control mechanism,
>    despite that appearance in the draft. (Just the opposite 
> -- see (4))
>    The document will need to define adequate rate control
>    and notification queueing mechanisms (better than FIFO logjam).

OK

> 
> 6) meltdown avoidance and repair
>    It is very likely the NE device will be generating lots of
>    event notifications when things go bad in a hurry, just as
>    the operator is trying to reconfigure the device to make the
>    root cause of the notification flood go away.  The document
>    needs to define mechanisms to detect and prevent this type of
>    scenario.
> 
OK
> 
> 
> Andy
> 
> 
> 
> --
> to unsubscribe send a message to [email-masked] 
> with the word 'unsubscribe' in a single line as the message text body.
> archive: <http://ops.ietf.org/lists/netconf/>
> 

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

---

From: Phil Shafer <[email-masked]>
Date: Thu, 23 Mar 2006 13:52:03 -0500

"Hector Trevino \(htrevino\)" writes:
>out below.

If it's a lose-lose, why allow it?

>One thing that could be done if need be is to allow

This seems like adding more complexity to handle the additional
complexity of a complex solution.

A long-lived RPC running on a distinct channel is trivial and is
allowed within the existing netconf spec for no additional
protocol-level work.  Just make a capability that defines your RPCs
and you are golden.

Thanks,
 Phil

---

From: Shane Kerr <[email-masked]>
Date: Thu, 23 Mar 2006 20:38:05 +0100

> situations it is ok to simply interleave messages. 

Why not perform the query to retrieve additional information on a
separate channel?

As a client developer, it would be simplier. You avoid the case:

server -> client
  event A
client -> server
  RPC asking for more info about event A
server -> client
  event B (sent before RPC began)

The client has to either handle the event asynchronously, or save it in
some buffer somewhere (or perhaps ignore it).

- --
Shane

---

From: Andy Bierman <[email-masked]>
Date: Thu, 23 Mar 2006 11:58:22 -0800

> protocol-level work.  Just make a capability that defines your RPCs

Here is a variant of your proposal that gets rid of the partial
XML document problem.

New Capability:

   :notification-mode

Architecture changes:

   An active NETCONF session has two modes:

   rpc-mode: The session is being used for RPC operations.
   The manager can send <rpc> or <stop-notifications/>.
   The agent can send <rpc-reply>.

   notification-mode: The session is being used for notifications.
   The manager can send <start-notifications> or <stop-notifications/>.
   The agent can send <notification> and <rpc-reply>.

   Initial mode: rpc

Protocol changes:

  New RPC method (i.e., from manager to agent):
 
   <start-notifications>

     Parameters:

        (parameter details TBA -- to-be-argued)

      If the agent is in 'rpc' mode, and the parameters are
      acceptable, then a positive response is returned and the
      agent will enter notification mode. 

      If the agent is already in notification mode, and the agent
      will allow the new parameter set values (if different than
      the current values), then the agent will return a positive
      response and start using the new parameter values.  If the
      new parameter values are the same as the current values,
      then a positive response is returned.

      Otherwise a negative response is returned.

      While in notification mode, all <rpc> requests will be returned
      with negative response <rpc-reply> messages.  In this mode the
      manager should expect to receive <notification> messages that
      meet the selection criteria specified in the 'xxx' parameters,
      generated as various events occur within the agent.

    Positive Response:
 
      <ok/>

    Negative response:

     If the agent cannot honor the requested the request for any
     reason a <rpc-error> response will be generated.

  <stop-notifications>

      If the agent is in rpc mode it will simply return a positive
      response and stay in rpc mode.

      If the agent is in notification mode it will finish sending
      any <notification> message in progress to maintain a well-formed
      XML message stream, and then terminate notification mode, and
      switch to rpc mode.

      Regardless of the current mode, the agent will return a positive
      response.

   Parameters:

      none

   Positive Response:

      <ok/>

   Negative Response:

      none

Andy

---

From: Phil Shafer <[email-masked]>
Date: Thu, 23 Mar 2006 15:23:41 -0500

>   An active NETCONF session has two modes:

This introduces modality into the netconf session, which
is increasing the complexity.

Instead, we could just have a simple RPC that is long-lived:

C:  <rpc ...>
C:    <get-syslog-notifications>
C:      <level>warning</level>
C:    </get-syslog-notifications>
C:  </rpc>

The server responds with an <rpc-reply> containing an
unending series of notifications which match the criteria from
the <get-syslog-notifications> RPC:

S:  <rpc-reply ...>
S:    <notification> ... data for one notification ... </notification>
S:    <notification> ... data for another notification ... </notification>
S:    <notification> ... data for yet another notification ... </notification>
S:    <notification> ... data for one more notification ... </notification>

The notifications would continue until the channel is closed.  Or
until we resurrect the <abort/> mechanism ;^)

This is simple, direct, and uses the existing netconf framework.
If interleaving is a lose-lose, this is a win-win.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Thu, 23 Mar 2006 13:20:52 -0800

> If interleaving is a lose-lose, this is a win-win.

The objection to this approach in the WG meeting was that off-the-shelf 
validation tools
will not be able to work because they will be waiting for the </rpc-reply>
before finishing the validation.  Hence the validation phase never ends.

I love your approach.
It would be simple for me to implement.
I'm not using off-the-shelf validation tools though.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Thu, 23 Mar 2006 16:37:55 -0500

>before finishing the validation.  Hence the validation phase never ends.

What off-the-shelf validation tools?  The ones that understand the
<rpc-one-way> element?  If you are able to toss an <rpc-one-way>
element to your validator, you can toss a <notification> element
to one.  But given that the contents will likely just be a syslog
line, there's really no point.  It's just a line of text.  You hook
you incoming netconf channel up to a sax parser (or a pull parser)
and get ready for a stream of startElement, characters, and endElement
events.

>I love your approach.

Cool....

>It would be simple for me to implement.

Yup, it would be simple for anyone to implement.  That's
the beauty.

Thanks,
 Phil

---

From: Shane Kerr <[email-masked]>
Date: Fri, 24 Mar 2006 00:53:45 +0100

> I'm not using off-the-shelf validation tools though.

I wonder why not do something like:

C:  <rpc ...>
C:    <get-syslog-notifications>
C:      <level>warning</level>
C:    </get-syslog-notifications>
C:  </rpc>
S:  <rpc-reply ...>
S:     <okay/>
S:  </rpc-reply>
S:  <notification>
S:    ... data for notification ...
S:  </notification>
S:  <notification>
S:    ... data for notification ...
S:  </notification>
S:  <notification>
S:    ... data for notification ...
S:  </notification>
   .
   .
   .

Each notification is valid XML, self contained. Do we really need to
wrap the set for some reason?

- --
Shane

---

From: Phil Shafer <[email-masked]>
Date: Thu, 23 Mar 2006 21:29:43 -0500

>wrap the set for some reason?

I see the other side.  Do we really need to break the existing RPC
mechanism for something that can be easily handled within it?  Given
that the notifications are likely to get simple syslog messages,
the requirement to be self contained is easily fulfilled.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Thu, 23 Mar 2006 19:10:07 -0800

> the requirement to be self contained is easily fulfilled.

I believe the kind of tool that only does stand-alone full
document parsing should not be the focus of our design.
This seems to be like designing SNMP for MIB walker tools.
When that happened, it became almost all SNMP was good for.

I suspect that all the actual agent and manager implementers
will be using an event-driven parser, not a document-driven parser.

Since the WG doesn't agree on the functional requirements
for notifications, it is not surprising that there is no
agreement on the solution.  Since all Phil and I really want to do
with NETCONF notifications is deliver text or XML syslog,
this would be fine.  I'm not sure what the rest of the WG
wants to do with these things.

I hate Requirements Documents but this document has hardly
been read or reviewed by anybody in the WG.  I think we
should try to agree on what we want to do with NETCONF
notifications before going any further.  Real use cases,
based on real products, used in real networks -- not
"nice-to-have", "maybe need this someday" features.

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Fri, 24 Mar 2006 16:19:21 +0100

Hello,
We definitely want to use notifications for alarms (something that the operator must 
handle), events (something that the operator should notice) besides log messages.

A never ending rpc-reply just seems strange/ugly to me. Having an XML element that might 
have infinite size is something I don't like.
I feel that using a never ending rpc-reply is a much bigger deviation from the original 
rpc usage then
- just saying that we don't wrap notifications in an rpc or
- using one-way message wrappers instead of the rpc

Balazs

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
TSP System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320     email: [email-masked]

---

From: Phil Shafer <[email-masked]>
Date: Fri, 24 Mar 2006 10:33:36 -0500

>I feel that using a never ending rpc-reply is a much bigger deviation from the original 

But it's not a deviation; it's completely within the spec.  For
example, JUNOScript has a <ping> RPC.  When you perform this RPC,
the ping operation continue until the connection is terminated or
the <abort/> operation is sent.  There's another RPC called
<get-accounting-record-information> which will feed you accounting
records with the same unending paradigm.  The endless reply is
really quite useful.  Yes, it requires your client to understand
that it cannot buffer data endlessly before passing it up to
interested parties, but a <get-route-information> RPC on a box
with a million routes will do this too.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Fri, 24 Mar 2006 08:20:06 -0800

> with a million routes will do this too.

We already have unbounded strings in NETCONF
(not that I didn't try to get rid of all of them).
Unbounded == infinite length.  I'm not exactly sure
why XSD and the WG think this is so important,
but it's in there, so Phil's proposal wouldn't be a new
implementation burden.  For event-driven parsers,
it is not a problem at all.

Here is yet another solution variant:

---------------------------------------------------

2 New capabilities:

   :notification

OR

   :notification-mixed

Make mixed-mode (RPCs and notifications on the same channel)
a capability.  That means:

  - if the :notification capability is advertised the
     agent will process <rpc> messages (while sending <notification>
     messages) by immediately returning an <rpc-error>

  - if the :notification-mixed capability is advertised instead, the
    agent will process <rpc> messages (while sending <notification>
    messages) in the normal manner.

Neither capability is mandatory-to-implement.

Message flow:

      Manager                 Agent

      rpc/start-notif.   --->
      rpc-reply/ok       <---
      notification       <---
      notification       <---
      rpc/*              --->
      notification       <---
      rpc-reply/*        <---

This is essentially the same as the the current draft,
except the manager has to check for 2 variants of
the notification capability instead of 1, and use
that data to know if it needs 1 session or 2 sessions.

If people find the mixed mode so useful, it will
get implemented and deployed.

This gets rid of never-ending element with capabilities
instead of runtime modes, and lets implementors decide
whether they think mixing notifications and operations
on the same channel is something they want to do.

Andy

---

From: Steve Berl <[email-masked]>
Date: Fri, 24 Mar 2006 09:57:42 -0800

The most widely implemented and deployed example of the "never ending reply
XML" is Jabber. There are dozens, if not hundreds of implementations of this
in many different implementation languages. So the argument that it would be
too hard for a manager to implement seems a bit questionable.

-steve

---

From: Steve Berl <[email-masked]>
Date: Fri, 24 Mar 2006 10:03:09 -0800

The main use that I see for notifications is for "configuration change
notifications". A netconf based client would like to know if someone else is
changing the configuration, and just what it is that has been changed. The
notification needs to include information about who made the change, when
they made the change, and exactly what they changed. Just a dumb
notification that "something changed" is not good enough. We need to send a
potentially large message describing the actual changes made. This data is
typically too large to send over a standard syslog message or SNMP trap.

-steve

---

From: Andy Bierman <[email-masked]>
Date: Fri, 24 Mar 2006 10:19:50 -0800

> typically too large to send over a standard syslog message or SNMP trap.

This makes sense.

So far I see a compelling need for 2 standard notification types:

<syslog-event> and <config-change-event>

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 27 Mar 2006 14:14:05 +0200

If we want to use notifications to send multiple results for an <rpc> is this considered a 
<config-change-event> or not a compelling need or ?
Balazs

---

From: Andy Bierman <[email-masked]>
Date: Mon, 27 Mar 2006 07:09:17 -0800

Balazs Lengyel wrote:
> If we want to use notifications to send multiple results for an <rpc> is 
> this considered a <config-change-event> or not a compelling need or ?

Sounds like an experiment with a new RPC model.
The IRTF does research, not the IETF.

Andy

---

From: Kunihiko Toumura <[email-masked]>
Date: Tue, 28 Mar 2006 23:17:06 +0900

Hi,

IMHO, it is unreasonable to adhere to use <rpc> for notification.

In notification, a server (=router/switch) sends message to client (=manager),
and there is no need to reply for the notification message.
This far differs from RPC model (client->server, request-reply).

I prefer separating <rpc> and <notification>, and providing different
application protocol for each method.

-- 
TOUMURA Kunihiko - [email-masked]
Central Research Laboratory, Hitachi, Ltd.

---

From: Phil Shafer <[email-masked]>
Date: Tue, 28 Mar 2006 11:45:42 -0500

>This far differs from RPC model (client->server, request-reply).

It's a matter of perspective, I guess.  I see it as an RPC asking
for a realtime feed of specific syslog data.  The reply contains
the data, and continues feeding data until the request is aborted.
The client is requesting data, the server is serving it up, in a
classic client/server model.

In the subscription-model, the application asks the device to send
notifications as a background task, interleaved with the normal
<rpc-reply>s that are the netconf mainstay.  I dislike this "oh by
the way, if you're bored, please send some notifications" metaphor.
It takes a fairly simple request for data and turns into something
more complex.  It puts multiplexing/interleaving into an RPC mechanism
where it is not well suited, imho.

Thanks,
 Phil