Subject: draft-shafer-netconf-syslog-00.txt

From: Phil Shafer <[email-masked]>
Date: Sat, 17 Jun 2006 00:07:30 -0400

I've finally got an initial version of the draft for a netconf
capability that makes syslog data available over long-lived RPCs.
I just sent it off to the RFC Editor, so it should show up on
ietf.org sometime this weekend, but an advance copy is available.

http://professional.juniper.net/phil/ietf/draft-shafer-netconf-syslog-00.txt

Thanks,
 Phil

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 18 Jun 2006 22:24:53 +0200 (CEST)

> capability that makes syslog data available over long-lived RPCs.

I have read this well-written draft, and I have a few comments.

  o  The <event> parameter is defined as a regexp which must match the
     name of the syslog event.  What is the name of a syslog event?
     From the examples it looks like it might be the MSGID defined in
     draft-ietf-syslog-protocol-17.txt?
     
  o  In 3.2.1, you write 

       The <priority>, <event>, and <parameter> parameters are only
       supported if the requested stream is in the "structured-data"
       format.
   
     Why is priority supported only for structured-data?  

  o  Is it correct to assume that a stream is fully defined by the
     parameters in the <stream> element?  If so, should there really
     be a new rpc operation <get-syslog-streams>?  You also write that
     the stream definitions may be part of the configuration.  Thus,
     shouldn't the stream definitions be available by using <get> or
     <get-config> instead?

  o  You use the terms "recorded events" and "historical data".  I
     assume they mean the same thing?

  o  For <text-pattern>, does it match the MSG part of a
     "structured-data" event, or the entire formatted event?

  o  The examples in 3.2.2.1 are supposed to show the same events in
     different formats, but they are not.  (nit: the 'syslog-events'
     element is closed prematurely in these examples (same thing for
     get-syslog-event in 3.2.1.2 as well)).

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 19 Jun 2006 09:09:16 +0200 (CEST)

Hi,

I discussed this briefly with Andy, and it seems the <rpc-reply> in
section 3.2.2 is wrong.  It has to be wrapped in a <data> element in
order to be valid NETCONF.  I.e.

       <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
         <data>
           <syslog-events xmlns="http://ietf.org/netconf/syslog/1.0"/>
             <syslog>
               Jun 11 21:34:52  dent chassisd[2971]:
                 CHASSISD_IFDEV_DETACH_ALL_PSEUDO:
                   ifdev_detach(pseudo devices: all)
             </syslog>
           </syslog-event>
         </data>
       </rpc-reply>

/martin

---

From: Shane Kerr <[email-masked]>
Date: Mon, 19 Jun 2006 12:26:06 +0200

> http://professional.juniper.net/phil/ietf/draft-shafer-netconf-syslog-00.txt

Nicely done. One concern I have is difficulty for clients to actually
get historical messages without duplication.

On my system here, syslog records events with second-precision. A lot
can happen in a second, on a busy system. So, I get a message from syslog:

Jun 19 09:38:36 r00t3r kern.warn kernel: groove is in the heart

Then my NETCONF session ends (for whatever reason... number of messages,
network failure, or so on). So, when my NETCONF session restarts, I can
either:

- - Ask for <start-time>2006-06-19T09:38:36Z</start-time>, and filter out
any messages that I think I have seen (I guess a client could keep a
buffer with all messages from the current second). This presumes the
client knows that the server won't have sub-second accuracy.

- - Ask for <start-time>2006-06-19T09:38:37Z</start-time>, and hope I
didn't miss any messages.

Plus, using time means server clock skew would cause almost unlimited
confusion. :)

I think it would be much nicer to assign a unique identifier to each
event. I recommend an increasing number. In this case, the server would
only need to output the number at the beginning of the answer (and maybe
periodically also). The client knows what the last event it received is,
because it increments for each one.

I think <start-time> still makes sense, as I can see administrators
wanting to always the last 5 minutes of logs in their client when they
start it (for example).

- --
Shane

---

From: Andy Bierman <[email-masked]>
Date: Mon, 19 Jun 2006 08:19:25 -0700

>                    ifdev_detach(pseudo devices: all)

Yes -- This is a feature, not a bug.

All netconf RPC requests share the same basic format:

  <rpc> element in the netconf NS
    <method> and optional parameters in the netconf NS for standard
             operations, and any different NS for a vendor extension
  </rpc>

All netconf RPC replies follow 1 of 2 basic formats:

  <rpc-reply> element in the netconf NS
    <ok/> element in the netconf  NS
  </rpc-reply>

OR

  <rpc-reply> element in the netconf NS
    <rpc-error>*    - Zero or more <rpc-error> elements in the netconf  NS
    <data>?         - Zero or one <data> element in the netconf NS, which
                      can contain zero or more XML elements from any namespace,
                      or no namespace at all.
  </rpc-reply>

<rant>
Just because lots of vendor CLI commands are unorganized,
I mean, ad-hoc, does not mean that IETF standards should
be the same way.

In a standard, consistency and interoperability are far more
important than saving the 13 bytes on the wire that the <data>
element represents.
</rant>

> /martin

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 19 Jun 2006 16:28:45 -0400

>                      can contain zero or more XML elements from any namespace,

The text in netconf-prot does not constrain the top level element
of the reply, nor does it mandate that it be under a <data> element.

   The response name and response data are encoded as the contents of
   the <rpc-reply> element.  The name of the reply is an element
   directly inside the <rpc-reply> element, and any data is encoded
   inside this element.

Unfortunately, the xml schema does constain the reply to <data>.
So the question is: can we correct the schema?
 
>important than saving the 13 bytes on the wire that the <data>

<taking-the-bait>
I fail to see how the existence of the <data> element does anything
to improve consistency or interoperability.  It's just a noise word
that we'll be stuck putting in rpc replies, xpath expressions and
sax filters for the rest of our lives, unless we can fix it now.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 19 Jun 2006 16:39:30 -0400

>get historical messages without duplication.

The <start-time> doesn't really give a duplication-free guarantee,
but one can increase the resolution of the log files (we optionally
add milliseconds) to reduce the duplication.

>Plus, using time means server clock skew would cause almost unlimited

The value for <start-time> will most likely be the timestamp of the
last message received from the server, assuming a connection failure.
In this case, the timestamp will be assigned by the server and it
will be interpreted by the server, so clock skew shouldn't matter.

>because it increments for each one.

A sequence number will interfere with filtering, since the client
won't see monotonically increasing numbers, as messages will be
dropped.  It also introduces a number of complex issues that go
against my KISS attitude, such as the lifespan of the sequence,
sequences for multiple streams, and the interaction for messages
that match multiple streams.  Not unsolvable, but not a priority.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 19 Jun 2006 16:58:50 -0400

>     draft-ietf-syslog-protocol-17.txt?

Yes, that's it.  I would have used the same term, but "msgid" sounded
particular to the message instance (like sequence id).  I used event
name instead, but this is probably just needlessly confusing.

>     Why is priority supported only for structured-data?  

Traditional syslog daemons don't record these fields, so they
can't be supported on recorded data.  It seemed too fine a line,
so I widened it a bit.

>     shouldn't the stream definitions be available by using <get> or

The stream definition may not be configuration, since the device may
be preconfigured with certain streams or have default streams.  I
shy away from the generic <get>, well, it's too generic.  If I want
to add search criteria or fancy options to the RPC, it's less of
a pain if I've got a specific method to enhance.

Also, I'm a big fan of the way netconf allows for the definition
of new RPCs, so we can use more specific queries that do exactly
what we want, allowing us to expose APIs that looks like:

    $device->get_syslog_streams();

instead of:

    #device->get("data/logs/filtered/syslog/streams");

>     assume they mean the same thing?

Yup, I wasn't happy with either term, but I should have settled on
one of the other.

>     "structured-data" event, or the entire formatted event?

Yes, it matches the MSG part:

   <text-pattern> contains a regular expression (as defined by IEEE Std
   1003.2/POSIX.2 [5]) which the message of the SYSLOG must match.  Any
   event whose message string matches this expression is carried in the
   stream.

Both 3164 and syslog-protocol call their message payload "MSG", so I
guess I could s/the message/the MSG part/, but it seems less clear.

>     get-syslog-event in 3.2.1.2 as well)).

Yup,  I'll fix that.  I couldn't find the first set on SD format.
The second error was a cut-n-paste thing.  Thanks for spotting
these.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 19 Jun 2006 14:14:06 -0700

> sax filters for the rest of our lives, unless we can fix it now.

I disagree.
It allows generic tools a consistent format to work with,
instead of a different ad-hoc format for every RPC.

If the reply does not contain 0 - N <rpc-error>,
followed by 1 optional <data>, then SW to pre-process the replies
before the application gets it is much more complicated.
It has to understand every schema instead of just the netconf
schema, just to validate the "RPC layer" portion of the reply.

Currently, we have a consistent way to say "your request
yielded no data":

   <rpc-reply>
     <data/>
   </rpc-reply>

This would be lost if the XSD were changed.

I don't want to give vendors an opportunity to ignore
the standard <rpc-error> requirements.  Allowing the
<rpc-reply> to contain 'anything' will do that.

Andy

---

From: Shane Kerr <[email-masked]>
Date: Tue, 20 Jun 2006 11:15:17 +0200

> add milliseconds) to reduce the duplication.

I was hoping to allow a way for a client to:

1. Not lose log messages.
2. Not get duplicate log messages.

The client has no way to know what the resolution on the server is. The
only way I can think of for the client to remove duplicate messages is
to store all of the messages down to the resolution of the time.

So, if there is a 1-second resolution, the client would have to store
all of the messages that arrived in the same second as the last message.
That is:

2006-06-20T10:55:27 msg1
2006-06-20T10:55:27 msg2
2006-06-20T10:55:27 msg3

If the client disconnects and reconnects to the server, then selects a
<start-time>2006-06-20T10:55:27</start-time>, it would get all of these
again, and discard them as already seen. But it would *need* to pick
that time, in case a 4th message arrived that it did not want to miss.
(Minor issue here that the server must re-send these messages, but I
consider this will be a rare occurrence.)

Whether the resolution is 1-second or 0.1-second, the process is the same.

A client could attempt to minimize the buffer size by figuring out the
resolution of the time on the server. (For instance, by remembering the
smallest non-zero time difference between two events.)

It seems like a lot of work, for what I think is a reasonable set of
goals, which is why I suggested the addition/change.

> will be interpreted by the server, so clock skew shouldn't matter.

My mistake. I should not have said "skew". I meant if the server jumps
the time forward or backwards.

> that match multiple streams.  Not unsolvable, but not a priority.

True, filtering complicates things a bit, as do multiple streams. You're
probably right that it is a bit too complicated for the gain, and a
small buffer on the client is the way to go if one cares.

- --
Shane

---

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 20 Jun 2006 13:31:15 +0200

Phil,

congrats to this good draft. I've just reviewed it from the syslog guy's
perspective. I will provide some thoughts to the currently ongoing
discussions and then provide a summary from the syslog perspective. I am
a lurker on this list, so I might misstate some netconf-related issues,
but I have tried to focus on syslog as much as possible.

> add milliseconds) to reduce the duplication.

The timestamp syslog-protocol allows for microsecond resolution. So
refering to it should be sufficient. It might be a good idea to request
implementations of netconf syslog receivers to  support microsecond
resolution, which is optional in syslog.

Rainer

---

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 20 Jun 2006 13:43:24 +0200

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Phil Shafer
> Sent: Monday, June 19, 2006 10:59 PM
> To: Martin Bjorklund
> Cc: [email-masked]
> Subject: Re: draft-shafer-netconf-syslog-00.txt 
> 
> Martin Bjorklund writes:

I would prefer to stick with the same terminology as in syslog, because
using different terms for the same things indeed confuse.

> can't be supported on recorded data.  It seemed too fine a line,

Actually, PRI is the only thing that is consistently recorded. It is
defined in 4.1.1 in RFC 3164. In the (rare) case where it is absent,
there are rules outlined in RFC 3164. Traditional syslogds filter
exclusively on PRI. So I strongly suggest to support it for both
formats.

For a discussion on syslog format in the wild, please see:
http://www.syslog.cc/ietf/existing-syslog.html

[snip]

> guess I could s/the message/the MSG part/, but it seems less clear.

I strongly suggest to identify the field by calling it "MSG". The reason
is that in RFC 3164 and syslog-protocol, the term "message" is defined
as the complete syslog message, that is HEADER, [STRUCTURED-DATA], and
MSG (e.g. in 6. of syslog-protocol-17).

Rainer

---

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 20 Jun 2006 14:39:15 +0200

Phil,

as promised, I have gone through the netconf-syslog draft from the
syslog guy's perspective. I think netconf-syslog is a very useful
addition to the syslog community, too. I am cross-posting my message to
the syslog WG.

The most important thing that strikes me is terminology. Even though
syslog terminology might not be familiar (or sound even strange) to the
NETCONF WG, I think the draft should stick with it. The reason is that
otherwise you will create confusion. One sample already under
discussion:

In <text-pattern> you specify "the message". As a syslog-guy, I would
apply the regex to the message as whole and not just the MSG part. Thus,
I would find a match e.g. in STRUCTURED-DATA. This is a very natural
approach for someone who implements syslog. Using the same terms also
eases the work for the implementor. From my understanding, the server to
support netconf-syslog must support both netconf as well as syslog. It
is quite confusing in code if the thing the syslog code side e.g. calls
"severity" becomes "level" as soon as another part of the same code is
called. In syslog-protocol, we tried hard to use the most common terms
for new header fields. Unfortunately, syslog has a lot a variance, but I
think we actually managed to find quite good terms.

So I suggest the following term changes:

netconf-syslog          syslog
priority			PRI
level				severity
process			PROCID
event				MSGID
text				MSG (because it works on the MSG field)

I also suggest to add APP-NAME as a filter in netconf-syslog. Adding TAG
(from RFC 3164 & 3195) will also be useful.

Some things of more substance:
<format>
In syslog-protocol, we have introduced the VERSION field. It denotes the
version of syslog-rptocol message format of the message. It deliberately
starts at 1 and increases monotonically (if considerable changed new
versions of syslog should be specified).

I suggest that in netconf-syslog <format> is replaced by <VERSION>,
where zero denotes RFC 3164/3195 format and non-zero denotes a version
according to syslog-protocol. This also facilitates parsing the actual
syslog message on the receiving netconf syslog parser.

<parameter>
I suggest to rename it to STRUCTURED-DATA. I understand that the format
is quite different from the actual syslog STRUCTURED-DATA format. But it
filters on the STURUCTURED-DATA field. This is the same principle I
applied above for text/MSG. I think it would be consistent to call the
filter properties like the fields they operate on.

Names for severity and facility
RFC 3164 and syslog-protocol both list names (e.g. "authorization" and
"emergency") as well as integer numbers for PRI contents. However, only
the integers are normative (see ABNF in syslog-protocol, PRI description
in RFC 3164). It has also been observed in practice that the names are
not without ambiguity. So I strongly suggest to use the integer values
only in netconf-syslog filters. If there is no consensus on this, I
propose to support both the names as well as the numbers.

Remotly Received Events 
From my understanding of netconf-syslog, a netconf-syslog server
supports forwarding via netconf both locally generated syslog messages
as well remotely received ones. I think the ability to process remotely
received messages is a vital asset. It enables a netconf-syslog server
to act as a hub/gateway to a number of netconf-unaware syslog clients.

To fully support remote syslog clients, a HOSTNAME filter is needed.
That would be a (regex?) filter on the hostname. Similar filters are
widely deployed on syslogds in practice today.

When multiple host message may be present in a single netconf-syslog
stream, it might also be that different VERSIONS of the syslog protocol
be present in a single stream. The syslog WG expects this to be a
frequent scenario once syslog-protocol is finished. syslog-protocol
already contains an identification for this (via the VERSION fields) as
well as some specifics on transforming RFC 3164 syslog in a
syslog-protocol receiver (A.1 in syslog-protocol-17). I suggest that
netconf-syslog supports the same.

Finally one small correction. On page 4 of netconf-syslog (at the top)
it states:

"The SYSLOG protocol lacks security and reliability."

This is not true. There is RFC 3195 (syslog over BEEP) which supports
security and reliability. 3195 is currently only weakly accepted, but
there as some implementations of it. I suggest a reference is made to
3195 to clarify on that issue (and so that a casual reader will become
aware that there is more to look at then 3164 and syslog-protocol).

I hope these comments are useful.

Best regards,
Rainer Gerhards

> Sent: Saturday, June 17, 2006 6:08 AM
> To: netconf
> Subject: draft-shafer-netconf-syslog-00.txt
>

---

From: Vincent Cridlig <[email-masked]>
Date: Tue, 20 Jun 2006 17:34:42 +0200

> http://professional.juniper.net/phil/ietf/draft-shafer-netconf-syslog-00.txt
A few comments and questions:

I don't really understand the design choice of the get-syslog-streams 
operation because I think it breaks the idea of the get-config operation.
IMHO, even if it is feasible and allowed by Netconf, nothing here 
justifies the need of a new operation.

<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <get-syslog-streams xmlns="http://ietf.org/netconf/syslog/1.0"/>
</rpc>

Wouldn't it be more conform to Netconf to define a short data model for 
syslog streams and use the regular get-config with a selection node ?
The reply would be exactly the same as the one you described in the 
draft (except the <data> element of course).
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <get-config>
    <source>
      <running/>
    </source>
    <filter>
      <top>
        <syslog-streams xmlns="http://ietf.org/netconf/syslog/1.0"/>
      </top>
    </filter>
  </get-config>
</rpc>

For the second part 3.2, I am confused. What happens if I send a 
get-syslog-events with a stop-time one hour later ? Can I still use this 
Netconf session to send other RPC during this time ? I guess no. The 
Netconf session would be locked for a long period of time.

If I understand well, this approach assumes that the agent is parsing 
the XML data stream on the fly (like with SAX, but not DOM). Because the 
rpc-reply does not seem to be closed until a stop condition happens and 
DOM can not read a partial document. If this is the case, must the agent 
assume that the rpc-reply message will be valid before it received the 
whole message ? And process it ?

Regards,
Vincent

---

From: Phil Shafer <[email-masked]>
Date: Tue, 20 Jun 2006 12:18:57 -0400

>operation because I think it breaks the idea of the get-config operation.

Syslog stream definitions are not limited to configuration, since
the device may define default streams.

>syslog streams and use the regular get-config with a selection node ?

I'm not prepared to be the first lamb on that altar.  ;^)

>Netconf session would be locked for a long period of time.

Yup, exactly.

>whole message ? And process it ?

Yes, if you are looking for a real-time stream, you'll need to
use a technology that allows you to parse it as it arrives.  SAX
is perfect for this, giving you callbacks to drive a simple state
machine.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 20 Jun 2006 09:29:56 -0700

>..
...

I agree with you.
I don't understand how specialized <get> functions for standard data
really helps here.  It sure can't be because there are so many
standard data models for netconf that specialized <get> functions
help us navigate the vast quantity and complexity of this data.

I also don't understand the obsession with calling everything
"subscription data" instead of "configuration data", and insisting
that NV-storing this subscription data is not important, not useful,
or not "simple enough" for the manager to use.

IMO, this problem is pretty simple:

  * Use configuration data and standard existing RPCs to set and get
    the notification generation parameters, what has been called a
    "profile" and now called a "stream"

  * Use a special RPC to start generation of notifications,
    which has a parameter to indicate the stored profile to use.

  * Some parameters like "retrieve since time N" are session-specific,
    and belong only in the RPC, not the config (obviously).  Some
    parameters like "get me the next 3804 notifications" seem pretty
    complicated and not very useful.  I prefer only 1 mode:

      - get since time N (or sequence-id X) and keep going until the session
        is shut down

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 20 Jun 2006 12:30:19 -0400

>addition to the syslog community, too. I am cross-posting my message to

Truly appreciate your comments, especially the terminology ones.  I'll
update the draft with these next week.

>propose to support both the names as well as the numbers.

I strongly dislike using numbers, but if there's consensus
to use them, I'll survive.

>to act as a hub/gateway to a number of netconf-unaware syslog clients.

The target audience for NETCONF (network service providers) don't
configure their boxes to forward syslog messages.  These devices
are the source of events and operators dislike having open ports
on their boxes that aren't strictly required.

But I can see the utility of this for other environments and
the cost is minimal, so I'd consider rolling it in, except....

>be present in a single stream.

... this sounds like it will be an issue.  Can we avoid mixing
VERSIONs in a single stream?

>aware that there is more to look at then 3164 and syslog-protocol).

The target of this comment was 3164, since 3195 doesn't seem to
have been picked up by the operator community, but I'll add a
reference to 3195 and clarify my comment.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Tue, 20 Jun 2006 12:42:45 -0400

>or not "simple enough" for the manager to use.

I don't understand this comment wrt my draft.

The draft's model is that the device defines a set of streams.  This
is not subscription data, nor necessarily configuration data.  What
I like is that it gives the device control over what applications
can see, and prevents applications from asking for everything if
the admin denies it.

The stream approach also gives a recording mechanism (the device
records each stream, not all events) so that an application user
can hit the "what went wrong" button and see a list of recent
messages, possibly from before the application was launched.

>      - get since time N (or sequence-id X) and keep going until the session

The two main scenarios I see are monitoring a stream and viewing the
recorded data from a stream.  Both are important facets of how syslog
events are handled today.

It's a "more" .vs. "tail -f" issue.  ;^)

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 20 Jun 2006 10:00:39 -0700

> I don't understand this comment wrt my draft.

I prefer to use <edit-config>, <get-config>, and standard data models
to configure aspects of this feature such as filtering parameters.

Sharon started in the right direction with a data model, but just
for monitoring.  IMO we should really get out of the SNMP and CLI
mindset that standard data models for configuration are either too hard
or too proprietary for the IETF to touch.

The notion that CLI is all "special command driven" and not actually
data is not really true.  ACLs, Dialer plans, static routes, etc.
are all NV-stored data.  (Imagine how well ACLs would work if
there were none until the root user logged in and "configured"
them all with RPC calls. ;-)

Andy

---

From: Andy Bierman <[email-masked]>
Date: Tue, 20 Jun 2006 10:11:30 -0700

> It's a "more" .vs. "tail -f" issue.  ;^)

Ok.

I was wondering about how live notifications and recorded notifications
were mixed together.  These models imply that they are not
mixed, but rather that live notifications are buffered specially
for the session, and delivery deferred until all the requested
recorded notifications have been delivered.

This is fine, as long as we stay away from complicated
options to provide any other delivery order than this.

Andy

---

From: Cridlig Vincent <[email-masked]>
Date: Tue, 20 Jun 2006 21:11:55 +0200

> is perfect for this, giving you callbacks to drive a simple state
I personally prefer the solution embedding syslog messages in a 
<notification> element
(specified in 
http://www.ietf.org/internet-drafts/draft-ietf-netconf-notification-01.txt)
for the following reasons:

- It does not block the Netconf session, so you can receive syslog 
notifications and still do other things, (Otherwise you need two Netconf 
sessions)
- The processing of messages is easier because parsing partial messages 
is not necessary. Just considering the special caracter ]]>]]> for 
sequencing the messages. I prefer to process a buffer of full Netconf 
messages than start processing a stream which could contain errors lately,
- You can check message well-formness and validity before processing it 
=> more secure,
- You can have the same functionnality (like receiving the next 100 
syslog messages).

Minor point which is more a taste problem:
Whatever will be the solution, I think syslog messages should be parsed 
and rebuilt in an XML structure on the agent side, before being sent to 
the manager. This is easy to do (there are plenty of parser generator) 
and makes the management application more consistent, because everything 
is formatted in the same way. The agent would behave like a full 
syslog/Netconf gateway, similar to what was done with XML/SNMP gateways.

Regards,
Vincent

		
___________________________________________________________________________ 
Yahoo! Mail réinvente le mail ! Découvrez le nouveau Yahoo! Mail et son interface révolutionnaire.
http://fr.mail.yahoo.com

---

From: Andy Bierman <[email-masked]>
Date: Tue, 20 Jun 2006 12:27:53 -0700

Cridlig Vincent wrote:
>>> Wouldn't it be more conform to Netconf to define a short data model 
>>> for syslog streams and use the regular get-config with a selection 
>>> node ?
>>>     
>>

Unfortunately, somebody has to go first.

I suspect that the co-Chairs, ADs, and rest of the IESG are not
going to be very impressed with this excuse as a justification
for what appears to be a very questionable design decision.

Andy

>>> get-syslog-events with a stop-time one hour later ? Can I still use 
>>> this Netconf session to send other RPC during this time ? I guess no. 
>>> The Netconf session would be locked for a long period of time.
>>>     
>>
>> Yup, exactly.
>>
>>  
>>> the XML data stream on the fly (like with SAX, but not DOM). Because 
>>> the rpc-reply does not seem to be closed until a stop condition 
>>> happens and DOM can not read a partial document. If this is the case, 
>>> must the agent assume that the rpc-reply message will be valid before 
>>> it received the whole message ? And process it ?
>>>     
>>

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 21 Jun 2006 10:17:18 +0200

> syslog/Netconf gateway, similar to what was done with 

I essentially agree, BUT... The syslog WG is working on digitial
signatures for syslog messages (syslog-sign I-D). The intention is to
provide a long-lifed record of the authenticy of the log messages, no
matter which transports and gateways have been used. Thus, this initial
sender will sign the messages and the final destination will store an
exact same copy of that message. Then, the original signature can be
verified even years later (think about evidence in court).

The bottom-line to make this happen is that the orginal message is
available on the final destination. Parsing and XML-formatting it
invalidates the message.

One might argue if this is of concern for netconf. Probably not, if only
syslog is used for long term archiving. But you never know.

Besides that concern, I think a standard data model for syslog messages
is definitely needed. Unfortunately, the syslog WG is not yet chartered
to provide it. The current syslog-protocol draft has been written with
the data model in mind. It is fairly trivial to define a standard data
model based on it. It even contains hints for parsing RFC 3164 message
in a way consistent with such a model. The data model might also
optionally contain the original message, which solves the signature
problem (at the cost of a large message size, but that should not be too
much of a concern these days).

I personally wouldn't care if that model is created by the netconf or
syslog WG (though this sound like the more appropriate place). Given the
current participation in both WGs, netconf would, practically thinking,
be a better place to do it.

Rainer

---

From: Vincent Cridlig <[email-masked]>
Date: Wed, 21 Jun 2006 10:43:30 +0200

> syslog is used for long term archiving. But you never know.
Sure. That would be a problem.
I was thinking of an alternative where the agent generates a new 
signature of the XML document (the translated syslog message) using 
XML-DigitalSignature from W3C.
(Before doing so, the agent may check the original syslog signature.)
As usual with security considerations, it would be more costly both for:
- processing time on agent side (check old and generate new signature),
- storage on the manager side (An XML-DS document might consume more 
storage than a signed syslog message).

Vincent

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 21 Jun 2006 10:49:17 +0200

Phil,

> to use them, I'll survive.

I understand your position. At a minimum, I strongly suggest that you
allow either numbers or names. I think most of the filter data is
operator-entered, so it is left to the operators decisions what to use.
This is especially important when different syslog message sources use
different names (though this could be handled by the gateway
exclusively).

> VERSIONs in a single stream?

Let me ask where you see an issue with multiple versions. For me it
seems fairly easy to support them. I've also thought a little more about
the local environment. I think there is no way you can guarantee that
even a single machine will provide consistently one message format
version or the other.

Reasoning: The diagram in 2.1 of netconf-syslog precisely describes how
the syslog subsystem works in many environments, namely Unix and Linux
based systems. The syslog daemon is not the actual message generator. In
fact, it is more like a relay. From the on-the-wire point of view, it is
the initial sender. But if you think about local API, it is just a
receiver forwarding (and potentially transforming) messages. The actual
message generators are system components (called c1, c2, .... cn in your
diagram). These components generate messages via API calls. The API
covers only very limited syslog header fields. Most of the header is
generated by the *system component*. So you are actually not dealing
with a single syslogd and its format but with a potential myriad of
different components on the box. This has been identified as a major
issue moving legacy (rfc 3164) syslog to a standard (syslog-protocol).
This is also one reason why there are so many different formats.

In order to guarantee a consistent data model, you would need to change
the (POSIX) logging API. As this change can not be done evolutionary (it
would break existing apps), the old API would still needed to be
supported. It is thought in the syslog WG that that API change should
happen some time. But we are definitely some years away from that. Not
to mention how long the old API will be utilized...

So my firm believe is that it will be the absolutely common case for
syslog implementations to have to deal with different message formats.

Much of the evil in that can be avoided, if a syslog data model is
defined. This, together with transformation rules, would make it very
easy to handle different versions. As I have written in another mail
this morning, syslog-protocol has such a data model "on its mind" but
can not spell it out because the syslog WG is not chartered to do that.
But it is fairly easy. I have also a working proof-of-concept
implementation of the most important transformations in actual software
(www.rsyslog.com). Its fairly easy to extend that to a full data model
transformation.

> reference to 3195 and clarify my comment.

I agree that 3195 is not overly successful ;) However, you never know
what the future has in stock. As there is no additional effort to
supporting 3195 in netconf-syslog, I think it would be useful to mention
it and to also mention that it can be used together with netconf-syslog
(it is VERSION 0 format, just like 3164).

Rainer

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 21 Jun 2006 10:57:59 +0200

Vincent, 

> storage than a signed syslog message).

There is an importnat legal issue: AFAIK (I am not a lawyer), this would
be a "derived signature" and not an "original signature". I have been
told that (at least in the US) you need to have an original signature to
use the log as evidence in court. Other voices said that if one can
argue that the derived signature is as good as the original one AND can
proove this point, then it *might* be used as evidence, too. The problem
is that you must very carefully argue that no tampering is possible at
the gateway.

Rainer

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 21 Jun 2006 12:41:27 +0200 (CEST)

> the device may define default streams.

So a manager can do <get> to retreive this info.

Suppose an agent actually did make this configurable.  Without a
standard data model, the vendor will have to invent his own model for
this data.  That means that different vendors will use different
models.  So even if there's a standard way to view the data
(get-syslog-streams), there's no standard way to write it.
Furthermore, this data will be available through two different rpcs,
get-syslog-streams and with a vendor-specific namespace through
get-config.

If on the other hand a standard data model is used, there would be a
standard way to read (<get>) and write (for the agents that support
it).

/martin

---

From: Vincent Cridlig <[email-masked]>
Date: Wed, 21 Jun 2006 11:13:52 +0200

> is that you must very carefully argue that no tampering is possible at
You are probably right. I don't know about this legal issue that can be 
different among different countries.
Vincent

---

From: Andy Bierman <[email-masked]>
Date: Wed, 21 Jun 2006 07:39:38 -0700

> standard way to read (<get>) and write (for the agents that support

IMO this is a serious issue.
I completely agree with Martin.
What is the excuse for NETCONF to ignore configuration
and do read-only data models instead?  Should we tell
the ADs and rest of the IESG "That network configuration
problem is really hard -- you should get a WG to do something
about it! Good think we only have to worry about monitoring
in the NETWORK CONFIGURATION PROTOCOL WG."

For years vendors would not do read-write because SNMP was
insecure (but passing cleartext telnet passwords around
didn't bother them too much at the time.)  Then the excuse
was "SNMP is too hard, XML is easier."  Now what's the excuse?
It's time to put up or shut up wrt/ standards based configuration.

Andy

---

From: "David Harrington" <[email-masked]>
Date: Wed, 21 Jun 2006 09:10:31 -0400

Hi Rainer,

I suggest that it would be appropriate to charter a syslog-data WG in
the OPS area to work on standardizing the syslog data modeling format.

dbh 

> From: Rainer Gerhards [mailto:[email-masked]] 
> Sent: Wednesday, June 21, 2006 4:17 AM
> To: Cridlig Vincent; Phil Shafer
> Cc: [email-masked]; Chris Lonvick; [email-masked]
> Subject: RE: draft-shafer-netconf-syslog-00.txt
>

---

From: Balazs Lengyel <[email-masked]>
Date: Thu, 22 Jun 2006 11:55:04 +0200

Hello,
1) I want to fetch recorded data from yesterday 12:00Z till 13:00Z. The draft doesn't seem 
to allow this as the stop time is already in the past. I would allow this!
I would write something like: The server closes the stream based on stop-time if stop-time 
has passed and all events up to stop-time, available at the server have been sent over the 
session.

2) Same comments as for Sharon's draft:
Use a data model for streams and use the get and edit-config operations instead of a 
custom operation for a small set of special (?) data. Even if streams can be defined in 
multiple ways (configuration, system default, etc.) they still can nicely be modeled as 
read-only data.

3)
 > I'm not prepared to be the first lamb on that altar.  ;^)
Someone has to be :-) otherwise NETCONF is dead already.

Balazs

---

From: Andy Bierman <[email-masked]>
Date: Thu, 22 Jun 2006 07:22:09 -0700

> Someone has to be :-) otherwise NETCONF is dead already.

Agreed.
What is especially depressing:

  1) Notification filtering and delivery is a well understood problem,
     traditionally handled with NV-stored configuration on the
     notification generator.

  2) This is a relatively simple configurable feature on a
     complex networking device such as a router.

  3) If the NETCONF WG cannot (or does not) use the
     NETCONF protocol to configure the NETCONF protocol,
     then nobody else will use it either.

  4) If something as basic as a filtering table and notification
     generation parameters are not appropriate for implementation as a
     standard data model (accessed with <edit-config> and <get-config>),
     then what is an appropriate use of the NETCONF protocol
     standard operations and standard data models?

  5) If you can define a representation of a 'feature' in a data model
     for monitoring, then you've already done most of the work for
     configuration.  Read-only vs. read-write is an artificial
     attribute.

Andy

---

From: "Sharon Chisholm" <[email-masked]>
Date: Thu, 22 Jun 2006 10:44:41 -0400

hi

If the working group gets too obsessed with architectural purity at the
expense of usability to the operators and management applications trying
to use an implementation of the protocol I think we are more in trouble.

I've suggested previously the following criteria before deciding whether
it was appropriate to create a new verb:

1. Is it possible to perform this with existing verbs without exploiting
data model side efforts or being ridiculously complicated?
2. Is this operation a management operation and if so does it make sense
for a management protocol to provide this function natively?

I think the argument for this verb falls more out of the second
question. Providing it native in the protocol operation keeps it simple,
easy to use and highlights its importance over arbitrary bits of
configuration. 

And why do we have <get> and <get-config> again? 

Sharon

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Andy Bierman
Sent: Thursday, June 22, 2006 10:22 AM
To: Balazs Lengyel
Cc: Phil Shafer; [email-masked]
Subject: Re: draft-shafer-netconf-syslog-00.txt

Balazs Lengyel wrote:
>...
> 3)
>  > I'm not prepared to be the first lamb on that altar.  ;^)  Someone 
>has to be :-) otherwise NETCONF is dead already.
> 

Agreed.

Andy

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 22 Jun 2006 08:20:18 -0700

> to use an implementation of the protocol I think we are more in trouble.

I really disagree with this comment.
We spent about 5 years developing a set of standard operations
for the standard manipulation of NV-stored configuration data
models encoded in XML.  Instead of using it, some people
want to pretend that filtering parameters do not fall in
the category of NV-stored configuration data.

> easy to use and highlights its importance over arbitrary bits of

I think you are missing the point.
Balazs and I are not saying that a verb to start
the delivery of notifications is a problem.
I totally agree with this approach, rather
than the SNMP MIB approach of turning the knob
to 'enabled' in the data model.  Let's get past this point, okay?

The problem is that many of the parameters
(especially the most complicated part -- the filters)
should be implemented as a read-create standard
data model using the existing <edit-config> operation,
and retrieval of these parameters and notification
state information should be done with the <get> operation.

> And why do we have <get> and <get-config> again? 

To retrieve data

Andy

---

From: "Sharon Chisholm" <[email-masked]>
Date: Thu, 22 Jun 2006 11:58:49 -0400

Eureka. 

So, we are closer then I thought. This is good to news. So, the current
proposal has two approaches:

1) Provide the filters as parameters 
2) Provide a pointer to the named profile which is a chunk of XML Schema

and you think that 1)is rubbish and much prefer 2).

My main problem with 2) is that it opens things up for over-engineering
and it also makes the network element the authoritative source of the
subscription parameters while I really want the netconf client to be. It
also has some garbage collection issues.

Sharon

-----Original Message-----
From: Andy Bierman [mailto:[email-masked]] 
Sent: Thursday, June 22, 2006 11:20 AM
To: Chisholm, Sharon [CAR:ZZ00:EXCH]
Cc: [email-masked]
Subject: Re: Verbs Again (was RE: draft-shafer-netconf-syslog-00.txt)

Sharon Chisholm wrote:
> hi
> 
> If the working group gets too obsessed with architectural purity at 
> the expense of usability to the operators and management applications 
> trying to use an implementation of the protocol I think we are more in

> trouble.

I really disagree with this comment.
We spent about 5 years developing a set of standard operations for the
standard manipulation of NV-stored configuration data models encoded in
XML.  Instead of using it, some people want to pretend that filtering
parameters do not fall in the category of NV-stored configuration data.

> I've suggested previously the following criteria before deciding 
> whether it was appropriate to create a new verb:
> 
> 1. Is it possible to perform this with existing verbs without 
> exploiting data model side efforts or being ridiculously complicated? 
> 2. Is this operation a management operation and if so does it make 
> sense for a management protocol to provide this function natively?
> 
> question. Providing it native in the protocol operation keeps it 
> simple, easy to use and highlights its importance over arbitrary bits 
> of configuration.
> 

To retrieve data

Andy

> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Andy Bierman

---

From: Andy Bierman <[email-masked]>
Date: Thu, 22 Jun 2006 09:36:19 -0700

> also has some garbage collection issues.

Forcing every manager application to come up with the correct
filtering parameters every time a new session is started is
extremely inefficient for both the manager and the agent.

Using sharable, NV-stored parameters on the agent, configured by
the authoritative manager, applications are not forced to
maintain the possibly massive set of notification filters on their
own.

It is very inefficient for the agent to parse, validate, setup
and store all the filtering parameters, every time a session
that wants notifications starts up, and duplicated for every session.

IMO, these inefficiencies dwarf the task of removing
NV-stored filters on the agent that are no longer of interest
to any management application.

Andy

---

From: "Kent Watsen" <[email-masked]>
Date: Thu, 22 Jun 2006 11:07:08 -0700

I agree with Sharon on this.  There are clear differences between
session-specific and device-wide parameters.  There is the obvious bit
about session-specific parameters being cleared when the session drops;
but more importantly is that no two sessions (i.e. applications) should
be aware of what any other session/app is up to - this principle is
violated if session-specific params are NV-stored

Also, while Andy is correct that session-specific filters do require
more parsing, validating, and setup - this overhead is dwarfed by the
device actually forwarding the notifications to the client over the
course of the session

BTW, Phil's proposal is good wrt all this in that it uses device-wide
streams (i.e. topics) on which session-specific filters can be applied.

Kent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Andy Bierman
Sent: Thursday, June 22, 2006 9:36 AM
To: Sharon Chisholm
Cc: [email-masked]
Subject: Re: Verbs Again (was RE: draft-shafer-netconf-syslog-00.txt)

Andy

>>>  > I'm not prepared to be the first lamb on that altar.  ;^)
Someone
>>      configuration.  Read-only vs. read-write is an artificial
the
>> archive: <http://ops.ietf.org/lists/netconf/>
the

---

From: Andy Bierman <[email-masked]>
Date: Thu, 22 Jun 2006 11:21:12 -0700

> violated if session-specific params are NV-stored

fine

> streams (i.e. topics) on which session-specific filters can be applied.

Neither proposal allows for NV-storage of parameters
which are not session-specific.  This is a major problem.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Thu, 22 Jun 2006 11:27:25 -0700

> device actually forwarding the notifications to the client over the

This is irrelevant to the discussion because the agent
has to send the notification regardless of how it is configured.
We are comparing the relative overhead of different schemes
to start notification delivery.

Andy

---

From: "David T. Perkins" <[email-masked]>
Date: Thu, 22 Jun 2006 11:44:03 -0700 (PDT)

HI,

I'm a little confused about the recent messages about notifications.

It seems like there are at least two independent issues
that are being mashed together.

These are:
1) in real time - who do you send notification to, and
     what filtering (if any) do you apply before sending
     them? Also, in addition (or instead) of sending
     notifications, you might want put them in a log.
     To me, this is configuration of 
        a) notification targets (a network entity or
                                 "local" log)
        b) filters
        c) log management

2) historical - on systems that have logs (with one being
     a notification log), extracting entries from the log.
     To me, this is state retrieval of:
        a) list of logs (and attributes)
        b) content of a log (specific log records)

In both cases, there are security concerns, primarily
access control to the notification info (in real-time,
or later looking through a log).

So, tell me, are they separate, or have I missed the
connecting part. If so, what is it?

Regards,
/david t. perkins

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 22 Jun 2006 21:00:13 +0200 (CEST)

"Sharon Chisholm" <[email-masked]> wrote:
> question.

Are we still talking about <get-syslog-streams> from Phil's draft?
That's the verb that some us thought was better done as a data model
and <get> and <get-config>.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 22 Jun 2006 12:04:38 -0700

> 1) in real time - who do you send notification to, and

This part is already determined by the session.

Left out of this entire discussion is the assumption that
the Callhome mechanism will be generalized and fully specified
in a separate document, so the agent can connect to the manager
over SSH or BEEP and say "ask me to send you notifications".

...

Andy

---

From: "Kent Watsen" <[email-masked]>
Date: Thu, 22 Jun 2006 12:32:49 -0700

> which are not session-specific.  

There is nothing in either proposal that limits the device from also
having static configuration for forwarding events to a standard syslog
server -   these proposals are specific to forwarding events over a
NetConf session...

Kent

-----Original Message-----
From: Andy Bierman [mailto:[email-masked]] 
Sent: Thursday, June 22, 2006 11:21 AM
To: Kent Watsen
Cc: Sharon Chisholm; [email-masked]
Subject: Re: Verbs Again (was RE: draft-shafer-netconf-syslog-00.txt)

fine

Andy

> From: [email-masked] [mailto:[email-masked]]
On

---

From: Andy Bierman <[email-masked]>
Date: Thu, 22 Jun 2006 13:25:04 -0700

> server -   these proposals are specific to forwarding events over a

I do not think we should replicate the mechanisms already
in place for NV-config in any manner whatsoever in the standard.
If vendors want to ignore the standard operations and invent their own,
then they can do that.

Defining new standard mechanisms to get and set specific chunks
of standard data, and also defining a separate standard data model
for use with the existing standard mechanisms, is a poor design choice.

The most complicated part of the notification config
is the filtering.  This is clearly not a session-specific
parameter, and something that several people have said they
want to be able to NV-store on the agent.

Saying that the standard solution for configuring the
netconf standard does not preclude someone from defining
additional solutions which provide for NV-storage
does not really address the problem.

Andy

---

From: "David T. Perkins" <[email-masked]>
Date: Thu, 22 Jun 2006 14:20:34 -0700 (PDT)

HI,

I'd like to reinforce my previous message with some
tweeks to the conceptal model found in Phil's document.
(Yes, already I've lost half or more of the readers.)

There is the unmodified figure from section 2.1:

     +----+
     | c1 |---+
     +----+   |    +------+                        (off-box)
     +----+   |    |      |
     | c2 |   +--->|syslog|          +-------+     +-------+
     +----+   |    |daemon|--------->|netconf|<--->|netconf|
      ...     |    |      |         >|server |     |client |
     System   |    +------+        / +-------+     +-------+
    Components|          \        /
      ...     |           v      /
     +----+   |         (----------)
     | cn |---+         (historical)
     +----+             (  events  )  (e.g. log files)
                        (----------)

And here it is with a few modifications that change how
you might want to look at it:

                                 off-box                           
     +----+              ------->collection of
     | c1 |---+         /        "syslog servers"
     +----+   |    +------+                        (off-box)
     +----+   |    |      |
     | c2 |   +--->|syslog|          +-------+     +-------+
     +----+   |    |daemon|          |netconf|<--->|netconf|
      ...     |    |      |         -|server |     |client |
     System   |    +------+        />+-------+     +-------+
    Components|          \        //
      ...     |           v      v/
     +----+   |         (----------)
     | cn |---+         (collection)
     +----+             (of on-box )
                        (log files )
                        (----------)

The configuration for the syslog daemon is the
addresses of the off box syslog servers, and
the on-box "logs", and the filtering to apply
before sending to a syslog server or saving
in a log. The document calls each paired
target and filter an "event stream". 

I see setting up the event streams as
configuration. However, I see that getting
syslog data as monitoring status (and
state).

If you compare the two pictures, you will notice
that there is no longer a direct connection
between the syslog daemon and the netconf server.
I did this to clarify that getting syslog
data via netconf was a "monitoring" operation
and not a "configuration" operation.
Note that the event streams may be filtered
before saved in a log file, and that data
from a log file may be filtered before
it is returned to the netconf client.

Now it seems like what is being argued about
is how to specify a filter for a monitoring
operation. And this could be done in two
ways. First, one could configure named filter
expressions on the system. Then when a
monitor operation was started, it could specify
the name of an existing filter expression.
Secondly, one could allow a monitor operation
to specify an unnamed filter expression as
a parameter to the monitor operation.

I believe that either one provides the same result.

Hope this helps.

Regards,
/david t. perkins

---

From: Andy Bierman <[email-masked]>
Date: Thu, 22 Jun 2006 14:56:46 -0700

> HI,

thanks for this email Dave.
I think it helps a lot.
Your diagram gets to the architecture issue that
Dave H> and I have raised.
but I don't see the key distinction as off-box or on-box,
or historical vs. live events.

IMO, if you change "off-box" to "traditional syslog service"
(NV-configured syslog feature on a NE device) and "on-box"
to "netconf session",  it represents the real architecture
that is going to be present in the agent:

                                   traditional
       +----+              ------->delivery via
       | c1 |---+         /        syslog protocol
       +----+   |    +------+
       +----+   |    |      |
       | c2 |   +--->|syslog|  netconf    +-------+     +-------+
       +----+   |    |daemon|  session    |netconf|<--->|netconf|
        ...     |    |      | ----------> |server |     |client |
       System   |    +------+  delivery   +-------+     +-------+
      Components|        |                  //
        ...     |        |                 //
       +----+   |        |       (------------)
       | cn |---+        |       (notification)
       +----+            +-----> (  logging   )
                                 (  service   )
                                 (------------)

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 23 Jun 2006 08:51:16 +0200

On Fri, Jun 23, 2006 at 08:37:34AM +0200, Rainer Gerhards wrote:
> I have made two other slight modification to the diagram:
> 
> 1. I have removed the term "traditional", because that sounds like
>    it implies that only RFC 3164 syslog is supported. As far as I see
>    it, netconf-syslog can support any flavor of syslog, including the
>    upcoming standards.
> 
> 2. I have added SYSLOG delivery on the receiving side of the 
>    syslog daemon. 
> 
>                                    delivery
>        +----+              ------->via syslog
>        | c1 |---+         /        protocol
>        +----+   |    +------+
>        +----+   |    |      |
>        +----+   |        +-----> (  logging   )
>                 |                (  service   )
>   delivery      |                (------------)
>   via syslog ---+ 
>   protocol

I do not understand what "netconf session delivery" means. For me, a
netconf session exists between a netconf server and a netconf client.
It is also unclear what the double lines // mean. If the figure is
meant to indicate the syslog data flow, then also the double arrow
between netconf server and netconf client is wrong. If the arrows are
supposed to mean something else, I think a legend is required to
discuss the figure. Well, a legend is required in any case.

/js

-- 
Juergen Schoenwaelder		    International University Bremen
<http://www.eecs.iu-bremen.de/>	    P.O. Box 750 561, 28725 Bremen, Germany

---

From: "Rainer Gerhards" <[email-masked]>
Date: Fri, 23 Jun 2006 08:37:34 +0200

delivery

Rainer

> [mailto:[email-masked]] On Behalf Of Andy Bierman
> Sent: Thursday, June 22, 2006 11:57 PM
> To: David T. Perkins
> Cc: [email-masked]
> Subject: Re: Tweeks to draft-shafer-netconf-syslog-00.txt
> 
> David T. Perkins wrote:
> > HI,
> > 
>

---

From: "Rainer Gerhards" <[email-masked]>
Date: Fri, 23 Jun 2006 09:48:49 +0200

If I understood the spirit of the previous diagrams right, then how
about that:

                                     remote
       +----+             --SYSLOG-->syslog
       | c1 |---+        /           receiver
       +----+   |    +------+
       +----+   |    |      |
       | c2 |   +--->|syslog|       +-------+           +-------+
       +----+   |    |server|       |netconf|<-NETCONF->|netconf|
        ...     |    |      |------>|server |           |client |
       System   |    +------+       +-------+           +-------+
      Components|     ^  |              ^
        ...     |     |  |              |
       +----+   |     S  |              v 
       | cn |---+     Y  |      (------------)
       +----+         S  +----->(notification)
                      L         ( repository )
                      O         (------------)
       remote         G
       syslog --------+
       sender

Arrows with protocol names indicate message transport via the named
protocol. Arrows without names use no specific protocols and may be
internal APIs. SYSLOG terminology (sender, receiver) is defined in
draft-ietf-syslog-protocol-17. The arrow between the "netconf server"
and the "notification repository" indicates that the netconf server
accesses the repository. The arrow between "syslog server" and
"notification repository" indicates that the syslog server submits data
to the repository but is not expected to pull any data out of it. The
data flow shown is informative, only. It does not demand any specific
implementation.

Some more points:

1. I have changed the "notification logging service" to "notification
repository". IMO, the important point is that there is a repository for
notifications. In actual software, I'd expect this to be a data store
which is written by the syslogd and read by the netconf server. It could
(should?) also be written to (configured) by the netconf server (e.g.
expiration of log data, access policies). It could be discussed if the
netconf server should have other write capabilities (e.g. delete
events), but I think this is irrelevant wrt this draft).

2. I have replaced the term "syslog daemon" by "syslog server". Though
the former is well-known, the later is more generic and
platform-independent.

3. One word on the relationship of the syslog to the netconf
server.There is no pure syslog or pure netconf server on either side. A
pure syslog server can not talk netconf to a netconf server because it
does not know about that. Actually, I would expect both of them to be
inside a single app (at least from the architecture point of view),
talking to each other via API, local RPC or simple internal function
calls. Something like this:

                  +---------------------------+
                  |  syslog/netconf app       |
                  |  +------+      +-------+  |        
           --SYSLOG->|syslog|      |netconf|<-NETCONF->
                  |  |server|-API->|server |  |        
                  |  +------+      +-------+  |
                  +---------------------------+

One might now argue that this is the reason that all events should go
through the the event repository. IMHO this is not a practical option as
we would like to see realtime delivery. In any case it is nothing that
happens on the wire, so it is out of the scope of any IETF WG. I still
find it useful to have the conceptual data flow in the diagram.

Rainer

---

From: Balazs Lengyel <[email-masked]>
Date: Fri, 23 Jun 2006 09:48:08 +0200

I think the problem is common for both drafts.
You have to define a group of data to tell the netconf server which events to forward / 
filter (and for call home where to send them).
Do we want to do this with a edit-config and get or do we want new protocol operations ?

Balazs

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
TSP System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320     email: [email-masked]

---

From: Balazs Lengyel <[email-masked]>
Date: Fri, 23 Jun 2006 09:43:54 +0200

Hello,
IMO when designing a standard "does not preclude" is not enough. We should either define 
the NV configuration model or forget it altogether (which still does not prevent 
individual implementations but is clearly outside the standard).
Balazs

---

From: Balazs Lengyel <[email-masked]>
Date: Fri, 23 Jun 2006 09:59:37 +0200

Hello,
Some comments to the netconf-syslog draft:
1) If I want to refer to data in my configuration store how do I do that in the syslog 
solution?
2) If I want to refer to a previous Netconf operation in syslog I have to embed the 
Netconf sessionId, messageId and some additional XML in syslog in a proprietary way. Not 
so nice.
3) Do we need call-home for syslog?

While this solution is good for nodes that already use syslog, for those that do not, this 
is an additional protocol layer that you have to wrap your information in. Additional work.
Also I see a risk that we might delay standardizing a number of important features like:
how to refer to the data model and other netconf operations in netconf notifications

I would rather support Sharon's idea to provide syslog as one option for netconf 
notifications but definitely not the main method.

Balazs

---

From: Andy Bierman <[email-masked]>
Date: Fri, 23 Jun 2006 01:34:57 -0700

> I do not understand what "netconf session delivery" means. For me, a

I like Rainer's latest diagram.
I wanted the diagram to show is that
syslog protocol does not go away in this new architecture.

I wanted to the diagram to clearly show that the netconf
syslog delivery service is a "nice-to-have" copy-service
for the benefit of netconf NMS developers, not a "must-have"
notification service for the benefit of network operators.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 26 Jun 2006 17:21:40 -0400

>to allow this as the stop time is already in the past. I would allow this!

The draft allows this; see 3.2.1.1 (Closing the Response).

>Use a data model for streams and use the get and edit-config
>operations instead of a custom operation for a small set of special
>(?) data. Even if streams can be defined in multiple ways
>(configuration, system default, etc.) they still can nicely be modeled
>as read-only data. 

System defaults aren't read-only data; see 1.3 in the netconf-prot
spec.

>Someone has to be :-) otherwise NETCONF is dead already.

Nope, it's the opposite.  NETCONF is in its infancy.  The raw config
parts are starting to gel and get widely implemented but the role of
standard data models, how they are defined, where they fit, how and
when the translation between the standard and the device-specific
occurs, and what to do when things don't quite translate are all wide
open areas which need tons of work.

We should work to make our puppy grow up, but should continue to do
reasonable work in parallel to allow the growing pup is useful asap
for taming real world problems.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 26 Jun 2006 17:22:43 -0400

>also has some garbage collection issues.

I go the other way, since I want the device to define what the client
will be able to receive.  I want to look at my device's config and
know that I'm safe and sane.  I don't want to have to depend on the
safety and sanity of every application.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 26 Jun 2006 17:22:29 -0400

Andy Bierman writes:
>models encoded in XML.

Five years?  Wow...  "And thank you so much for bringing up such a
painful subject. While you're at it, why don't you give me a nice
paper cut and pour lemon juice on it?"  (-- Miracle Max)

;^)

>> And why do we have <get> and <get-config> again? 

<get> was the "thar be dragons" part of the spec.  <get-config>
seemed so much safer. ;^)

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 26 Jun 2006 17:22:57 -0400

>which are not session-specific.  This is a major problem.

The netconf-syslog draft allows this, since it has a set of streams
defined by the device which specify a set notification data available
from the device.  The stream may have filters (and other parameters)
associated with it.  Not every stream must exist in the device's
configuration, but they are certainly allowed to.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 26 Jun 2006 17:23:24 -0400

Balazs Lengyel writes:
>1) If I want to refer to data in my configuration store how do I do
>that in the syslog solution?

There's no current mechanism, but we should be able to use
the <error-path> from <rpc-error> (with a different tag name).

>2) If I want to refer to a previous Netconf operation in syslog
>I have to embed the Netconf sessionId, messageId and some
>additional XML in syslog in a proprietary way. Not so nice.

We could do these as standard SD parameters.

>3) Do we need call-home for syslog?

They are complementary, but not prerequisites.

>While this solution is good for nodes that already use syslog,
>for those that do not, this is an additional protocol layer that
>you have to wrap your information in. Additional work.

Considering the overwhelming volume of syslog data and the minimal
size of the wrapping, this seems like an easy call.

>how to refer to the data model and other netconf operations in

I don't really follow you here.  How does this draft increase the risk
of delay for standardizing other features?

>I would rather support Sharon's idea to provide syslog as one option
>for netconf notifications but definitely not the main method.

The three options are snmp, syslog, and roll-your-own.  Operators and
vendors are fully invested in both snmp and syslog, so rolling your
own really is not an acceptable option.  The volume of existing syslog
data, as well as the extensibility, make this my choice.

Basically, my goal is to take the biggest available data set and move
it onto the netconf.  That's what the draft attempts to do.  It's not
an ideal solution, nor one that one would aim for on a green field
world, but given the zillions of lines of syslog() calls and the
current way that operators are using this data, it's the best path to
the maximum content.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 26 Jun 2006 17:23:07 -0400

Most excellent feedback.  I'll update the draft accordingly.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 26 Jun 2006 14:58:11 -0700

> seemed so much safer. ;^)

Doesn't really matter if it's <get> or <get-config>.

I'm not willing to accept a solution path that starts out with
little piles of unrelated "pseudo-data models" which do not
utilize the standard operations we already have.

I think that managing 100s or 1000s of <get-foo> and
even <set-foo> functions, which ignores the <candidate>,
<running>, <startup> model variants, ignores <edit-config>,
and does not show up at all when I do a <copy-config>,
<get-config>, or <get> operation -- is a fairly broken design.

This isn't a hard problem.
We just define a data model and instead of reflexively
labeling everything max-access read-only, we label it
read-create instead.

As far as data organization, again, not a hard problem.

1) Define a namespace for netconf data:

    urn:ietf:params:xml:ns:netconf:data:1.0

2) Define a structure to organize data and separate config
    from state data, e.g.:

<rpc message-id="101"
    xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <get>
    <filter type="subtree">
     <netconf xmlns="urn:ietf:params:xml:ns:netconf:data:1.0">
       <notifications>
         <config/>
         <data/>
       </notifications>
     </netconf>
    </filter>
  </get>
</rpc>

3) Transfer the pseudo-data model to the appropriate container
    and call it a real data model.

Andy

---

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 27 Jun 2006 09:53:10 +0200

> We could do these as standard SD parameters.

STRUCTURED-DATA in syslog was intentionally choosen to be not XML. The
reason is that very low end receivers do not need to take care of XML
and existing applications can quickly be adopted. It is arguable if that
is the best choice (right now this topic is again brought up on the
syslog mailing list). HOWEVER, that does not mean that STRUCTURED-DATA
can not contain XML. It can, but as part of a value. So to embed NETCONF
session information, you could use something like this in structured
data:

[netconf rpc="<rpc-reply message-id=\"101\"
              xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\" />"]

I agree it looks somewhat ugly, but it is an absolutely "clean" solution
and in no way proprietary. You need to take into account message size
limitations in syslog, which may limit your ability to use large chunks
of XML inside the actual syslog message. If syslog is transported over
netconf, this obviously is a no-issue, so you do not need to place
restrictions here. All of this is in the spirit of the syslog I-Ds,
which provision of highly different capablities of senders and
receivers. syslog does this because these capability differences do
exist in practice and we can not ignore them.

> current way that operators are using this data, it's the best path to

I support Phil's position. SYSLOG is a de-facto standard and very widely
deployed. It seems irrealistically that any time soon the POSIX logging
API *and* all applications using it will be upgrade to NETCONF
(notifications) or whatever. Not even the syslog WG expects this for
upcoming syslog changes. My personal position is that we will have
legacy syslog for at least another 10 years, if not longer. The
challenge is to mine syslog data while allowing to integrate it into a
more sophisticated system. NETCONF seems to have some potential as a
notification receiver (though I wouldn't have expected that from a
network *configuration* protocol in the first place). 

The most important thing obviously is a united event notification data
model (for syslog, SNMP, netconf, name-your-own). But, IMHO, we are very
far from that. Even if we had, we would need to use transforming
gateways for quite a while. So before we go for this glory united data
model, we should try to get all the data into a single location. That
can only be done via an evolutionary approach (look at RFC 3195 in the
syslog space and you'll see what happens if things are too different).
This is where I see Phil's draft. It is better to have something
not-so-perfect within a (relatively) short period of time than to wait
many years for the ultimate solution. You always can upgrade the
not-so-perfect one, but if you do it first, you can enjoy its benefits
while upgrading it.

Just my 2cts...

Rainer

---

From: Phil Shafer <[email-masked]>
Date: Tue, 27 Jun 2006 12:20:21 -0400

><get-config>, or <get> operation -- is a fairly broken design.

I don't think it's broken at all.  I think this is the point of using
an RPC-based design.  There are two reasons that I think this is a
win.  First, it's a better match for applications.  More specific
information in the API makes it a better API, both more readable and
more easily implemented.  Your unix program doesn't get filesystem
information using get() or even ioctl().  It uses statfs() to get this
data.  It's a direct, easily documented mechanism for getting
this particular content.  The same is true for RPCs.  I'd rather have:

    $results = $dev->get_syslog_streams();

than:

    $results = $dev->get("netconf/notifcations/config/whatever");

(or worse, having to build the xml object)

It's similar to calling read() verses calling syscall() with the SYS_read
system call number.

Second, we're not making a green field solution.  Device vendors have
a ton of code supporting existing CLI commands, all brimming with
information accessible via the command line.  So I go in and make a
<rpc-name> to "command line" mapping, and get all the fancy command
line arguments, types, ranges, help strings, etc for free.  All this
is fairly trivial code.

But to do this scoped generic stuff, I need to understand a _whole_
lot more about the output of the commands, including not only which
commands make which output tags, but which _options_ to those
commands make the output.  "show interfaces extensive" .vs. "show
interfaces statistics".  This is an immense pain.

I'd basically have to build a completely new management system.  I've
said it before, but I continue to believe that netconf will fail if it
becomes the "third way" (where first=cli and second=snmp).  In the
real world, the CLI gets implemented first, then SNMP (first
proprietary MIBs, then standard MIBs).  A completely disjoint netconf
implementation will a distant third.  We need to maximize the leverage
we can get off the existing implementations, not become the third way.

This was discussed at length in the prelude to netconf's creation.
One of our goals was allowing the vendor to reuse their cli
implementation to the greatest extent possible.

In addition, the use of explicit RPCs gives you explicit arguments,
which you need (or may need in the future) to refine the information
retrieved.  Or you may want your API for retrieving data to match the
API for manipulating it, in the same way the <source> parameter for
<get-config> matches the <source> parameter for <lock-config>,
<unlock-config>, <edit-config>, <validate>, etc.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 27 Jun 2006 10:24:09 -0700

Phil Shafer wrote:

What vendors do with their proprietary CLI/XML data models
does not have a significant impact on what the IETF
or this WG should do wrt/ standard data models.

One significant operator requirement that came out
of RFC 3535 is that the entire device configuration
and state information could be retrieved as a whole tree,
or portions of the tree could be retrieved, in a single operation.

The concept of an ad-hoc collection of specialized RPCs
used INSTEAD of standard data models and standard operations
does not meet this requirement.

Unless standard data shows up in exactly the same place,
with the same syntax and same semantics, using the <get> operation,
then it isn't a standard that meets this requirement.

Data model discovery becomes quite a nightmare if a manager
has to know priori, each different set of special RPCs to
use for every device it manages, and try them all to discover
the entire device configuration and state.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 27 Jun 2006 13:44:04 -0400

>then it isn't a standard that meets this requirement.

Does <get-config> pass that test?

>the entire device configuration and state.

Yes, this is a hard problem.  But thinking that all devices will
instantly converge on a new standard data model won't fly.  The
difference between the model and the device can be ignored or painted
over for monitoring (ala SNMP MIBs) but when you move into
configuration, it's the details that matter.

So my proposal is to fly little fish first.  If your application
wants to get notification data, it has to know a priori that it
gets the list of streams via one RPC and gets the notification data
via another RPC.  Come to think of it, even if you had all this
hidden under <get>, your application would still have to know the
magic filter to pass in to get the list of streams and the magic
filter to pass in to get the notification data.  The difference is
whether you put this knowledge in the RPC (name and arguments) or
the filter's XML content (and the schema that defines it).

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Tue, 27 Jun 2006 13:53:50 -0400

Phil Shafer writes:
>So my proposal is to fly little fish first.

Maybe "fry" those little fish, eh?  Fry flying fish?

Even the spellchecker can't help me......

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Tue, 27 Jun 2006 11:52:54 -0700

Phil Shafer wrote:
> instantly converge on a new standard data model won't fly. 

This is where we strongly disagree.
If they want to conform to the standard "notification" capability,
then they will do everything that is specified, including
implementing the tiny standard data model in the spec.

I will be even more clear -- the standard data models
for netconf will be usable with the existing standard netconf
operations.  I doubt the co-Chairs and ADs will except
a solution that did otherwise.

If the data is particularly complicated (e.g., deeply
nested data structures with complex keys), then special
RPCs in ADDITION to the standard RPCs may be useful,
which do not redefine data, but rather simplify access to
the nested data instances within the overall target data model.

In this case, I don't think the notification data is complicated
enough to warrant additional special RPCs, but we haven't finalized
the data model at all, so who knows right now.

Andy

  The

---

From: Phil Shafer <[email-masked]>
Date: Tue, 27 Jun 2006 16:26:17 -0400

>implementing the tiny standard data model in the spec.

For netconf to be widely accepted we need to minimize the price of
implementation, leveraging existing software features and configuration.

For example, JUNOS configures its syslog logging under the "system
syslog" statement, where a set of files (and remote destinations)
are configured.  Using the mechanism in the draft, I can expose
these files as streams for netconf clients with fairly low
implementation costs.

If the cost of implementing netconf is moving every config statement
to a new hierarchy and reimplement all software to fit this new
netconf world, then netconf is going nowhere.

Thanks,
 Phil

---

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 27 Jun 2006 21:50:37 +0200

Andy, 

> Sent: Tuesday, June 27, 2006 8:53 PM
> To: Phil Shafer

Phil wants to transport syslog messages. I do not think there is a
specific data model for a syslog messages needed, at least at this time
and in the NETCONF WG alone. The entity "syslog message" is
well-defined, so you do have a single element inside a syslog event
notification. Over time, I agree, it would be useful to have an extended
data model that specfies detailled semantics for syslog messages (and
probably notifiation messages at all). This, however, I think is far
beyond the current discussion. I consider it to be a separate effort,
probably even in a separate WG that draws people from the syslog and
netconf WGs.

Syslog stream configuration, IMHO, could relatively quickly be done
because this is something that merely happens in the netconf-syslog
draft context. I can also envison that a generic syslog rule
configuration data model can quickly be done. One, that takes into
account what almost all syslogd implementations supports. It may require
two optional capabilities, but you would immediately be able to capture
the majority of syslog receivers and relays. I am not so experienced
with schema defintion, but I am tempted to write a quick draft on a
basic syslog config schema. I defintely know what is required inside the
model (acting as a least common denominator approach).

> the data model at all, so who knows right now.

I think this is not just a data model question. Something like
<get-syslog-events> is not a datamodel thing, at least I think so. It
definitely is not configuration data, so <get-config> is out of
question. I also do not consider it to be state data, so I have some
concerns on <get>. What else would be the right verb to transport
notification data?

Rainer

---

From: Andy Bierman <[email-masked]>
Date: Tue, 27 Jun 2006 13:00:05 -0700

> probably even in a separate WG that draws people from the syslog and

I'm not sure there is WG consensus that the only type
of notification data that can possibly be delivered
in netconf is syslog/RAW.

But you misunderstood my comments anyway.

The netconf WG is defining standard parameters for the
configuration of agent filtering of notification delivery.
There is also the possibility of standard monitoring variables
for netconf notifications.

These are the standard data models that I am referring to,
not the content of syslog messages or any other notification
content.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 27 Jun 2006 23:02:42 +0200 (CEST)

> implementation, leveraging existing software features and configuration.

Again, suppose an agent actually did make this data configurable
through NETCONF.  First of all, it would have to do this in a
vendor-specific way.  Second, it would have to make this data
available _both_ as a normal <get-config> part, _and_ as a new RPC,
<get-syslog-streams>.

An agent that only provides read-only access to this data would
provide the data as a new RPC instead of a <get> part.  I don't
understand why the former would be easier to implement than the
latter?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 27 Jun 2006 14:46:20 -0700

> netconf world, then netconf is going nowhere.

This doesn't make a lot of sense.
You are okay with standard parameters for syslog delivery
when they are passed in a special RPC, but if those standard
parameters are moved to a data model, suddenly you can't
map your proprietary mechanism to the standard anymore.
If you could handle the burden of standard parameters
passed in a special RPC, then you can handle the burden
if they are passed in a existing RPC.

Remember that those existing RPCs represent the minimal
conformance level for a netconf agent.  An implementation MUST
advertise the base capability, which includes all the
operations we have been discussing.  Any other capabilities
MUST be in addition to the base capability.  So it is
not a burden for a conforming netconf agent or manager
to use the base operations, since they are never optional.

As for the need for standard data models at all...
It doesn't really matter from a standards POV what any particular
vendor has implemented for configuration.  What matters is
that an NMS developer can code to the standard, and it works.

We are not chartered to produce standard data models for the
entire IETF -- just the NETCONF protocol (just as the SNMP WG
developed standard MIBs for the management of the SNMP protocol itself.)

The bar has been raised in this WG.
We are going to eat our own recipe.
We are going to provide fully configurable features
for the NETCONF protocol.  It may be okay for other WGs
to say "leave configuration to the vendors, that is not our focus",
but not this WG.  Standards based configuration is our focus.
(Not syslog delivery, BTW.)

Andy

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 28 Jun 2006 09:33:02 +0200

Andy,

I have to admit that I actually misunderstood you. Both your reply below
as well as the message sent somewhat later made this clear.

If I now understand you correctly, you remind us that the charter calls
for a standardized notification method and related data model - and not
something syslog-specific. I see the reasoning for this position.

HOWEVER, IMHO that implies the discussion about verbs vs. data model on
Phil's draft is the wrong one. What should be discussed firsthand is if
the draft itself is supported by the WG or not. Phil's draft is all
about transporting syslog over netconf. The argument is that there is a
wealth of information in syslog that a netconf manager might want to see
(natively). But still, the focus of the draft is transporting syslog.
The quesion of verbs vs. data model is a secondary one (and I think
there my comments still applys).

I would suggest that we first discuss if the overall idea (transporting
syslog over netconf in *addition* to a generic event notification
transport) is within the scope of this WG. If so, we can then look at
the specifics (like verb usage). I am too inexperienced with netconf to
offer a qualified opinion on that firsthand question. I can, however,
contribute to the details if it is WG consensus to follow that route.

Rainer

---

From: Andy Bierman <[email-masked]>
Date: Wed, 28 Jun 2006 06:56:07 -0700

> Andy,

You are correct.
We have separate issues here, but we also do not
all have the same end result in mind, and that is a problem.
The WG meeting in Montreal will be devoted to reaching
more agreement on the functional requirements.

NETCONF is content-neutral protocol, so syslog content
is certainly within scope, but so is anything else.
If designed correctly (like the <rpc> and <rpc-reply> PDUs)
the notification content will just be "well formed XML payload"
within the notification PDU.  The namespace URI for the
payload elements will identify the structure of its contents.

As for verbs vs. data, there 2 points not open to debate
wrt/ any standard mechanisms developed by the NETCONF WG:

  1) Any NV-stored parameters will be defined in a standard data model
     suitable for use with the base protocol operations for
     manipulating NV-stored config data.

  2) Any monitoring data (e.g., netconf protocol statistics and state)
     will be defined in a standard data model suitable for use with the
     <get> operation.

I have already asked the WG to comment on both the I-Ds
as they pertain to the requirements list Juergen has been maintaining.
There are (different) parts of both drafts I like
and don't like.  I don't think this is a "one draft or the other draft"
situation across the board.  There are many "1 or the other" decisions
to make though.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 28 Jun 2006 11:09:52 -0400

>available _both_ as a normal <get-config> part, _and_ as a new RPC,

The config is vendor-specific, by nature, and the translation from
this form to a standard data model isn't always straight-forward.

Take a look at the config groups and commit scripts features in JUNOS
and think about how the mapping from device config to standard config
isn't trivial:

http://www.juniperfederal.net/techpubs/software/junos/junos61/swconfig61-getting-started/html/cli-config-groups2.html

http://www.juniper.fi/techpubs/software/junos/junos76/swconfig76-automation/html/commit-scripts-overview.html

These are slick features that reduce the size and complexity of the
configuration.  The trade-off is that what you see from "show
configuration" (or <get-config>) isn't the config the system
components see (unless you use the "display" pipes).  In addition,
stuff like "inactive" configuration can cloud the issue further.

http://www.juniper.net/techpubs/software/junos/junos57/swconfig57-getting-started/html/cli-configuration-mode43.html

While these features are outside the WG's concerns, they are inside
mine ;^)

In operational mode, the user sees full operational impact of the
expanded configuration, so the CLI's "show log" will see all the
defined syslog streams, regardless of where they came from.  This
same information is available via the <get-syslog-streams> operation.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 28 Jun 2006 08:41:03 -0700

Phil Shafer wrote:
> Sure.  I think mapping to an abstract model is reasonably
> easy (think SNMP MIBs), but mapping from an abstract model
> to device configuration is not easy (think SNMP MIBs).

Think how unusable SNMP would be if the MIB objects
didn't have OID assignments and therefore every NMS
has to hard-code the vendor-assigned OID for every MIB
object, for every version of every device for every vendor.

Think how unusable SNMP would be if there was no MIB-walk,
and instead the NMS had to hard-wire the special PDU to use
for every MIB object, for every version of every device for
every vendor.

That's not exactly how standards are supposed to work.

>> So it is
> The point is that <get-config> and <edit-config> for device-specific
> configuration is simple.  Doing them for a data model that is not
> implemented verbatim on the device is not.  I think you imagine
> people doing verbatim implementations of whatever data model that
> netconf defines.  I don't.  I see a big network with lots of deployed
> boxes and lots of configuration data that isn't doing to move to a
> brave new world in my lifetime.  I want to make something that's
> useful in my lifetime, so mandating this isn't doing it for me.

It doesn't matter what proprietary stuff you have on your router.
There are zero standard data models for NETCONF.  For new
NETCONF features, you can expect new standard data models
to configure them. We go from zero to one, then two, etc.
We get WG consensus on the contents of every standard data model.

This is no different than standard MIB modules.
IETF WGs have managed for years to define standard MIB objects,
even though proprietary MIB objects also exist.  With SNMP
you can use the same PDUs to access vendor data as standard data.

> The goal is to make something that solves today's problems in
> reasonably efficient, reasonably cost-effective ways, that
> vendors will implement and operators will use.

There are many people (including myself) who believe that
standard configuration data models are a critical component
to a complete standards based solution for network configuration.

This will take time, but standard protocols should be configurable
in a standard way.  Vendor extensions to standards will always
exist, as well as proprietary solutions.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 28 Jun 2006 08:56:50 -0700

> this form to a standard data model isn't always straight-forward.

We spent a long time getting WG consensus, IESG consensus, and
IETF consensus on the operations in prot-12.  We are going to
use them.

The SNMP WG managed to define a standard data model to configure
which SNMP notifications to send, and where to send them.
If we can't do the same for NETCONF, then I don't see the point of this
exercise at all.

We managed to define a filtering mechanism (+ Xpath) for filtering
XML payloads in <get-config> and <get> operations without too
much trouble.  Sharon managed to define some filtering mechanisms
for notifications that are content-neutral.

The NETCONF WG is not the right place to standardize syslog-specific
details.  We should only be defining netconf-specific notification delivery
parameters.

> While these features are outside the WG's concerns, they are inside

The WG is just trying to standardize what is common and
has WG consensus.  This process is independent of proprietary
efforts that may overlap in some areas.

Andy

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 28 Jun 2006 17:58:07 +0200

Phil,

I can not fully agree with you here. My understanding is that you can do
something like this (probably with a different URN):

      <get-config>
         <source>
           <running/>
         </source>
         <filter type="subtree">
           <top xmlns="http://example.com/schema/1.2/config">
             <syslog-streams/>
           </top>
         </filter>
       </get-config>

I understood netconf-prot so that a config must not necessarily be
writeable. Of course, that would be best. But looking at your scenario
below: what if you create the syslog-streams object by whatever
vendor-specific method you have but still provide read-only access to a
standardized (but virtual) syslog-streams object? So even the
configuration does not natively support syslog-streams, it still allows
a manager to deal with your syslog-streams, provided you advertise the
capability. If then I modify the stock linux syslogd to support netconf,
I could also provide read-only access to a standardized syslog-streams
object. Again, it would be virtual, because my stock syslogd config is
quite different from what you have in your JUNOS config.

The benefit of these virtual read-only syslog-streams config objects
would still be that a manager could handle them in a vendor-neutral way.
Even though different vendor-specific data models are required to setup
that virtual config, it could be useed in a device-independent way.

As far as syslog is concerned, I think a useful least common denominator
writeable syslog-config object could also be defined. It is my firm
belief that it would not be much of a burden to implement this even on
very different configuration models (because the base objects are quite
obvious and supported by all implementations anyway).

I hope I have not overlooked anything obvious.

Rainer

> Sent: Wednesday, June 28, 2006 5:10 PM
> To: Martin Bjorklund
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: Re: Verbs Again 
> 
> Martin Bjorklund writes:
> http://www.juniperfederal.net/techpubs/software/junos/junos61/
> swconfig61-getting-started/html/cli-config-groups2.html
> 
> http://www.juniper.fi/techpubs/software/junos/junos76/swconfig
> 76-automation/html/commit-scripts-overview.html
> 
> http://www.juniper.net/techpubs/software/junos/junos57/swconfi
> g57-getting-started/html/cli-configuration-mode43.html
>

---

From: Phil Shafer <[email-masked]>
Date: Wed, 28 Jun 2006 12:00:41 -0400

>This is no different than standard MIB modules.

We _have_ to be different than the process that made standard MIBs
if we want to be usable for configuration.  We can't follow the
same path and expect that folks will use ours just because it's in
XML.  There has to be more.

>even though proprietary MIB objects also exist.

If you see this as successful (for configuration), why are we here?

>to a complete standards based solution for network configuration.

I completely agree that standard configuration data models are
critical, but I do not agree that they are trivial.  I think this
is the next big hurdle for this working group.  The question on the
floor is whether we should stop other work until we have a real
meta-model for configuration and a way of specifying standard data
models that will work in the real world.  If so, let's stop talking
about notifications and get our butts in gear on modeling.

Me, I don't want to wait for it.  I want to see usable capabilities
for doing operational tasks that applications need in order to
manage devices.  Things like file transfer, call-home, software
installation, reboot, and system status.  I think we can progress
in these areas in parallel.

But if the concensus if that we can't, we should stop talking about
other topics while we work on modeling.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Wed, 28 Jun 2006 12:09:19 -0400

>would still be that a manager could handle them in a vendor-neutral way.

The benefits of having this appear as config are lost if it's
read-only.  At the point, it becomes state data, where I agree, one
should be able to reasonably represent device-specific state (whether
from defaults or config) in a standard way.  This is pretty much
what <get-syslog-streams> does.

Thanks,
 Phil

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 28 Jun 2006 18:35:04 +0200

Phil,

I see one big difference between state data and config data that is read
only on a *specific* device. My point is that config data is potentially
writable, just not on the device in question. To keep with the syslog
sample: I could modify stock syslogd to accept configuration via a
hypothetical syslog-streams configuration object (the same that's
queried). So in theory it is for read-write access. Some devices might
actually grant read-write access, some just read access (because their
vendor doesn't care about implementing write access for whatever reason)
and some others do not implement it at all.

I think this idea is beyond syslog. I agree with Andy that the netconf
WG most probably is not the right place to discuss syslog data models.
Unfortunately, the syslog-sec WG is also not the right place (being in
the security area and chartered to just provide a secure transport). It
might make sense to try to form a WG that looks at syslog configuration
and content standardization.

Rainer

> From: Phil Shafer [mailto:[email-masked]] 
> Sent: Wednesday, June 28, 2006 6:09 PM
> To: Rainer Gerhards
> Cc: Martin Bjorklund; [email-masked]; 
> [email-masked]; [email-masked]
> Subject: Re: Verbs Again 
>

---

From: Andy Bierman <[email-masked]>
Date: Wed, 28 Jun 2006 09:35:48 -0700

> If you see this as successful (for configuration), why are we here?

SNMP has problems as an API for configuration that NETCONF
addresses fairly well.  However, the basic concept of a WG
reaching consensus on the syntax and semantics of individual
standard knobs for standard protocols is still valid.

> other topics while we work on modeling.

I know we should be working on data modeling instead of notifications
but we aren't.  That is still no excuse for not using the
standard <edit-config> and <get> operations.

I don't think data organization should be that hard,
and we don't have to organize the entire world right now.
We just have to organize the netconf protocol data.

I use the following organization (which can be nested and repeat)

  <application-name  xmlns="owner-namespace">
    <parameter-set-name>
      <parameter-name>
    </parameter-set-name>
    <monitor-set-name>
       <mon-object-name>
    </monitor-set-name>
  </application-name>

E.g.;

  <netconf xmlns="netconf-data">
    <notification-config>
      ....
    </notification-config>
    <notification-mon>
      ...
    </notification-mon>
  </netconf>

(read-only data can be defined in a parameter set, but the
monitor set is supposed to be used instead for data sets
that are all read-only)

Something simple like this should be good enough to get us started.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Wed, 28 Jun 2006 13:08:54 -0400

"Rainer Gerhards" writes:
>Some devices might

If the goal is read-only access to config data, then the problem's
not hard.  But I think we're aiming for more.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Wed, 28 Jun 2006 11:18:17 -0400

Thanks,
 Phil