Subject: [Syslog] RE: draft-shafer-netconf-syslog-00.txt

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 20 Jun 2006 14:39:15 +0200

Phil,

as promised, I have gone through the netconf-syslog draft from the
syslog guy's perspective. I think netconf-syslog is a very useful
addition to the syslog community, too. I am cross-posting my message to
the syslog WG.

The most important thing that strikes me is terminology. Even though
syslog terminology might not be familiar (or sound even strange) to the
NETCONF WG, I think the draft should stick with it. The reason is that
otherwise you will create confusion. One sample already under
discussion:

In <text-pattern> you specify "the message". As a syslog-guy, I would
apply the regex to the message as whole and not just the MSG part. Thus,
I would find a match e.g. in STRUCTURED-DATA. This is a very natural
approach for someone who implements syslog. Using the same terms also
eases the work for the implementor. From my understanding, the server to
support netconf-syslog must support both netconf as well as syslog. It
is quite confusing in code if the thing the syslog code side e.g. calls
"severity" becomes "level" as soon as another part of the same code is
called. In syslog-protocol, we tried hard to use the most common terms
for new header fields. Unfortunately, syslog has a lot a variance, but I
think we actually managed to find quite good terms.

So I suggest the following term changes:

netconf-syslog          syslog
priority			PRI
level				severity
process			PROCID
event				MSGID
text				MSG (because it works on the MSG field)

I also suggest to add APP-NAME as a filter in netconf-syslog. Adding TAG
(from RFC 3164 & 3195) will also be useful.

Some things of more substance:
<format>
In syslog-protocol, we have introduced the VERSION field. It denotes the
version of syslog-rptocol message format of the message. It deliberately
starts at 1 and increases monotonically (if considerable changed new
versions of syslog should be specified).

I suggest that in netconf-syslog <format> is replaced by <VERSION>,
where zero denotes RFC 3164/3195 format and non-zero denotes a version
according to syslog-protocol. This also facilitates parsing the actual
syslog message on the receiving netconf syslog parser.

<parameter>
I suggest to rename it to STRUCTURED-DATA. I understand that the format
is quite different from the actual syslog STRUCTURED-DATA format. But it
filters on the STURUCTURED-DATA field. This is the same principle I
applied above for text/MSG. I think it would be consistent to call the
filter properties like the fields they operate on.

Names for severity and facility
RFC 3164 and syslog-protocol both list names (e.g. "authorization" and
"emergency") as well as integer numbers for PRI contents. However, only
the integers are normative (see ABNF in syslog-protocol, PRI description
in RFC 3164). It has also been observed in practice that the names are
not without ambiguity. So I strongly suggest to use the integer values
only in netconf-syslog filters. If there is no consensus on this, I
propose to support both the names as well as the numbers.

Remotly Received Events 
>From my understanding of netconf-syslog, a netconf-syslog server
supports forwarding via netconf both locally generated syslog messages
as well remotely received ones. I think the ability to process remotely
received messages is a vital asset. It enables a netconf-syslog server
to act as a hub/gateway to a number of netconf-unaware syslog clients.

To fully support remote syslog clients, a HOSTNAME filter is needed.
That would be a (regex?) filter on the hostname. Similar filters are
widely deployed on syslogds in practice today.

When multiple host message may be present in a single netconf-syslog
stream, it might also be that different VERSIONS of the syslog protocol
be present in a single stream. The syslog WG expects this to be a
frequent scenario once syslog-protocol is finished. syslog-protocol
already contains an identification for this (via the VERSION fields) as
well as some specifics on transforming RFC 3164 syslog in a
syslog-protocol receiver (A.1 in syslog-protocol-17). I suggest that
netconf-syslog supports the same.

Finally one small correction. On page 4 of netconf-syslog (at the top)
it states:

"The SYSLOG protocol lacks security and reliability."

This is not true. There is RFC 3195 (syslog over BEEP) which supports
security and reliability. 3195 is currently only weakly accepted, but
there as some implementations of it. I suggest a reference is made to
3195 to clarify on that issue (and so that a casual reader will become
aware that there is more to look at then 3164 and syslog-protocol).

I hope these comments are useful.

Best regards,
Rainer Gerhards

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Phil Shafer
> Sent: Saturday, June 17, 2006 6:08 AM
> To: netconf
> Subject: draft-shafer-netconf-syslog-00.txt
> 
> I've finally got an initial version of the draft for a netconf
> capability that makes syslog data available over long-lived RPCs.
> I just sent it off to the RFC Editor, so it should show up on
> ietf.org sometime this weekend, but an advance copy is available.
> 
> http://professional.juniper.net/phil/ietf/draft-shafer-netconf
> -syslog-00.txt
> 
> Thanks,
>  Phil
> 
> --
> to unsubscribe send a message to [email-masked] with
> the word 'unsubscribe' in a single line as the message text body.
> archive: <http://ops.ietf.org/lists/netconf/>
> 

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: Phil Shafer <[email-masked]>
Date: Tue, 20 Jun 2006 12:30:19 -0400

>addition to the syslog community, too. I am cross-posting my message to

Truly appreciate your comments, especially the terminology ones.  I'll
update the draft with these next week.

>propose to support both the names as well as the numbers.

I strongly dislike using numbers, but if there's consensus
to use them, I'll survive.

>to act as a hub/gateway to a number of netconf-unaware syslog clients.

The target audience for NETCONF (network service providers) don't
configure their boxes to forward syslog messages.  These devices
are the source of events and operators dislike having open ports
on their boxes that aren't strictly required.

But I can see the utility of this for other environments and
the cost is minimal, so I'd consider rolling it in, except....

>be present in a single stream.

... this sounds like it will be an issue.  Can we avoid mixing
VERSIONs in a single stream?

>aware that there is more to look at then 3164 and syslog-protocol).

The target of this comment was 3164, since 3195 doesn't seem to
have been picked up by the operator community, but I'll add a
reference to 3195 and clarify my comment.

Thanks,
 Phil

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 21 Jun 2006 10:49:17 +0200

Phil,

> to use them, I'll survive.

I understand your position. At a minimum, I strongly suggest that you
allow either numbers or names. I think most of the filter data is
operator-entered, so it is left to the operators decisions what to use.
This is especially important when different syslog message sources use
different names (though this could be handled by the gateway
exclusively).

> VERSIONs in a single stream?

Let me ask where you see an issue with multiple versions. For me it
seems fairly easy to support them. I've also thought a little more about
the local environment. I think there is no way you can guarantee that
even a single machine will provide consistently one message format
version or the other.

Reasoning: The diagram in 2.1 of netconf-syslog precisely describes how
the syslog subsystem works in many environments, namely Unix and Linux
based systems. The syslog daemon is not the actual message generator. In
fact, it is more like a relay. From the on-the-wire point of view, it is
the initial sender. But if you think about local API, it is just a
receiver forwarding (and potentially transforming) messages. The actual
message generators are system components (called c1, c2, .... cn in your
diagram). These components generate messages via API calls. The API
covers only very limited syslog header fields. Most of the header is
generated by the *system component*. So you are actually not dealing
with a single syslogd and its format but with a potential myriad of
different components on the box. This has been identified as a major
issue moving legacy (rfc 3164) syslog to a standard (syslog-protocol).
This is also one reason why there are so many different formats.

In order to guarantee a consistent data model, you would need to change
the (POSIX) logging API. As this change can not be done evolutionary (it
would break existing apps), the old API would still needed to be
supported. It is thought in the syslog WG that that API change should
happen some time. But we are definitely some years away from that. Not
to mention how long the old API will be utilized...

So my firm believe is that it will be the absolutely common case for
syslog implementations to have to deal with different message formats.

Much of the evil in that can be avoided, if a syslog data model is
defined. This, together with transformation rules, would make it very
easy to handle different versions. As I have written in another mail
this morning, syslog-protocol has such a data model "on its mind" but
can not spell it out because the syslog WG is not chartered to do that.
But it is fairly easy. I have also a working proof-of-concept
implementation of the most important transformations in actual software
(www.rsyslog.com). Its fairly easy to extend that to a full data model
transformation.

> reference to 3195 and clarify my comment.

I agree that 3195 is not overly successful ;) However, you never know
what the future has in stock. As there is no additional effort to
supporting 3195 in netconf-syslog, I think it would be useful to mention
it and to also mention that it can be used together with netconf-syslog
(it is VERSION 0 format, just like 3164).

Rainer