Subject: RE: [Syslog] New direction and proposed charter

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 22 Nov 2005 11:55:52 +0100

WG,

I have completed the promised testing. I used various syslogds on Linux,
BSD and Windows platforms. The list obviously is not complete, but I
think I got a fair enough sample of what is deployed.

The good news is that by putting the <PRI> part in front of the message
all of them were able to put the otherwise syslog-protocol-15 formatted
message into the right bins. The format recorded was also acceptable,
the non-recognized hostname was replaced by the sender address and the
local date was added. This is within the typical current user experience
when different syslogds are being used. Please note that some (e.g. BSD
syslogd) do never pull the hostname from the message but always use the
sender's address.

The only culpit that I came along is associated with NUL octets. Many
syslogds can not handle them. The message is only recorded up to the
first NUL inside the message, no further interpretation happens. 

We have had discussion on this topic previously. As a reminder, it was
said that excluding the NUL would be a "crappy little rule" that could
open the door for many more CLRs. I still tend to think this is true and
the problem exposed is acceptable.

I try to sum up yesterday's discussion and my current position on it:

Once again, I think David's comments on the charter are in the right
direction
(http://www.mail-archive.com/syslog%40lists.ietf.org/msg00143.html). It
calls for some compatibility but puts emphasis on newer development. I
suggest we accept the wording.

We have had several comments on the field order in syslog-protocol.
Based on them, I propose the following format:

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID [SD-ID]s MSG

With (optional) SD-IDs for
- Extended Facility
- TRUNCATE 
- MSGID
- [Language Identification]

Please note that I moved MSGID to the optional SD-ID part and instead
re-introduced APP-NAME and PROCID as formal fields. I have done this
because these two were designed as a replacement for TAG, whereas MSGID
was meant to be something totally different. If we would prefer to have
one field, only, I recommend to name it TAG and use the same semantics
RFC 3164 describes.

I support Chris proposal to leave the current size limitations exactly
as they are in syslog-protocol. Everything else would cause the n-th
re-iteration of the message size issue (side note: exactly the same
discussion seems to have started on the NETCONF WG for event
notifications right now, so this is an ever-popular topic).

I have included an "Extended Facility" to provide finer-grain facility
control within the message. This was brought up by Anton and others and
does not seem to hurt anything if in SD-ID.

I have included the Language Identification. I don't object it, but I
question its usefulness.

If we follow the message defition above, we can probably recycle most of
the text in syslog-protocol, just shuffle it a little around. This has
two advantages:

- we do not loose what we already have discussed
- the work can progress rather quickly

Also, syslog-transport-udp most probably does not need any change at
all, at most in a minor way.

Modifying existing syslog receivers should not be very hard with the new
definitions. The only major issue I see from the implementors point of
view is UTF-8 decoding. But that is more of a storage problem. It is of
course possible to receive the message and store it as UTF-8. I do not
think this would cause non-compliance to the spec - or would it? If it
would, UTF-8 would most probably be a major drawback when it comes to
implementor acceptance. I am also a bit concerned about the NUL
character, which can only be handled in one of two ways with existing
syslog code base:

- implementing a byte-counted string class and do not use the C RTL
- replace NUL with an escape sequence upon reception (e.g. <00>)

I guess most implementations would take the later route. If we consider
this to be acceptable, the majority of syslogds should be fairly easy to
upgrade.

If we can agree on these points, I would volunteer to implement the
resulting document in C code so that we might see if there were any
hidden problems. I would try to apply as minimal changes as possible.

Suggestion for progressing:

- we need more comments from other list members!
- we should re-charter
- we should reach rough consensus on the new format
- once done, I can update syslog-protocol to it
- I (and others?) can do test implementations in
  parallel to the review
- discussion can show if (hopefully minor) adjustments
  need to be made
- the goal should still be to finish this work
  (including AD approval) by the next IETF meeting

Rainer

> -----Original Message-----
> From: Chris Lonvick [mailto:[email-masked]] 
> Sent: Monday, November 21, 2005 9:58 PM
> To: Rainer Gerhards
> Cc: [email-masked]; [email-masked]
> Subject: RE: [Syslog] New direction and proposed charter
> 
> Hi Rainer and all,
> 
> On Mon, 21 Nov 2005, Rainer Gerhards wrote:
> 
> > Chris & WG
> >
> >>
> >>> From the meeting, it sounds like we will get many more
> >> implementations if
> >> we continue to use <PRI>... at the start of syslog messages.
> >
> > ##############################################################
> >> This will
> >> allow current receivers to continue to receive messages and
> >> put them in
> >> the right bins.
> > ##############################################################
> >
> >> Does anyone disagree with this?
> >
> > Yes, disagreement here. For the reasons outline in mail from Friday,
> > this is *NOT* true. Existing syslog receivers will be 
> broken if we just
> > stick with <PRI> and do not adjust the other header fields.
> >
> > Of course, it is doable. For details, review
> >
> > http://www.mail-archive.com/syslog%40lists.ietf.org/msg00121.html
> >
> > (around the middle of the post).
> >
> 
> From my perspective, I think that having a syslog receiver 
> receive the 
> message and get it into the right bin is acceptable.  If we 
> don't have 
> that then we are breaking very much.  If we can accomplish that, then 
> receivers will continue to receive the messages and the 
> parsers will have 
> to be updated.
> 
> I believe that the alternative that you're saying, Rainer, is 
> that syslog 
> transmitters COULD keep their existing headers and our 
> Working Group only 
> put things in there.  If we go with that, then the parsers 
> will still need 
> to be updated to understand the SD-ID information.  I'd 
> prefer to just 
> keep the <PRI> and modify the rest of the packet.
> 
> We need to hear from more people on this.
> 
> Thanks,
> Chris
> 

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: Darren Reed <[email-masked]>
Date: Wed, 23 Nov 2005 03:51:24 +1100 (EST)

> <PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID [SD-ID]s MSG

I would put the SD-IDs after the message.

The SD-IDs and detailed bits of meaning to the MSG and without the
MSG, are irrelevant.  The exception being a language marker.

> - replace NUL with an escape sequence upon reception (e.g. <00>)

Why not \0 ?

Darren

---

From: Balazs Scheidler <[email-masked]>
Date: Wed, 23 Nov 2005 10:15:39 +0100

Hi everyone,

Just in case someone doesn't know me (would not be too suprising as I
have not posted to the list recently) I'm the author for syslog-ng a
popular syslog implementation for various UNIXes. To be honest apart
from being subscribed to this list I have not followed the discussions
recently (the reasons are more or less explained in the last couple of
mails by others), but I see the new proposals for the changed charter a
sign in the good direction. And the new format for syslog-protocol is
something that I personally like, so for what it's worth I support the
format and I'm willing to implement support for this or a similar format
in syslog-ng.

-- 
Bazsi