Subject: RE: [Syslog] New direction and proposed charter

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 23 Nov 2005 16:29:18 +0100

This is really disappointing...

I have done further testing with more syslogds to confirm the initial
findings. The more different programs / versions I try, the more a mess
this gets. OK, we knew FreeBSD syslogd does not include the hostname.
Next I tested with some Windows products, which looked promising. Then I
looked at the sysklogd source. It is the standard Linux package, so if
it works, a lot would be won. The source tells me that at least the
timestamp should correctly be extracted. Then I actually tried it out
with a Debian system - the timestamp was ignored. Checking that source
tree I see that *that* sysklogd does deliberately ignore the data and
always pulls the local date (or ist it a bug - who cares...). When it
comes to relaying it is even stranger: sysklogd does neither relay the
timestamp nor the host part. At that point, I have stopped further
analysis, because I think I would be able to find another good number of
variants of syslog behaviour.

Conclusion:

1. There is no point in trying to preserve backward
   compatibility besides sticking with the <PRI>. Everything other
   than <PRI> is handled differently by different implementations 
   and/or versions.

2. We can NOT expect that relaying over existing syslog 
   implementations will ever work. Please note that this would 
   also have broken syslog-sign without specifically implemented
   daemons.

As such, I revert back to proposing 

<PRI>VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP
[SD-ID]s SP MSG

or, somewhat incorrect but shorter:

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [SD-ID]s MSG

Please note that I have added the MSGID to the header.

Rainer

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> Sent: Wednesday, November 23, 2005 3:04 PM
> To: Glenn Mansfield Keeni; [email-masked]
> Subject: RE: [Syslog] New direction and proposed charter
> 
> Glenn,
> 
> very interesting approach with the timestamp. I think your 
> ideas can be
> the key to maintaining a lot of backwards compatibility by still
> retaining new functionality.
> 
> First some bad news: I am not sure if by "BSD syslog" you are refering
> to RFC 3164 or a specific distribution of BSD. I have created a small
> script to test out your recommendation. I used FreeBSD stock 
> syslogd as
> the receiver.
> 
> It did NOT work as expected. There are two reasons
> 
> a) (that) BSD syslogd takes the sender always from the system 
> that send
> it
> b) even worse, when relaying, it puts "Forwarded from 
> <hostname>: " into
> 
>    the hostname part (yes, with all that spaces)
> 
> So while the idea sounds excellent, it does not work with 
> stock FreeBSD
> syslogd. I am not sure about other BSD variants, nor have I 
> checked the
> sysklogd package. I believe it will have less issues in this regard.
>    
> This was the message I sent (via perl script):
> 
> "<148>Oct 11 22:14:15 mymachine.example.com 1 ID47 2003T.003Z 
>  'su root'
> failed for lonvick on /dev/pts/8"
> 
> this was the raw message received after being relayed once by FreeBSD
> stock syslogd:
> 
> "<148>Oct 11 22:14:15 Forwarded from 172.19.2.7: 
> mymachine.example.com 1
> ID47 2003T.003Z  'su root' failed for lonvick on /dev/pts/8"
> 
> As you can see, the message is somewhat distorted - definitely enough
> for digital signatures to be broken. [Implementor's 
> side-note: This can
> be fixed on a syslog-application layer level, far beyond the 
> IETF scope.
> It's straightforward and easy to do, so it will probably happen.]
> 
> Even though this actual sample seems not to work, it paves 
> the way to a
> very elegant compatibility solution. The key is to add the extra
> information (e.g. Timestamp) in a different place. At least I was so
> focussed on fields at whole that I did not notice this possibility. I
> have experiemented a bit more with Glenn's proposal, shuffeling some
> fields. The result was this:
> 
> <PRI>BSD_syslog_timestamp FQDN TAG "@#"VERSION MSGID 
> Remainder_Timestamp
> [SD-ID]s MSG
> 
> or in an actual sample:
> 
> "<148>Oct 11 22:14:15 mymachine.example.com su[4711]: @#1 ID47
> 2003T.003Z [SD-IDs] 'su root' failed for lonvick on /dev/pts/8"
> 
> I have used the BSD timestamp and FAQN as Glenn suggested. 
> Then, I have
> added the "TAG" again. If we think in the spirit of my mail 
> this morning
> on syslog & non-IETF standards, it would not really hurt if we
> standardize TAG instead of two fields. If I would like to retain the
> APP-NAME and PROCESSID, I could do the following ABNF:
> 
> TAG = APP-NAME ["[" PROCESSID "]"] ":"
> 
> The side-effect of this is that almost all syslog-messages currently
> emited comply with that format. So I suggest that we strongly consider
> joining these two again. Out of the sudden, we have the "old" header,
> but it is parsable by a hyptothetical new syslogd. Next, I have used a
> trick from syslog-sign. I have changed the VERSION from a number to a
> number plus a cookie. The version would now be "@#1". I do not care if
> the cookie is "@#" or something else. The key point is that 
> it, together
> with the version should be very unlikely to exist at that place in
> old-style syslog. That would allow a "new" receiver to differentiate
> between old and new style syslog messages. The rest of the message is
> just applying Glenn's proposal again: it has the MSG, the 
> missing parts
> of the timestamp, SD-IDs and MSG. The Remainder_Timestamp 
> looks strange.
> We might like it, we might like something else. That's easy to change
> and discuss. It's the concept that matters right now, not the exact
> format.
> 
> If we take the outlined route, we would be able to extend the syslog
> protocol with as much backward compatibility as is possible in a
> not-yet-standardized world. I find this very desirable. I 
> think we even
> have good chances that many existing "old" syslogds would relay such
> messages without changing them, thus keeping digital 
> signatures intact.
> The required text changes for syslog-protocol should be moderate.
> 
> I strongly propose we go in that direction.
> 
> Rainer
> > -----Original Message-----
> > From: [email-masked] 
> > [mailto:[email-masked]] On Behalf Of Glenn 
> > Mansfield Keeni
> > Sent: Wednesday, November 23, 2005 1:39 PM
> > To: [email-masked]
> > Subject: Re: [Syslog] New direction and proposed charter
> > 
> > Chris/Rainer,
> > 
> > > we continue to use <PRI>... at the start of syslog 
> > messages.  This will
> > > allow current receivers to continue to receive messages and 
> > put them in
> > > the right bins.  Does anyone disagree with this?
> > 
> > Complete agreement.
> > > 
> > > 
> > > The WG has agreed to use the timestamp Rainer has in the current
> > > syslog-protocol.
> > In principle I agree with the timestamp format. It is good.
> > I may have missed the discussion on this matter,  in that 
> case please
> > accept my apologies and ignore the rest of the mail.
> > 
> > To get existing BSD syslog devices specifically relays into 
> > the compatibility
> > fold it WILL be good idea to keep the timestamp in two parts
> > 
> >        RainerTimestamp =  BSD_syslog_timestamp  + RemainderTimestamp
> > 
> > > One possibility would be to assemble a syslog message as:
> > > 
> > > <PRI>TIMESTAMP FQDN VERSION MSGID [SD-ID]s MSG
> > 
> > In the context of what has been said above about the 
> > timestamp. I would
> > suggest
> >   <PRI>BSD_syslog_timestamp FQDN VERSION MSGID 
> > Remainder_Timestamp [SD-ID]s MSG
> > 
> > That would allow existing BSD-syslog relays to handle the new 
> > syslog protocol
> > in a transparent manner. Everything from VERSION to the end 
> > is treated as "message".
> > 
> > We do not lose information.
> >      The Remainder_timestamp carries it - in a slightly less 
> > convenient place
> >      though.
> > 
> > On the other hand if we insist on using RainerTimestamp, 
> > existing BSD_syslog
> > relays will relay the message as
> > 
> > <PRI>BSD_syslog_timestamp FQDN RainerTimestamp FQDN VERSION 
> > MSGID [SD-ID]s MSG
> > 
> > The message does get distorted to some extent.
> > 
> > > If we can agree to this then I suspect that we can have a working
> > > document within Rainer's timeframe.  I'll propose the 
> > following charter
> > > to keep us focused.
> > > 
> > > -------- Proposed Charter  --------------
> > > 
> > > Syslog is a de-facto standard for logging system events.  
> > However, the
> > > protocol component of this event logging system has not 
> > been formally
> > > documented.  While the protocol has been very useful and 
> > scalable, it
> > > has some known security problems which were documented in 
> RFC 3164.
> > > 
> > > The goal of this working group is to address the security and
> > > integrity problems of the existing syslog mechanism while 
> > not breaking
> > > backwards compatibility.  The most obvious problems that 
> need to be
> > > addressed in the syslog protocol are the timestamp, which has not
> > > formally included a means to indicate the year, and the 
> > identification
> > > of the source which has been a hostname without a qualified domain
> > > name.  Additionally, a version, some type of message 
> > indicator, and a
> > > means to convey structured data will be included in the protocol.
> > > 
> > > syslog has traditionally been transported over UDP and this WG has
> > > already defined RFC 3195 for the reliable transport for the syslog
> > > messages.  The WG will separate the UDP transport from the 
> > protocol so
> > > that others may define additional transports in the future.
> > > 
> > > - A standard will be produced that formally documents the syslog
> > > protocol.  A mechanism will also be defined in this specification
> > > that will provide a means to convey structured data.
> > > 
> > > - A standard will be produced that documents the UDP transport for
> > > syslog.
> > > 
> > > - A standard will be produced that documents a mechanism to sign
> > > syslog messages to provide integrity checking and source
> > > authentication.
> > > 
> > > - A MIB definition for syslog will be produced.
> > > 
> > > -------------------------------------------
> > > 
> > > 
> > > PLEASE review this and respond.
> > > 
> > 
> > Glenn
> > 
> > 
> > _______________________________________________
> > Syslog mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/syslog
> > 
> 
> _______________________________________________
> Syslog mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/syslog
> 

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: Rainer Gerhards <[email-masked]>
Date: Wed, 23 Nov 2005 18:26:21 +0100

For what it is worth...

I have tried to create a send-template for rsyslogd, my *nix syslogd. It

> Please note that I have added the MSGID to the header.

It was easy to do, actually a 5 minute job. Of course, sending a message
is another beast then parsing it, but I think this will also be
sufficiently simple. As far as I know (some) other syslogds have similar
template systems, so this point eventually proves that the modifications
required will not break anything at all.

The only tricky issue that remains is the NUL octect. The more I think
about it, the more I think the CLR to disallow it is less evil than to
make it stay...

Rainer

---

From: "Andrew Ross" <[email-masked]>
Date: Thu, 24 Nov 2005 12:17:11 +1300

>The only tricky issue that remains is the NUL octet. The more I think

I agree that having the CLR for NUL octet exclusion is OK.

Quick question, if someone is sending international data in UTF-8 format,
can there ever be a valid UTF-8 sequence that includes a NUL octet value? If
so, we need to allow NUL values in the MSG. 

Does anyone on the list know UTF-8 well enough to answer this?

Cheers

Andrew

---

From: Glenn Mansfield Keeni <[email-masked]>
Date: Thu, 24 Nov 2005 20:16:27 +0900

Rainer,
      Let me try once again. In the following please read "RFC3164"
for "BSD_syslog".

Format-A: <PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [SD-ID]s MSG
Format-B: <PRI>BSD_syslog_timestamp HOSTNAME VERSION
           Remainder_Timestamp APP-NAME PROCID MSGID [SD-ID]s MSG

Format-B would allow existing BSD-syslog (RFC-3164) compliant daemons
AND relays AND applications to handle the new syslog protocol in a
transparent manner.

Everything from VERSION to the end is treated as "message".
We do not lose information.
     The Remainder_timestamp carries it - in a slightly less
      convenient place though.

On the other hand if we insist on using RainerTimestamp as in FORMAT-A,

RFC3164 compliant syslogd implementations and applications will
not find the timestamp.

RFC3164 compliant syslogd implementations and applications will
not find the hostname.

RFC-3164 compliant relays will relay the message as
  <PRI>BSD_syslog_timestamp FQDN RainerTimestamp FQDN VERSION
   MSGID [SD-ID]s MSG
The message does get distorted to some extent.

Now the question is : are there any RFC3164 compliant devices
(relays and syslogd's) and applications.

If there are then we gain some and lose very little. [ Correct
me here if am wrong in saying that we lose very little. I may
have missed something]

I will agree that many implementations are not RFC3164 compliant.
But, it will be difficult to convince IESG folks or, anyone for
that matter, that there are no RFC3164 compliant devices or
applications at all. [ Note that the applications matter too!]

My suggestion is- avoid changes wherever we can,

Cheers

Glenn