Subject: RE: [Syslog] New direction and proposed charter

From: "David B Harrington" <[email-masked]>
Date: Thu, 1 Dec 2005 11:21:52 -0500

Hi,

I recommend we stop talking about RFC3164-compliant, since RFC3164 is
only an Informational document and not standards-track.

dbh 

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> Sent: Thursday, November 24, 2005 8:01 AM
> To: Glenn Mansfield Keeni
> Cc: [email-masked]
> Subject: RE: [Syslog] New direction and proposed charter
> 
> Glenn,
> 
> > Now the question is : are there any RFC3164 compliant devices
> > (relays and syslogd's) and applications.
> 
> I have to admit that I have written at least one compliant to 
> RFC 3164.
> But my feeling is that I am very lonley with that. In one project,
we
> had to make RFC 3164 not only optional but to disable it by default,
> because as soon as you enable it, everything is broken. 
> 
> Also, I have done a lot of testing with different syslogds
yesterday.
> None of this quite popular systems is compliant. I neither know any
> compliant device. Anton has backed this point of view (for example,
he
> adds Solaris as being non-compliant).
> 
> RFC 3164 is an informational document whichs utility was primarily
to
> describe security issues. It is good in that. However, it 
> does not even
> remotely describe what BSD syslog is about (a good indication of
that
> might be that BSD syslog works totally different from RFC 3164...).
> 
> I question that it makes any sense to modify our efforts to 
> take care of
> an informational RFC that doesn't even describe what is current
> technology. Don't take me wrong: I myself based a lot of my 
> decisions on
> RFC 3164. Now, after extended testing, I thinkt that was a great
> failure. When it comes to the observed format, RFC 3164 should tell
> about <PRI> only. Other than that, there are no similarities 
> in observed
> syslog behaviour. Sorry for the harsh words, but if you go to the
lab,
> that's where you end up (notice my own disappointment yesterday...).
> 
> > If there are then we gain some and lose very little. [ Correct
> > me here if am wrong in saying that we lose very little. I may
> > have missed something]
> > 
> 
> No sure on "lose very little". I am tired of taklking theory. I will
> implement something and then tell. I guess we will loose some 
> things, at
> least in ease of parsing. And I still think we do not gain anything
at
> all...
> 
> > I will agree that many implementations are not RFC3164 compliant.
> > But, it will be difficult to convince IESG folks or, anyone for
> > that matter, that there are no RFC3164 compliant devices or
> > applications at all. [ Note that the applications matter too!]
> 
> OK, let's poll the list: who has implemented RFC 3164 including
relay
> behaviour described there?
> 
> I have two candidates, MonitorWare/WinSyslog (some core component)
on
> Windows and rsyslogd on *nix. MonitorWare is the one with
> default-disabled rfc3164. rsyslogd uses a much enhanced parser (non
> compliant) to get its work done. rsyslogd would probably benefit
from
> your suggestion. However, I do not think this to be worthwhile
enough,
> because a) there are currently too few deployed to make it 
> matter and b)
> the next version will natively support-syslog protocol and is an
> extremely easy upgrade.
> 
> > My suggestion is- avoid changes wherever we can,
> 
> I agree on this, but in that specific case I really think its 
> not worth
> it. RFC 3164 creates a false impression of format understanding.
Other
> than <PRI>, there is noting in common about existing syslog
> implementations.
> 
> Rainer
> > 
> > Cheers
> > 
> > Glenn
> > 
> > Rainer Gerhards wrote:
> > > This is really disappointing...
> > > 
> > > I have done further testing with more syslogds to confirm 
> > the initial
> > > findings. The more different programs / versions I try, the 
> > more a mess
> > > this gets. OK, we knew FreeBSD syslogd does not include the 
> > hostname.
> > > Next I tested with some Windows products, which looked 
> > promising. Then I
> > > looked at the sysklogd source. It is the standard Linux 
> > package, so if
> > > it works, a lot would be won. The source tells me that at 
> least the
> > > timestamp should correctly be extracted. Then I actually 
> > tried it out
> > > with a Debian system - the timestamp was ignored. Checking 
> > that source
> > > tree I see that *that* sysklogd does deliberately ignore 
> > the data and
> > > always pulls the local date (or ist it a bug - who 
> > cares...). When it
> > > comes to relaying it is even stranger: sysklogd does 
> > neither relay the
> > > timestamp nor the host part. At that point, I have stopped
further
> > > analysis, because I think I would be able to find another 
> > good number of
> > > variants of syslog behaviour.
> > > 
> > > Conclusion:
> > > 
> > > 1. There is no point in trying to preserve backward
> > >    compatibility besides sticking with the <PRI>. Everything
other
> > >    than <PRI> is handled differently by different
implementations 
> > >    and/or versions.
> > > 
> > > 2. We can NOT expect that relaying over existing syslog 
> > >    implementations will ever work. Please note that this would 
> > >    also have broken syslog-sign without specifically implemented
> > >    daemons.
> > > 
> > > As such, I revert back to proposing 
> > > 
> > > <PRI>VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID 
> > SP MSGID SP
> > > [SD-ID]s SP MSG
> > > 
> > > or, somewhat incorrect but shorter:
> > > 
> > > <PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [SD-ID]s
MSG
> > > 
> > > Please note that I have added the MSGID to the header.
> > > 
> > > Rainer
> > > 
> > > 
> > >>-----Original Message-----
> > >>From: [email-masked] 
> > >>[mailto:[email-masked]] On Behalf Of 
> Rainer Gerhards
> > >>Sent: Wednesday, November 23, 2005 3:04 PM
> > >>To: Glenn Mansfield Keeni; [email-masked]
> > >>Subject: RE: [Syslog] New direction and proposed charter
> > >>
> > >>Glenn,
> > >>
> > >>very interesting approach with the timestamp. I think your 
> > >>ideas can be
> > >>the key to maintaining a lot of backwards compatibility by still
> > >>retaining new functionality.
> > >>
> > >>First some bad news: I am not sure if by "BSD syslog" you 
> > are refering
> > >>to RFC 3164 or a specific distribution of BSD. I have 
> > created a small
> > >>script to test out your recommendation. I used FreeBSD stock 
> > >>syslogd as
> > >>the receiver.
> > >>
> > >>It did NOT work as expected. There are two reasons
> > >>
> > >>a) (that) BSD syslogd takes the sender always from the system 
> > >>that send
> > >>it
> > >>b) even worse, when relaying, it puts "Forwarded from 
> > >><hostname>: " into
> > >>
> > >>   the hostname part (yes, with all that spaces)
> > >>
> > >>So while the idea sounds excellent, it does not work with 
> > >>stock FreeBSD
> > >>syslogd. I am not sure about other BSD variants, nor have I 
> > >>checked the
> > >>sysklogd package. I believe it will have less issues in 
> this regard.
> > >>   
> > >>This was the message I sent (via perl script):
> > >>
> > >>"<148>Oct 11 22:14:15 mymachine.example.com 1 ID47 2003T.003Z 
> > >> 'su root'
> > >>failed for lonvick on /dev/pts/8"
> > >>
> > >>this was the raw message received after being relayed once 
> > by FreeBSD
> > >>stock syslogd:
> > >>
> > >>"<148>Oct 11 22:14:15 Forwarded from 172.19.2.7: 
> > >>mymachine.example.com 1
> > >>ID47 2003T.003Z  'su root' failed for lonvick on /dev/pts/8"
> > >>
> > >>As you can see, the message is somewhat distorted - 
> > definitely enough
> > >>for digital signatures to be broken. [Implementor's 
> > >>side-note: This can
> > >>be fixed on a syslog-application layer level, far beyond the 
> > >>IETF scope.
> > >>It's straightforward and easy to do, so it will probably
happen.]
> > >>
> > >>Even though this actual sample seems not to work, it paves 
> > >>the way to a
> > >>very elegant compatibility solution. The key is to add the extra
> > >>information (e.g. Timestamp) in a different place. At 
> least I was so
> > >>focussed on fields at whole that I did not notice this 
> > possibility. I
> > >>have experiemented a bit more with Glenn's proposal, 
> shuffeling some
> > >>fields. The result was this:
> > >>
> > >><PRI>BSD_syslog_timestamp FQDN TAG "@#"VERSION MSGID 
> > >>Remainder_Timestamp
> > >>[SD-ID]s MSG
> > >>
> > >>or in an actual sample:
> > >>
> > >>"<148>Oct 11 22:14:15 mymachine.example.com su[4711]: @#1 ID47
> > >>2003T.003Z [SD-IDs] 'su root' failed for lonvick on /dev/pts/8"
> > >>
> > >>I have used the BSD timestamp and FAQN as Glenn suggested. 
> > >>Then, I have
> > >>added the "TAG" again. If we think in the spirit of my mail 
> > >>this morning
> > >>on syslog & non-IETF standards, it would not really hurt if we
> > >>standardize TAG instead of two fields. If I would like to 
> retain the
> > >>APP-NAME and PROCESSID, I could do the following ABNF:
> > >>
> > >>TAG = APP-NAME ["[" PROCESSID "]"] ":"
> > >>
> > >>The side-effect of this is that almost all 
> syslog-messages currently
> > >>emited comply with that format. So I suggest that we 
> > strongly consider
> > >>joining these two again. Out of the sudden, we have the 
> > "old" header,
> > >>but it is parsable by a hyptothetical new syslogd. Next, I 
> > have used a
> > >>trick from syslog-sign. I have changed the VERSION from a 
> > number to a
> > >>number plus a cookie. The version would now be "@#1". I do 
> > not care if
> > >>the cookie is "@#" or something else. The key point is that 
> > >>it, together
> > >>with the version should be very unlikely to exist at that place
in
> > >>old-style syslog. That would allow a "new" receiver to 
> differentiate
> > >>between old and new style syslog messages. The rest of the 
> > message is
> > >>just applying Glenn's proposal again: it has the MSG, the 
> > >>missing parts
> > >>of the timestamp, SD-IDs and MSG. The Remainder_Timestamp 
> > >>looks strange.
> > >>We might like it, we might like something else. That's easy 
> > to change
> > >>and discuss. It's the concept that matters right now, not 
> the exact
> > >>format.
> > >>
> > >>If we take the outlined route, we would be able to extend 
> the syslog
> > >>protocol with as much backward compatibility as is possible in a
> > >>not-yet-standardized world. I find this very desirable. I 
> > >>think we even
> > >>have good chances that many existing "old" syslogds would 
> relay such
> > >>messages without changing them, thus keeping digital 
> > >>signatures intact.
> > >>The required text changes for syslog-protocol should be
moderate.
> > >>
> > >>I strongly propose we go in that direction.
> > >>
> > >>Rainer
> > >>
> > >>>-----Original Message-----
> > >>>From: [email-masked] 
> > >>>[mailto:[email-masked]] On Behalf Of Glenn 
> > >>>Mansfield Keeni
> > >>>Sent: Wednesday, November 23, 2005 1:39 PM
> > >>>To: [email-masked]
> > >>>Subject: Re: [Syslog] New direction and proposed charter
> > >>>
> > >>>Chris/Rainer,
> > >>>
> > >>>
> > >>>>we continue to use <PRI>... at the start of syslog 
> > >>>
> > >>>messages.  This will
> > >>>
> > >>>>allow current receivers to continue to receive messages and 
> > >>>
> > >>>put them in
> > >>>
> > >>>>the right bins.  Does anyone disagree with this?
> > >>>
> > >>>Complete agreement.
> > >>>
> > >>>>
> > >>>>The WG has agreed to use the timestamp Rainer has in the
current
> > >>>>syslog-protocol.
> > >>>
> > >>>In principle I agree with the timestamp format. It is good.
> > >>>I may have missed the discussion on this matter,  in that 
> > >>
> > >>case please
> > >>
> > >>>accept my apologies and ignore the rest of the mail.
> > >>>
> > >>>To get existing BSD syslog devices specifically relays into 
> > >>>the compatibility
> > >>>fold it WILL be good idea to keep the timestamp in two parts
> > >>>
> > >>>       RainerTimestamp =  BSD_syslog_timestamp  + 
> > RemainderTimestamp
> > >>>
> > >>>
> > >>>>One possibility would be to assemble a syslog message as:
> > >>>>
> > >>>><PRI>TIMESTAMP FQDN VERSION MSGID [SD-ID]s MSG
> > >>>
> > >>>In the context of what has been said above about the 
> > >>>timestamp. I would
> > >>>suggest
> > >>>  <PRI>BSD_syslog_timestamp FQDN VERSION MSGID 
> > >>>Remainder_Timestamp [SD-ID]s MSG
> > >>>
> > >>>That would allow existing BSD-syslog relays to handle the new 
> > >>>syslog protocol
> > >>>in a transparent manner. Everything from VERSION to the end 
> > >>>is treated as "message".
> > >>>
> > >>>We do not lose information.
> > >>>     The Remainder_timestamp carries it - in a slightly less 
> > >>>convenient place
> > >>>     though.
> > >>>
> > >>>On the other hand if we insist on using RainerTimestamp, 
> > >>>existing BSD_syslog
> > >>>relays will relay the message as
> > >>>
> > >>><PRI>BSD_syslog_timestamp FQDN RainerTimestamp FQDN VERSION 
> > >>>MSGID [SD-ID]s MSG
> > >>>
> > >>>The message does get distorted to some extent.
> > >>>
> > >>>
> > >>>>If we can agree to this then I suspect that we can have 
> a working
> > >>>>document within Rainer's timeframe.  I'll propose the 
> > >>>
> > >>>following charter
> > >>>
> > >>>>to keep us focused.
> > >>>>
> > >>>>-------- Proposed Charter  --------------
> > >>>>
> > >>>>Syslog is a de-facto standard for logging system events.  
> > >>>
> > >>>However, the
> > >>>
> > >>>>protocol component of this event logging system has not 
> > >>>
> > >>>been formally
> > >>>
> > >>>>documented.  While the protocol has been very useful and 
> > >>>
> > >>>scalable, it
> > >>>
> > >>>>has some known security problems which were documented in 
> > >>
> > >>RFC 3164.
> > >>
> > >>>>The goal of this working group is to address the security and
> > >>>>integrity problems of the existing syslog mechanism while 
> > >>>
> > >>>not breaking
> > >>>
> > >>>>backwards compatibility.  The most obvious problems that 
> > >>
> > >>need to be
> > >>
> > >>>>addressed in the syslog protocol are the timestamp, 
> which has not
> > >>>>formally included a means to indicate the year, and the 
> > >>>
> > >>>identification
> > >>>
> > >>>>of the source which has been a hostname without a 
> qualified domain
> > >>>>name.  Additionally, a version, some type of message 
> > >>>
> > >>>indicator, and a
> > >>>
> > >>>>means to convey structured data will be included in the 
> protocol.
> > >>>>
> > >>>>syslog has traditionally been transported over UDP and 
> this WG has
> > >>>>already defined RFC 3195 for the reliable transport for 
> the syslog
> > >>>>messages.  The WG will separate the UDP transport from the 
> > >>>
> > >>>protocol so
> > >>>
> > >>>>that others may define additional transports in the future.
> > >>>>
> > >>>>- A standard will be produced that formally documents the
syslog
> > >>>>protocol.  A mechanism will also be defined in this 
> specification
> > >>>>that will provide a means to convey structured data.
> > >>>>
> > >>>>- A standard will be produced that documents the UDP 
> transport for
> > >>>>syslog.
> > >>>>
> > >>>>- A standard will be produced that documents a mechanism to
sign
> > >>>>syslog messages to provide integrity checking and source
> > >>>>authentication.
> > >>>>
> > >>>>- A MIB definition for syslog will be produced.
> > >>>>
> > >>>>-------------------------------------------
> > >>>>
> > >>>>
> > >>>>PLEASE review this and respond.
> > >>>>
> > >>>
> > >>>Glenn
> > >>>
> > >>>
> > >>>_______________________________________________
> > >>>Syslog mailing list
> > >>>[email-masked]
> > >>>https://www1.ietf.org/mailman/listinfo/syslog
> > >>>
> > >>
> > >>_______________________________________________
> > >>Syslog mailing list
> > >>[email-masked]
> > >>https://www1.ietf.org/mailman/listinfo/syslog
> > >>
> > > 
> > > 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Syslog mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/syslog
> > 
> 
> _______________________________________________
> Syslog mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/syslog
> 

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog